libzfs: Fix bounds checks for float parsing

UINT64_MAX is not exactly representable as a double.

The closest representation is UINT64_MAX + 1, so we can use a >=
comparison instead of > for the bounds check.

Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: Ryan Moeller <[email protected]>
Closes #10127 

1 files changed, 6 insertions, 1 deletions

diff --git a/lib/libzfs/libzfs_util.c b/lib/libzfs/libzfs_util.c
index a8e9ac9cb..c6debd576 100644
--- a/lib/libzfs/libzfs_util.c
+++ b/lib/libzfs/libzfs_util.c
@@ -1411,7 +1411,12 @@ zfs_nicestrtonum(libzfs_handle_t *hdl, const char *value, uint64_t *num)
 
 		fval *= pow(2, shift);
 
-		if (fval > UINT64_MAX) {
+		/*
+		 * UINT64_MAX is not exactly representable as a double.
+		 * The closest representation is UINT64_MAX + 1, so we
+		 * use a >= comparison instead of > for the bounds check.
+		 */
+		if (fval >= (double)UINT64_MAX) {
 			if (hdl)
 				zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
 				    "numeric value is too large"));