vn_getf/vn_releasef should not accept negative file descriptors

C type coercion rules require that negative numbers be converted into
positive numbers via wraparound such that a negative -1 becomes a
positive 1. This causes vn_getf to return a file handle when it should
return NULL whenever a positive file descriptor existed with the same
value. We should check for a negative file descriptor and return NULL
instead.

This was caught by ClusterHQ's unit testing.

Reference:
http://stackoverflow.com/questions/50605/signed-to-unsigned-conversion-in-c-is-it-always-safe

Signed-off-by: Richard Yao <[email protected]>
Signed-off-by: Andriy Gapon <[email protected]>
Signed-off-by: Brian Behlendorf <[email protected]>
Closes #450

1 files changed, 6 insertions, 0 deletions

diff --git a/module/spl/spl-vnode.c b/module/spl/spl-vnode.c
index 1e26b8e29..4c62097dc 100644
--- a/module/spl/spl-vnode.c
+++ b/module/spl/spl-vnode.c
@@ -648,6 +648,9 @@ vn_getf(int fd)
 	vnode_t *vp;
 	int rc = 0;
 
+	if (fd < 0)
+		return (NULL);
+
 	/* Already open just take an extra reference */
 	spin_lock(&vn_file_lock);
 
@@ -733,6 +736,9 @@ vn_releasef(int fd)
 {
 	file_t *fp;
 
+	if (fd < 0)
+		return;
+
 	spin_lock(&vn_file_lock);
 	fp = file_find(fd);
 	if (fp) {