Handle possible null pointers from malloc/strdup/strndup()

GCC 12.1.1_p20220625's static analyzer caught these.

Of the two in the btree test, one had previously been caught by Coverity
and Smatch, but GCC flagged it as a false positive. Upon examining how
other test cases handle this, the solution was changed from
`ASSERT3P(node, !=, NULL);` to using `perror()` to be consistent with
the fixes to the other fixes done to the ZTS code.

That approach was also used in ZED since I did not see a better way of
handling this there. Also, upon inspection, additional unchecked
pointers from malloc()/calloc()/strdup() were found in ZED, so those
were handled too.

In other parts of the code, the existing methods to avoid issues from
memory allocators returning NULL were used, such as using
`umem_alloc(size, UMEM_NOFAIL)` or returning `ENOMEM`.

Reviewed-by: Brian Behlendorf <[email protected]>
Reviewed-by: Tony Hutter <[email protected]>
Signed-off-by: Richard Yao <[email protected]>
Closes #13979

1 files changed, 8 insertions, 1 deletions

diff --git a/tests/zfs-tests/cmd/btree_test.c b/tests/zfs-tests/cmd/btree_test.c
index 4e2023003..ab8967b22 100644
--- a/tests/zfs-tests/cmd/btree_test.c
+++ b/tests/zfs-tests/cmd/btree_test.c
@@ -242,7 +242,10 @@ drain_tree(zfs_btree_t *bt, char *why)
 		zfs_btree_add_idx(bt, &randval, &bt_idx);
 
 		node = malloc(sizeof (int_node_t));
-		ASSERT3P(node, !=, NULL);
+		if (node == NULL) {
+			perror("malloc");
+			exit(EXIT_FAILURE);
+		}
 
 		node->data = randval;
 		if ((ret = avl_find(&avl, node, &avl_idx)) != NULL) {
@@ -319,6 +322,10 @@ stress_tree(zfs_btree_t *bt, char *why)
 
 		uint64_t randval = random() % tree_limit;
 		node = malloc(sizeof (*node));
+		if (node == NULL) {
+			perror("malloc");
+			exit(EXIT_FAILURE);
+		}
 		node->data = randval;
 
 		max = randval > max ? randval : max;