OpenZFS 7614, 9064 - zfs device evacuation/removal

OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete

This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk.  The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.

The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool.  An entry becomes obsolete when all the blocks that use
it are freed.  An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones).  Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible.  This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.

Note that when a device is removed, we do not verify the checksum of
the data that is copied.  This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.

At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.

Porting Notes:

* Avoid zero-sized kmem_alloc() in vdev_compact_children().

    The device evacuation code adds a dependency that
    vdev_compact_children() be able to properly empty the vdev_child
    array by setting it to NULL and zeroing vdev_children.  Under Linux,
    kmem_alloc() and related functions return a sentinel pointer rather
    than NULL for zero-sized allocations.

* Remove comment regarding "mpt" driver where zfs_remove_max_segment
  is initialized to SPA_MAXBLOCKSIZE.

  Change zfs_condense_indirect_commit_entry_delay_ticks to
  zfs_condense_indirect_commit_entry_delay_ms for consistency with
  most other tunables in which delays are specified in ms.

* ZTS changes:

    Use set_tunable rather than mdb
    Use zpool sync as appropriate
    Use sync_pool instead of sync
    Kill jobs during test_removal_with_operation to allow unmount/export
    Don't add non-disk names such as "mirror" or "raidz" to $DISKS
    Use $TEST_BASE_DIR instead of /tmp
    Increase HZ from 100 to 1000 which is more common on Linux

    removal_multiple_indirection.ksh
        Reduce iterations in order to not time out on the code
        coverage builders.

    removal_resume_export:
        Functionally, the test case is correct but there exists a race
        where the kernel thread hasn't been fully started yet and is
        not visible.  Wait for up to 1 second for the removal thread
        to be started before giving up on it.  Also, increase the
        amount of data copied in order that the removal not finish
        before the export has a chance to fail.

* MMP compatibility, the concept of concrete versus non-concrete devices
  has slightly changed the semantics of vdev_writeable().  Update
  mmp_random_leaf_impl() accordingly.

* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
  feature which is not supported by OpenZFS.

* Added support for new vdev removal tracepoints.

* Test cases removal_with_zdb and removal_condense_export have been
  intentionally disabled.  When run manually they pass as intended,
  but when running in the automated test environment they produce
  unreliable results on the latest Fedora release.

  They may work better once the upstream pool import refectoring is
  merged into ZoL at which point they will be re-enabled.

Authored by: Matthew Ahrens <[email protected]>
Reviewed-by: Alex Reece <[email protected]>
Reviewed-by: George Wilson <[email protected]>
Reviewed-by: John Kennedy <[email protected]>
Reviewed-by: Prakash Surya <[email protected]>
Reviewed by: Richard Laager <[email protected]>
Reviewed by: Tim Chase <[email protected]>
Reviewed by: Brian Behlendorf <[email protected]>
Approved by: Garrett D'Amore <[email protected]>
Ported-by: Tim Chase <[email protected]>
Signed-off-by: Tim Chase <[email protected]>

OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900

34 files changed, 1755 insertions, 4 deletions

diff --git a/tests/zfs-tests/include/libtest.shlib b/tests/zfs-tests/include/libtest.shlib
index 327da2b9f..00326dcdc 100644
--- a/tests/zfs-tests/include/libtest.shlib
+++ b/tests/zfs-tests/include/libtest.shlib
@@ -2076,6 +2076,8 @@ function check_pool_status # pool token keyword <verbose>
 #	is_pool_scrubbed - to check if the pool is scrub completed
 #	is_pool_scrub_stopped - to check if the pool is scrub stopped
 #	is_pool_scrub_paused - to check if the pool has scrub paused
+#	is_pool_removing - to check if the pool is removing a vdev
+#	is_pool_removed - to check if the pool is remove completed
 #
 function is_pool_resilvering #pool <verbose>
 {
@@ -2113,6 +2115,18 @@ function is_pool_scrub_paused #pool <verbose>
 	return $?
 }
 
+function is_pool_removing #pool
+{
+	check_pool_status "$1" "remove" "in progress since "
+	return $?
+}
+
+function is_pool_removed #pool
+{
+	check_pool_status "$1" "remove" "completed on"
+	return $?
+}
+
 #
 # Use create_pool()/destroy_pool() to clean up the information in
 # in the given disk to avoid slice overlapping.
@@ -3422,3 +3436,46 @@ function get_tunable_impl
 
 	return 1
 }
+
+#
+# Prints the current time in seconds since UNIX Epoch.
+#
+function current_epoch
+{
+	printf '%(%s)T'
+}
+
+#
+# Get decimal value of global uint32_t variable using mdb.
+#
+function mdb_get_uint32
+{
+	typeset variable=$1
+	typeset value
+
+	value=$(mdb -k -e "$variable/X | ::eval .=U")
+	if [[ $? -ne 0 ]]; then
+		log_fail "Failed to get value of '$variable' from mdb."
+		return 1
+	fi
+
+	echo $value
+	return 0
+}
+
+#
+# Set global uint32_t variable to a decimal value using mdb.
+#
+function mdb_set_uint32
+{
+	typeset variable=$1
+	typeset value=$2
+
+	mdb -kw -e "$variable/W 0t$value" > /dev/null
+	if [[ $? -ne 0 ]]; then
+		echo "Failed to set '$variable' to '$value' in mdb."
+		return 1
+	fi
+
+	return 0
+}
diff --git a/tests/zfs-tests/tests/functional/Makefile.am b/tests/zfs-tests/tests/functional/Makefile.am
index 4510d5112..a5bbb36e0 100644
--- a/tests/zfs-tests/tests/functional/Makefile.am
+++ b/tests/zfs-tests/tests/functional/Makefile.am
@@ -49,6 +49,7 @@ SUBDIRS = \
 	redundancy \
 	refquota \
 	refreserv \
+	removal \
 	rename_dirs \
 	replacement \
 	reservation \
diff --git a/tests/zfs-tests/tests/functional/bootfs/bootfs_001_pos.ksh b/tests/zfs-tests/tests/functional/bootfs/bootfs_001_pos.ksh
index 3f60d9eed..3e9357063 100755
--- a/tests/zfs-tests/tests/functional/bootfs/bootfs_001_pos.ksh
+++ b/tests/zfs-tests/tests/functional/bootfs/bootfs_001_pos.ksh
@@ -31,6 +31,10 @@
 # Copyright (c) 2012, 2016 by Delphix. All rights reserved.
 #
 
+#
+# Copyright (c) 2012, 2015 by Delphix. All rights reserved.
+#
+
 . $STF_SUITE/include/libtest.shlib
 
 #
diff --git a/tests/zfs-tests/tests/functional/bootfs/bootfs_002_neg.ksh b/tests/zfs-tests/tests/functional/bootfs/bootfs_002_neg.ksh
index 8ee48dba5..6a72bfcdc 100755
--- a/tests/zfs-tests/tests/functional/bootfs/bootfs_002_neg.ksh
+++ b/tests/zfs-tests/tests/functional/bootfs/bootfs_002_neg.ksh
@@ -31,6 +31,10 @@
 # Copyright (c) 2012, 2016 by Delphix. All rights reserved.
 #
 
+#
+# Copyright (c) 2012, 2015 by Delphix. All rights reserved.
+#
+
 . $STF_SUITE/include/libtest.shlib
 
 #
diff --git a/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg b/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg
index d5791372d..2ea82f0f6 100644
--- a/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg
+++ b/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg
@@ -25,7 +25,7 @@
 #
 
 #
-# Copyright (c) 2013 by Delphix. All rights reserved.
+# Copyright (c) 2013, 2014 by Delphix. All rights reserved.
 # Copyright 2016 Nexenta Systems, Inc. All rights reserved.
 #
 
@@ -70,6 +70,8 @@ typeset -a properties=(
     "feature@sha512"
     "feature@skein"
     "feature@edonr"
+    "feature@device_removal"
+    "feature@obsolete_counts"
 )
 
 # Additional properties added for Linux.
diff --git a/tests/zfs-tests/tests/functional/features/async_destroy/async_destroy_001_pos.ksh b/tests/zfs-tests/tests/functional/features/async_destroy/async_destroy_001_pos.ksh
index a6ef964ac..20b61da92 100755
--- a/tests/zfs-tests/tests/functional/features/async_destroy/async_destroy_001_pos.ksh
+++ b/tests/zfs-tests/tests/functional/features/async_destroy/async_destroy_001_pos.ksh
@@ -49,21 +49,29 @@ verify_runnable "both"
 function cleanup
 {
 	datasetexists $TEST_FS && log_must zfs destroy $TEST_FS
+	log_must set_tunable64 zfs_async_block_max_blocks 100000
 }
 
 log_onexit cleanup
 log_assert "async_destroy can suspend and resume traversal"
 
-log_must zfs create -o recordsize=512 -o compression=off $TEST_FS
+log_must zfs create -o recordsize=1k -o compression=off $TEST_FS
 
-# Create enough blocks that it will take multiple TXGs to free them all.
+# Fill with 128,000 blocks.
 log_must dd bs=1024k count=128 if=/dev/zero of=/$TEST_FS/file
+
+#
+# Decrease the max blocks to free each txg, so that freeing takes
+# long enough that we can observe it.
+#
+log_must set_tunable64 zfs_async_block_max_blocks 100
+
 log_must sync
 log_must zfs destroy $TEST_FS
 
 #
 # We monitor the freeing property, to verify we can see blocks being
-# freed while the suspend/resume code is exerciesd.
+# freed while the suspend/resume code is exercised.
 #
 t0=$SECONDS
 count=0
@@ -75,6 +83,13 @@ done
 
 [[ $count -eq 0 ]] && log_fail "Freeing property remained empty"
 
+#
+# After a bit, go back to allowing an unlimited amount of freeing
+# per txg.
+#
+sleep 10
+log_must set_tunable64 zfs_async_block_max_blocks 100000
+
 # Wait for everything to be freed.
 while [[ "0" != "$(zpool list -Ho freeing $TESTPOOL)" ]]; do
 	[[ $((SECONDS - t0)) -gt 180 ]] && \
diff --git a/tests/zfs-tests/tests/functional/removal/Makefile.am b/tests/zfs-tests/tests/functional/removal/Makefile.am
new file mode 100644
index 000000000..eac82a2f1
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/Makefile.am
@@ -0,0 +1,32 @@
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+
+#
+# Copyright (c) 2014, 2015 by Delphix. All rights reserved.
+#
+
+pkgdatadir = $(datadir)/@PACKAGE@/zfs-tests/tests/functional/removal
+
+dist_pkgdata_SCRIPTS = \
+	cleanup.ksh removal_all_vdev.ksh removal_check_space.ksh \
+	removal_condense_export.ksh removal_multiple_indirection.ksh \
+	removal_remap_deadlists.ksh removal_remap.ksh \
+	removal_reservation.ksh removal_resume_export.ksh \
+	removal_sanity.ksh removal_with_add.ksh removal_with_create_fs.ksh \
+	removal_with_dedup.ksh removal_with_export.ksh \
+	removal_with_ganging.ksh removal_with_remap.ksh \
+	removal_with_remove.ksh removal_with_scrub.ksh \
+	removal_with_send.ksh removal_with_send_recv.ksh \
+	removal_with_snapshot.ksh removal_with_write.ksh \
+	removal_with_zdb.ksh remove_mirror.ksh remove_mirror_sanity.ksh \
+	remove_raidz.ksh removal.kshlib
+
+pkgexecdir = $(datadir)/@PACKAGE@/zfs-tests/tests/functional/removal
diff --git a/tests/zfs-tests/tests/functional/removal/cleanup.ksh b/tests/zfs-tests/tests/functional/removal/cleanup.ksh
new file mode 100755
index 000000000..352a9fe10
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/cleanup.ksh
@@ -0,0 +1,23 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+
+default_cleanup
diff --git a/tests/zfs-tests/tests/functional/removal/removal.kshlib b/tests/zfs-tests/tests/functional/removal/removal.kshlib
new file mode 100644
index 000000000..54a2fb3bd
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal.kshlib
@@ -0,0 +1,158 @@
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+export REMOVEDISK=${DISKS%% *}
+export NOTREMOVEDISK=${DISKS##* }
+
+#
+# Waits for the pool to finish a removal. If an optional callback is given,
+# execute it every 0.5s.
+#
+# Example usage:
+#
+#    wait_for_removal $TESTPOOL dd if=/dev/urandom of=/$TESTPOOL/file count=1
+#
+function wait_for_removal # pool [callback args]
+{
+	typeset pool=$1
+	typeset callback=$2
+
+	[[ -n $callback ]] && shift 2
+
+	while is_pool_removing $pool; do
+		[[ -z $callback ]] || log_must $callback "$@"
+		sleep 0.5
+	done
+
+	#
+	# The pool state changes before the TXG finishes syncing; wait for
+	# the removal to be completed on disk.
+	#
+	sync_pool
+
+	log_must is_pool_removed $pool
+	return 0
+}
+
+function indirect_vdev_mapping_size # pool
+{
+	typeset pool=$1
+	zdb -P $pool | grep 'indirect vdev' | \
+	    sed -E 's/.*\(([0-9]+) in memory\).*/\1/g'
+}
+
+function random_write # file write_size
+{
+	typeset file=$1
+	typeset block_size=$2
+	typeset file_size=$(stat -c%s $file 2>/dev/null)
+	typeset nblocks=$((file_size / block_size))
+
+	[[ -w $file ]] || return 1
+
+	dd if=/dev/urandom of=$file conv=notrunc \
+	    bs=$block_size count=1 seek=$((RANDOM % nblocks)) >/dev/null 2>&1
+}
+
+_test_removal_with_operation_count=0
+function _test_removal_with_operation_cb # real_callback
+{
+	typeset real_callback=$1
+
+	$real_callback $_test_removal_with_operation_count || \
+	    log_fail $real_callback "failed after" \
+		$_test_removal_with_operation_count "iterations"
+
+	(( _test_removal_with_operation_count++ ))
+
+	log_note "Callback called $((_test_removal_with_operation_count)) times"
+
+	return 0
+}
+
+function start_random_writer # file
+{
+	typeset file=$1
+	(
+		log_note "Starting writer for $file"
+		# This will fail when we destroy the pool.
+		while random_write $file $((2**12)); do
+			:
+		done
+		log_note "Stopping writer for $file"
+	) &
+}
+
+#
+# The callback should be a function that takes as input the number of
+# iterations and the given arguments.
+#
+function test_removal_with_operation # callback [count]
+{
+	typeset operation=$1
+	typeset count=$2
+
+	[[ -n $count ]] || count=0
+
+	#
+	# To ensure that the removal takes a while, we fragment the pool
+	# by writing random blocks and continue to do during the removal.
+	#
+	log_must mkfile 1g $TESTDIR/$TESTFILE0
+	for i in $(seq $((2**10))); do
+		random_write $TESTDIR/$TESTFILE0 $((2**12)) || \
+		    log_fail "Could not write to $TESTDIR/$TESTFILE0."
+	done
+	start_random_writer $TESTDIR/$TESTFILE0 1g
+	killpid=$!
+
+	log_must zpool remove $TESTPOOL $REMOVEDISK
+	log_must wait_for_removal $TESTPOOL \
+	    _test_removal_with_operation_cb $operation
+	log_mustnot vdevs_in_pool $TESTPOOL $REMOVEDISK
+	log_must zdb -cd $TESTPOOL
+
+	kill $killpid
+	wait
+
+	#
+	# We would love to assert that the callback happened *during* the
+	# removal, but we don't have the ability to be confident of that
+	# (via limiting bandwidth, etc.) yet. Instead, we try again.
+	#
+	if (( $_test_removal_with_operation_count <= 1 )); then
+		(( count <= 5 )) || log_fail "Attempted test too many times."
+
+		log_note "Callback only called" \
+		    $_test_removal_with_operation_count \
+		    "times, trying again."
+		default_setup_noexit "$DISKS"
+		test_removal_with_operation $operation $((count + 1))
+	fi
+}
+
+#
+# Kill the background job use by the test_removal_with_operation function.
+#
+function test_removal_with_operation_kill
+{
+	kill $killpid
+	wait $killpid
+	return 0
+}
diff --git a/tests/zfs-tests/tests/functional/removal/removal_all_vdev.ksh b/tests/zfs-tests/tests/functional/removal/removal_all_vdev.ksh
new file mode 100755
index 000000000..d3a594689
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_all_vdev.ksh
@@ -0,0 +1,39 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+default_setup_noexit "$DISKS"
+log_onexit default_cleanup_noexit
+
+for disk in $DISKS; do
+	if [[ "$disk" != "$REMOVEDISK" ]]; then
+		log_must zpool remove $TESTPOOL $disk
+		log_must wait_for_removal $TESTPOOL
+		log_mustnot vdevs_in_pool $TESTPOOL $disk
+	fi
+done
+
+log_must [ "x$(get_disklist $TESTPOOL)" = "x$REMOVEDISK" ]
+
+log_mustnot zpool remove $TESTPOOL $disk
+
+log_pass "Was not able to remove the last device in a pool."
diff --git a/tests/zfs-tests/tests/functional/removal/removal_check_space.ksh b/tests/zfs-tests/tests/functional/removal/removal_check_space.ksh
new file mode 100755
index 000000000..dec692ada
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_check_space.ksh
@@ -0,0 +1,44 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+TMPDIR=${TMPDIR:-$TEST_BASE_DIR}
+log_must mkfile $MINVDEVSIZE $TMPDIR/dsk1
+log_must mkfile $MINVDEVSIZE $TMPDIR/dsk2
+DISKS="$TMPDIR/dsk1 $TMPDIR/dsk2"
+REMOVEDISK=$TMPDIR/dsk1
+
+log_must default_setup_noexit "$DISKS"
+
+function cleanup
+{
+	default_cleanup_noexit
+	log_must rm -f $DISKS
+}
+log_onexit cleanup
+
+# Write a little more than half the pool.
+log_must dd if=/dev/urandom of=/$TESTDIR/$TESTFILE0 bs=$((2**20)) \
+    count=$((MINVDEVSIZE / (1024 * 1024)))
+log_mustnot zpool remove $TESTPOOL $TMPDIR/dsk1
+
+log_pass "Removal will not succeed if insufficient space."
diff --git a/tests/zfs-tests/tests/functional/removal/removal_condense_export.ksh b/tests/zfs-tests/tests/functional/removal/removal_condense_export.ksh
new file mode 100755
index 000000000..ad33caec8
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_condense_export.ksh
@@ -0,0 +1,90 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2015, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+if is_linux; then
+	log_unsupported "ZDB fails during concurrent pool activity."
+fi
+
+function reset
+{
+	log_must set_tunable64 zfs_condense_indirect_commit_entry_delay_ms 0
+	log_must set_tunable64 zfs_condense_min_mapping_bytes 131072
+	default_cleanup_noexit
+}
+
+default_setup_noexit "$DISKS" "true"
+log_onexit reset
+log_must set_tunable64 zfs_condense_indirect_commit_entry_delay_ms 1000
+log_must set_tunable64 zfs_condense_min_mapping_bytes 1
+
+log_must zfs set recordsize=512 $TESTPOOL/$TESTFS
+
+#
+# Create a large file so that we know some of the blocks will be on the
+# removed device, and hence eligible for remapping.
+#
+log_must dd if=/dev/urandom of=$TESTDIR/file bs=1024k count=10
+
+#
+# Create a file in the other filesystem, which will not be remapped.
+#
+log_must dd if=/dev/urandom of=$TESTDIR1/file bs=1024k count=10
+
+#
+# Randomly rewrite some of blocks in the file so that there will be holes and
+# we will not be able to remap the entire file in a few huge chunks.
+#
+for i in {1..4096}; do
+	#
+	# We have to sync periodically so that all the writes don't end up in
+	# the same txg. If they were all in the same txg, only the last write
+	# would go through and we would not have as many allocations to
+	# fragment the file.
+	#
+	((i % 100 > 0 )) || sync_pool $TESTPOOL || log_fail "Could not sync."
+        random_write $TESTDIR/file 512 || \
+            log_fail "Could not random write."
+done
+
+REMOVEDISKPATH=/dev
+case $REMOVEDISK in
+	/*)
+		REMOVEDISKPATH=$(dirname $REMOVEDISK)
+		;;
+esac
+
+log_must zpool remove $TESTPOOL $REMOVEDISK
+log_must wait_for_removal $TESTPOOL
+log_mustnot vdevs_in_pool $TESTPOOL $REMOVEDISK
+
+log_must zfs remap $TESTPOOL/$TESTFS
+sync_pool $TESTPOOL
+sleep 5
+sync_pool $TESTPOOL
+log_must zpool export $TESTPOOL
+zdb -e -p $REMOVEDISKPATH $TESTPOOL | grep 'Condensing indirect vdev' || \
+    log_fail "Did not export during a condense."
+log_must zdb -e -p $REMOVEDISKPATH -cudi $TESTPOOL
+log_must zpool import $TESTPOOL
+
+log_pass "Pool can be exported in the middle of a condense."
diff --git a/tests/zfs-tests/tests/functional/removal/removal_multiple_indirection.ksh b/tests/zfs-tests/tests/functional/removal/removal_multiple_indirection.ksh
new file mode 100755
index 000000000..1f71ec0e4
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_multiple_indirection.ksh
@@ -0,0 +1,93 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+#
+# DESCRIPTION:
+#
+# For device removal a file's contents should transfer
+# completely from one disk to another. That should remain
+# to be the case even if multiple levels of indirection
+# are introduced as we remove more and more devices.
+#
+# STRATEGY:
+#
+# 1. We create a file of size 128k and we save its contents
+#    in a local variable.
+# 2. We set the limit of the maximum copied segment size of
+#    removals to 32k, so during removal our 128k file will
+#    be split to 4 blocks.
+# 3. We start removing disks and adding them back in a loop.
+#    This way the file is moved around and introduces split
+#    blocks.
+# 4. The loop itself tests that we don't have any problem
+#    when removing many devices. Within the loop we test
+#    that the files contents remain the same across transfers.
+#
+
+TMPDIR=${TMPDIR:-$TEST_BASE_DIR}
+log_must mkfile $(($MINVDEVSIZE * 2)) $TMPDIR/dsk1
+log_must mkfile $(($MINVDEVSIZE * 2)) $TMPDIR/dsk2
+DISKS="$TMPDIR/dsk1 $TMPDIR/dsk2"
+REMOVEDISK=$TMPDIR/dsk1
+
+log_must default_setup_noexit "$DISKS"
+
+function cleanup
+{
+	default_cleanup_noexit
+	log_must rm -f $DISKS
+
+	# reset zfs_remove_max_segment to 1M
+	set_tunable32 zfs_remove_max_segment 1048576
+}
+
+log_onexit cleanup
+
+# set zfs_remove_max_segment to 32k
+log_must set_tunable32 zfs_remove_max_segment 32768
+
+log_must dd if=/dev/urandom of=$TESTDIR/$TESTFILE0 bs=128k count=1
+FILE_CONTENTS=`cat $TESTDIR/$TESTFILE0`
+log_must [ "x$(cat $TESTDIR/$TESTFILE0)" = "x$FILE_CONTENTS" ]
+
+for i in {1..10}; do
+	log_must zpool remove $TESTPOOL $TMPDIR/dsk1
+	log_must wait_for_removal $TESTPOOL
+	log_mustnot vdevs_in_pool $TESTPOOL $TMPDIR/dsk1
+	log_must zpool add $TESTPOOL $TMPDIR/dsk1
+
+	log_must zinject -a
+	log_must dd if=$TESTDIR/$TESTFILE0 of=/dev/null
+	log_must [ "x$(cat $TESTDIR/$TESTFILE0)" = "x$FILE_CONTENTS" ]
+
+	log_must zpool remove $TESTPOOL $TMPDIR/dsk2
+	log_must wait_for_removal $TESTPOOL
+	log_mustnot vdevs_in_pool $TESTPOOL $TMPDIR/dsk2
+	log_must zpool add $TESTPOOL $TMPDIR/dsk2
+
+	log_must zinject -a
+	log_must dd if=$TESTDIR/$TESTFILE0 of=/dev/null
+	log_must [ "x$(cat $TESTDIR/$TESTFILE0)" = "x$FILE_CONTENTS" ]
+done
+
+log_pass "File contents transferred completely from one disk to another."
diff --git a/tests/zfs-tests/tests/functional/removal/removal_remap.ksh b/tests/zfs-tests/tests/functional/removal/removal_remap.ksh
new file mode 100755
index 000000000..04d0c50e4
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_remap.ksh
@@ -0,0 +1,123 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2015, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+default_setup_noexit "$DISKS"
+
+
+function cleanup
+{
+	set_tunable64 zfs_condense_min_mapping_bytes 131072
+	default_cleanup_noexit
+}
+
+log_onexit cleanup
+
+log_must set_tunable64 zfs_condense_min_mapping_bytes 1
+
+log_must zfs set recordsize=512 $TESTPOOL/$TESTFS
+
+#
+# Create a large file so that we know some of the blocks will be on the
+# removed device, and hence eligible for remapping.
+#
+log_must dd if=/dev/urandom of=$TESTDIR/file bs=$((2**12)) count=$((2**9))
+
+#
+# Randomly rewrite some of blocks in the file so that there will be holes and
+# we will not be able to remap the entire file in a few huge chunks.
+#
+for i in $(seq $((2**12))); do
+	#
+	# We have to sync periodically so that all the writes don't end up in
+	# the same txg. If they were all in the same txg, only the last write
+	# would go through and we would not have as many allocations to
+	# fragment the file.
+	#
+	((i % 100 > 0 )) || sync_pool || log_fail "Could not sync."
+        random_write $TESTDIR/file $((2**9)) || \
+            log_fail "Could not random write."
+done
+
+#
+# Remap should quietly succeed as a noop before a removal.
+#
+log_must zfs remap $TESTPOOL/$TESTFS
+remaptxg_before=$(zfs get -H -o value remaptxg $TESTPOOL/$TESTFS)
+(( $? == 0 )) || log_fail "Could not get remaptxg."
+[[ $remaptxg_before == "-" ]] || \
+    log_fail "remaptxg ($remaptxg_before) had value before a removal"
+
+log_must zpool remove $TESTPOOL $REMOVEDISK
+log_must wait_for_removal $TESTPOOL
+log_mustnot vdevs_in_pool $TESTPOOL $REMOVEDISK
+
+#
+# remaptxg should not be set if we haven't done a remap.
+#
+remaptxg_before=$(zfs get -H -o value remaptxg $TESTPOOL/$TESTFS)
+(( $? == 0 )) || log_fail "Could not get remaptxg."
+[[ $remaptxg_before == "-" ]] || \
+    log_fail "remaptxg ($remaptxg_before) had value before a removal"
+
+mapping_size_before=$(indirect_vdev_mapping_size $TESTPOOL)
+log_must zfs remap $TESTPOOL/$TESTFS
+
+# Try to wait for a condense to finish.
+for i in {1..5}; do
+	sleep 5
+	sync_pool
+done
+mapping_size_after=$(indirect_vdev_mapping_size $TESTPOOL)
+
+#
+# After the remap, there should not be very many blocks referenced. The reason
+# why our threshold is as high as 512 is because our ratio of metadata to
+# user data is relatively high, with only 64M of user data on the file system.
+#
+(( mapping_size_after < mapping_size_before )) || \
+    log_fail "Mapping size did not decrease after remap: " \
+    "$mapping_size_before before to $mapping_size_after after."
+(( mapping_size_after < 512 )) || \
+    log_fail "Mapping size not small enough after remap: " \
+    "$mapping_size_before before to $mapping_size_after after."
+
+#
+# After a remap, the remaptxg should be set to a non-zero value.
+#
+remaptxg_after=$(zfs get -H -o value remaptxg $TESTPOOL/$TESTFS)
+(( $? == 0 )) || log_fail "Could not get remaptxg."
+log_note "remap txg after remap is $remaptxg_after"
+(( remaptxg_after > 0 )) || log_fail "remaptxg not increased"
+
+#
+# Remap should quietly succeed as a noop if there have been no removals since
+# the last remap.
+#
+log_must zfs remap $TESTPOOL/$TESTFS
+remaptxg_again=$(zfs get -H -o value remaptxg $TESTPOOL/$TESTFS)
+(( $? == 0 )) || log_fail "Could not get remaptxg."
+log_note "remap txg after second remap is $remaptxg_again"
+(( remaptxg_again == remaptxg_after )) || \
+    log_fail "remap not noop if there has been no removal"
+
+log_pass "Remapping a fs caused mapping size to decrease."
diff --git a/tests/zfs-tests/tests/functional/removal/removal_remap_deadlists.ksh b/tests/zfs-tests/tests/functional/removal/removal_remap_deadlists.ksh
new file mode 100755
index 000000000..5b5be66b3
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_remap_deadlists.ksh
@@ -0,0 +1,77 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2015, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+default_setup_noexit "$DISKS"
+log_onexit default_cleanup_noexit
+
+log_must dd if=/dev/zero of=$TESTDIR/file bs=1024k count=300
+
+log_must zfs snapshot $TESTPOOL/$TESTFS@snap-pre1
+log_must dd if=/dev/zero of=$TESTDIR/file bs=1024k count=100 \
+    conv=notrunc seek=100
+
+log_must zfs snapshot $TESTPOOL/$TESTFS@snap-pre2
+log_must dd if=/dev/zero of=$TESTDIR/file bs=1024k count=100 \
+    conv=notrunc seek=200
+
+log_must zpool remove $TESTPOOL $REMOVEDISK
+if is_linux; then
+	log_must wait_for_removal $TESTPOOL
+else
+	log_must wait_for_removal $TESTPOOL zdb -cd $TESTPOOL
+fi
+log_mustnot vdevs_in_pool $TESTPOOL $REMOVEDISK
+log_must zdb -cd $TESTPOOL
+
+log_must zfs remap $TESTPOOL/$TESTFS
+log_must zdb -cd $TESTPOOL
+
+log_must zfs snapshot $TESTPOOL/$TESTFS@snap-post3
+log_must zdb -cd $TESTPOOL
+
+log_must zfs snapshot $TESTPOOL/$TESTFS@snap-post4
+log_must zdb -cd $TESTPOOL
+
+#
+# Test case where block is moved from remap deadlist: blocks born before
+# snap-pre2 will be obsoleted.
+#
+log_must zfs destroy $TESTPOOL/$TESTFS@snap-pre2
+log_must zdb -cd $TESTPOOL
+
+#
+# Test case where we merge remap deadlists: blocks before snap-pre1 will
+# need to go on snap-post4's deadlist.
+#
+log_must zfs destroy $TESTPOOL/$TESTFS@snap-post3
+log_must zdb -cd $TESTPOOL
+
+log_must zfs destroy $TESTPOOL/$TESTFS@snap-post4
+
+#
+# Test rollback.
+#
+log_must zfs rollback $TESTPOOL/$TESTFS@snap-pre1
+log_must zfs destroy $TESTPOOL/$TESTFS@snap-pre1
+
+log_pass "Remove and remap works with snapshots and deadlists."
diff --git a/tests/zfs-tests/tests/functional/removal/removal_reservation.ksh b/tests/zfs-tests/tests/functional/removal/removal_reservation.ksh
new file mode 100755
index 000000000..b57f1777c
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_reservation.ksh
@@ -0,0 +1,68 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+TMPDIR=${TMPDIR:-$TEST_BASE_DIR}
+log_must mkfile 1g $TMPDIR/dsk1
+log_must mkfile 1g $TMPDIR/dsk2
+DISKS="$TMPDIR/dsk1 $TMPDIR/dsk2"
+REMOVEDISK=$TMPDIR/dsk1
+
+default_setup_noexit "$DISKS"
+
+function cleanup
+{
+	default_cleanup_noexit
+	log_must rm -f $DISKS
+}
+
+log_onexit cleanup
+
+log_must zfs set compression=off $TESTPOOL/$TESTFS
+
+# Write a little under half the pool.
+log_must file_write -o create -f $TESTDIR/$TESTFILE1 -b $((2**20)) -c $((2**9))
+
+#
+# Start a writing thread to ensure the removal will take a while.
+# This will automatically die when we destroy the pool.
+#
+start_random_writer $TESTDIR/$TESTFILE1
+
+callback_count=0
+function callback
+{
+	(( callback_count++ ))
+	(( callback_count == 1 )) || return 0
+
+	# Attempt to write more than the new pool will be able to handle.
+	file_write -o create -f $TESTDIR/$TESTFILE2 -b $((2**20)) -c $((2**9))
+	zret=$?
+	ENOSPC=28
+	log_note "file_write returned $zret"
+	(( $zret == $ENOSPC )) || log_fail "Did not get ENOSPC during removal."
+}
+
+log_must zpool remove $TESTPOOL $REMOVEDISK
+log_must wait_for_removal $TESTPOOL callback
+
+log_pass "Removal properly sets reservation."
diff --git a/tests/zfs-tests/tests/functional/removal/removal_resume_export.ksh b/tests/zfs-tests/tests/functional/removal/removal_resume_export.ksh
new file mode 100755
index 000000000..5cecfdb5d
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_resume_export.ksh
@@ -0,0 +1,134 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2017 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+#
+# DESCRIPTION:
+#
+# When a pool has an ongoing removal and it is exported ZFS
+# suspends the removal thread beforehand. This test ensures
+# that ZFS restarts the removal thread if the export fails
+# for some reason.
+#
+# STRATEGY:
+#
+# 1. Create a pool with one vdev and do some writes on it.
+# 2. Add a new vdev to the pool and start the removal of
+#    the first vdev.
+# 3. Inject a fault in the pool and attempt to export (it
+#    should fail).
+# 4. After the export fails ensure that the removal thread
+#    was restarted (i.e. the svr_thread field in the spa
+#    should be non-zero).
+#
+
+
+function cleanup
+{
+	log_must zinject -c all
+	default_cleanup_noexit
+}
+
+function ensure_thread_running # spa_address
+{
+	if is_linux; then
+		typeset TRIES=0
+		typeset THREAD_PID
+		while [[ $TRIES -lt 10 ]]; do
+			THREAD_PID=$(pgrep spa_vdev_remove)
+			[[ "$THREAD_PID" ]] && break
+			sleep 0.1
+			(( TRIES = TRIES + 1 ))
+		done
+		[[ "$THREAD_PID" ]] || \
+		    log_fail "removal thread is not running TRIES=$TRIES THREAD_PID=$THREAD_PID"
+	else
+		#
+		# Try to get the address of the removal thread.
+		#
+		typeset THREAD_ADDR=$(mdb -ke "$1::print \
+		    spa_t spa_vdev_removal->svr_thread" | awk "{print \$3}")
+
+		#
+		# if address is NULL it means that the thread is
+		# not running.
+		#
+		[[ "$THREAD_ADDR" = 0 ]] && \
+		    log_fail "removal thread is not running"
+	fi
+
+	return 0
+}
+
+log_onexit cleanup
+
+#
+# Create pool with one disk.
+#
+log_must default_setup_noexit "$REMOVEDISK"
+
+#
+# Save address of SPA in memory so you can check with mdb
+# if the removal thread is running.
+#
+is_linux || typeset SPA_ADDR=$(mdb -ke "::spa" | awk "/$TESTPOOL/ {print \$1}")
+
+#
+# Turn off compression to raise capacity as much as possible
+# for the little time that this test runs.
+#
+log_must zfs set compression=off $TESTPOOL/$TESTFS
+
+#
+# Write some data that will be evacuated from the device when
+# we start the removal.
+#
+log_must dd if=/dev/urandom of=$TESTDIR/$TESTFILE0 bs=64M count=32
+
+#
+# Add second device where all the data will be evacuated.
+#
+log_must zpool add -f $TESTPOOL $NOTREMOVEDISK
+
+#
+# Start removal.
+#
+log_must zpool remove $TESTPOOL $REMOVEDISK
+
+#
+# Inject an error so export fails after having just suspended
+# the removal thread. [spa_inject_ref gets incremented]
+#
+log_must zinject -d $REMOVEDISK -D 10:1 $TESTPOOL
+
+log_must ensure_thread_running $SPA_ADDR
+
+#
+# Because of the above error export should fail.
+#
+log_mustnot zpool export $TESTPOOL
+
+log_must ensure_thread_running $SPA_ADDR
+
+wait_for_removal $TESTPOOL
+
+log_pass "Device removal thread resumes after failed export"
diff --git a/tests/zfs-tests/tests/functional/removal/removal_sanity.ksh b/tests/zfs-tests/tests/functional/removal/removal_sanity.ksh
new file mode 100755
index 000000000..f7f69c7d2
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_sanity.ksh
@@ -0,0 +1,39 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+default_setup_noexit "$DISKS"
+log_onexit default_cleanup_noexit
+
+FILE_CONTENTS="Leeloo Dallas mul-ti-pass."
+
+echo $FILE_CONTENTS  >$TESTDIR/$TESTFILE0
+log_must [ "x$(cat $TESTDIR/$TESTFILE0)" = "x$FILE_CONTENTS" ]
+
+log_must zpool remove $TESTPOOL $REMOVEDISK
+log_must wait_for_removal $TESTPOOL
+log_mustnot vdevs_in_pool $TESTPOOL $REMOVEDISK
+
+log_must dd if=/$TESTDIR/$TESTFILE0 of=/dev/null
+log_must [ "x$(cat $TESTDIR/$TESTFILE0)" = "x$FILE_CONTENTS" ]
+
+log_pass "Removed device not in use after removal."
diff --git a/tests/zfs-tests/tests/functional/removal/removal_with_add.ksh b/tests/zfs-tests/tests/functional/removal/removal_with_add.ksh
new file mode 100755
index 000000000..e719a5ecc
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_with_add.ksh
@@ -0,0 +1,52 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+TMPDIR=${TMPDIR:-$TEST_BASE_DIR}
+log_must mkfile 1g $TMPDIR/dsk1
+log_must mkfile 1g $TMPDIR/dsk2
+
+function cleanup
+{
+	default_cleanup_noexit
+	log_must rm -f $TMPDIR/dsk1 $TMPDIR/dsk2
+}
+
+default_setup_noexit "$DISKS"
+log_onexit cleanup
+
+function callback
+{
+	typeset count=$1
+	if ((count == 0)); then
+		log_mustnot zpool attach -f $TESTPOOL $TMPDIR/dsk1 $TMPDIR/dsk2
+		log_mustnot zpool add -f $TESTPOOL \
+		    raidz $TMPDIR/dsk1 $TMPDIR/dsk2
+		log_must zpool add -f $TESTPOOL $TMPDIR/dsk1
+	fi
+
+	return 0
+}
+
+test_removal_with_operation callback
+
+log_pass "Removal can only add normal disks."
diff --git a/tests/zfs-tests/tests/functional/removal/removal_with_create_fs.ksh b/tests/zfs-tests/tests/functional/removal/removal_with_create_fs.ksh
new file mode 100755
index 000000000..403428290
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_with_create_fs.ksh
@@ -0,0 +1,39 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+default_setup_noexit "$DISKS"
+log_onexit default_cleanup_noexit
+
+function callback
+{
+	typeset count=$1
+	if ((count == 0)); then
+		log_must zfs create $TESTPOOL/$TESTFS1
+		log_must zfs destroy $TESTPOOL/$TESTFS1
+	fi
+	return 0
+}
+
+test_removal_with_operation callback
+
+log_pass "Can write to device during removal"
diff --git a/tests/zfs-tests/tests/functional/removal/removal_with_dedup.ksh b/tests/zfs-tests/tests/functional/removal/removal_with_dedup.ksh
new file mode 100755
index 000000000..4b6c94a80
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_with_dedup.ksh
@@ -0,0 +1,49 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+default_setup_noexit "$DISKS"
+log_onexit default_cleanup_noexit
+
+log_must zfs set dedup=on $TESTPOOL
+
+FILE_CONTENTS="Leeloo Dallas mul-ti-pass."
+
+echo $FILE_CONTENTS  >$TESTDIR/$TESTFILE0
+echo $FILE_CONTENTS  >$TESTDIR/$TESTFILE1
+echo $FILE_CONTENTS  >$TESTDIR/$TESTFILE2
+log_must [ "x$(cat $TESTDIR/$TESTFILE0)" = "x$FILE_CONTENTS" ]
+log_must [ "x$(cat $TESTDIR/$TESTFILE1)" = "x$FILE_CONTENTS" ]
+log_must [ "x$(cat $TESTDIR/$TESTFILE2)" = "x$FILE_CONTENTS" ]
+
+log_must zpool remove $TESTPOOL $REMOVEDISK
+log_must wait_for_removal $TESTPOOL
+log_mustnot vdevs_in_pool $TESTPOOL $REMOVEDISK
+
+log_must dd if=/$TESTDIR/$TESTFILE0 of=/dev/null
+log_must dd if=/$TESTDIR/$TESTFILE1 of=/dev/null
+log_must dd if=/$TESTDIR/$TESTFILE2 of=/dev/null
+log_must [ "x$(cat $TESTDIR/$TESTFILE0)" = "x$FILE_CONTENTS" ]
+log_must [ "x$(cat $TESTDIR/$TESTFILE1)" = "x$FILE_CONTENTS" ]
+log_must [ "x$(cat $TESTDIR/$TESTFILE2)" = "x$FILE_CONTENTS" ]
+
+log_pass "Removed device not in use after removal."
diff --git a/tests/zfs-tests/tests/functional/removal/removal_with_export.ksh b/tests/zfs-tests/tests/functional/removal/removal_with_export.ksh
new file mode 100755
index 000000000..38d6d53d4
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_with_export.ksh
@@ -0,0 +1,50 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+default_setup_noexit "$DISKS"
+log_onexit default_cleanup_noexit
+
+function callback # count
+{
+	typeset count=$1
+	if ((count == 0)); then
+		is_linux && test_removal_with_operation_kill
+		log_must zpool export $TESTPOOL
+
+		#
+		# We are concurrently starting dd processes that will
+		# create files in $TESTDIR.  These could cause the import
+		# to fail because it can't mount on the filesystem on a
+		# non-empty directory.  Therefore, remove the directory
+		# so that the dd process will fail.
+		#
+		log_must rm -rf $TESTDIR
+
+		log_must zpool import $TESTPOOL
+	fi
+	return 0
+}
+
+test_removal_with_operation callback
+
+log_pass "Can export and import pool during removal"
diff --git a/tests/zfs-tests/tests/functional/removal/removal_with_ganging.ksh b/tests/zfs-tests/tests/functional/removal/removal_with_ganging.ksh
new file mode 100755
index 000000000..cfbca89bd
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_with_ganging.ksh
@@ -0,0 +1,47 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+function cleanup
+{
+	log_must set_tunable64 metaslab_gang_bang $((2**17 + 1))
+	default_cleanup_noexit
+}
+
+default_setup_noexit "$DISKS"
+log_must set_tunable64 metaslab_gang_bang $((2**12))
+log_onexit cleanup
+
+FILE_CONTENTS="Leeloo Dallas mul-ti-pass."
+
+echo $FILE_CONTENTS  >$TESTDIR/$TESTFILE0
+log_must [ "x$(cat $TESTDIR/$TESTFILE0)" = "x$FILE_CONTENTS" ]
+log_must file_write -o create -f $TESTDIR/$TESTFILE1 -b $((2**20)) -c $((2**7))
+
+log_must zpool remove $TESTPOOL $REMOVEDISK
+log_must wait_for_removal $TESTPOOL
+log_mustnot vdevs_in_pool $TESTPOOL $REMOVEDISK
+
+log_must dd if=/$TESTDIR/$TESTFILE0 of=/dev/null
+log_must [ "x$(cat $TESTDIR/$TESTFILE0)" = "x$FILE_CONTENTS" ]
+
+log_pass "Removed device not in use after removal."
diff --git a/tests/zfs-tests/tests/functional/removal/removal_with_remap.ksh b/tests/zfs-tests/tests/functional/removal/removal_with_remap.ksh
new file mode 100755
index 000000000..63050a647
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_with_remap.ksh
@@ -0,0 +1,38 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2015, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+default_setup_noexit "$DISKS"
+log_onexit default_cleanup_noexit
+
+function callback
+{
+	typeset count=$1
+	if ((count == 0)); then
+		zfs remap $TESTPOOL/$TESTFS
+	fi
+	return 0
+}
+
+test_removal_with_operation callback
+
+log_pass "Can remap a filesystem during removal"
diff --git a/tests/zfs-tests/tests/functional/removal/removal_with_remove.ksh b/tests/zfs-tests/tests/functional/removal/removal_with_remove.ksh
new file mode 100755
index 000000000..fef7c293b
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_with_remove.ksh
@@ -0,0 +1,38 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+default_setup_noexit "$DISKS"
+log_onexit default_cleanup_noexit
+
+function callback # count
+{
+	typeset count=$1
+	if ((count == 0)); then
+		log_mustnot zpool remove $TESTPOOL $NOTREMOVEDISK
+	fi
+	return 0
+}
+
+test_removal_with_operation callback
+
+log_pass "Cannot remove a disk during removal"
diff --git a/tests/zfs-tests/tests/functional/removal/removal_with_scrub.ksh b/tests/zfs-tests/tests/functional/removal/removal_with_scrub.ksh
new file mode 100755
index 000000000..33eb41bf2
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_with_scrub.ksh
@@ -0,0 +1,38 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+default_setup_noexit "$DISKS"
+log_onexit default_cleanup_noexit
+
+function callback
+{
+	typeset count=$1
+	if ((count == 0)); then
+		log_must zpool scrub $TESTPOOL
+	fi
+	return 0
+}
+
+test_removal_with_operation callback
+
+log_pass "Can use scrub during removal"
diff --git a/tests/zfs-tests/tests/functional/removal/removal_with_send.ksh b/tests/zfs-tests/tests/functional/removal/removal_with_send.ksh
new file mode 100755
index 000000000..c5a92505c
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_with_send.ksh
@@ -0,0 +1,40 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+default_setup_noexit "$DISKS"
+log_onexit default_cleanup_noexit
+
+function callback
+{
+	typeset count=$1
+	if ((count == 0)); then
+		create_snapshot $TESTPOOL/$TESTFS $TESTSNAP
+		log_must ksh -c \
+		    "zfs send $TESTPOOL/$TESTFS@$TESTSNAP >/dev/null"
+	fi
+	return 0
+}
+
+test_removal_with_operation callback
+
+log_pass "Can use send during removal"
diff --git a/tests/zfs-tests/tests/functional/removal/removal_with_send_recv.ksh b/tests/zfs-tests/tests/functional/removal/removal_with_send_recv.ksh
new file mode 100755
index 000000000..c7d1c8a89
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_with_send_recv.ksh
@@ -0,0 +1,41 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+default_setup_noexit "$DISKS"
+log_onexit default_cleanup_noexit
+
+function callback
+{
+	typeset count=$1
+	if ((count == 0)); then
+		create_snapshot $TESTPOOL/$TESTFS $TESTSNAP
+		log_must ksh -o pipefail -c \
+		    "zfs send $TESTPOOL/$TESTFS@$TESTSNAP | \
+		    zfs recv $TESTPOOL/$TESTFS1"
+	fi
+	return 0
+}
+
+test_removal_with_operation callback
+
+log_pass "Can send and recv during removal"
diff --git a/tests/zfs-tests/tests/functional/removal/removal_with_snapshot.ksh b/tests/zfs-tests/tests/functional/removal/removal_with_snapshot.ksh
new file mode 100755
index 000000000..7fe36a94f
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_with_snapshot.ksh
@@ -0,0 +1,39 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+default_setup_noexit "$DISKS"
+log_onexit default_cleanup_noexit
+
+function callback
+{
+	typeset count=$1
+	if ((count == 0)); then
+		create_snapshot $TESTPOOL/$TESTFS $TESTSNAP
+		destroy_snapshot $TESTPOOL/$TESTFS@$TESTSNAP
+	fi
+	return 0
+}
+
+test_removal_with_operation callback
+
+log_pass "Can create and destroy snapshot during removal"
diff --git a/tests/zfs-tests/tests/functional/removal/removal_with_write.ksh b/tests/zfs-tests/tests/functional/removal/removal_with_write.ksh
new file mode 100755
index 000000000..5d37b903e
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_with_write.ksh
@@ -0,0 +1,34 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+default_setup_noexit "$DISKS"
+log_onexit default_cleanup_noexit
+
+function callback
+{
+	return 0
+}
+
+test_removal_with_operation callback
+
+log_pass "Can write to device during removal"
diff --git a/tests/zfs-tests/tests/functional/removal/removal_with_zdb.ksh b/tests/zfs-tests/tests/functional/removal/removal_with_zdb.ksh
new file mode 100755
index 000000000..1f609273c
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/removal_with_zdb.ksh
@@ -0,0 +1,78 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+zdbout=${TMPDIR:-$TEST_BASE_DIR}/zdbout.$$
+
+if is_linux; then
+	log_unsupported "ZDB fails during concurrent pool activity."
+fi
+
+function cleanup
+{
+	default_cleanup_noexit
+	log_must rm -f $zdbout
+}
+
+default_setup_noexit "$DISKS"
+log_onexit cleanup
+FIRSTDISK=${DISKS%% *}
+
+DISKPATH=/dev
+case $FIRSTDISK in
+	/*)
+		DISKPATH=$(dirname $FIRSTDISK)
+		;;
+esac
+
+function callback
+{
+	typeset count=$1
+	typeset zdbstat
+
+	log_must zpool set cachefile=none $TESTPOOL
+	zdb -e -p $DISKPATH -cudi $TESTPOOL >$zdbout 2>&1
+	zdbstat=$?
+	log_must zpool set cachefile= $TESTPOOL
+	if [[ $zdbstat != 0 ]]; then
+		log_note "Output: zdb -e -p $DISKPATH -cudi $TESTPOOL"
+		cat $zdbout
+		log_note "zdb detected errors with exist status $zdbstat."
+	fi
+	log_note "zdb -e -p $DISKPATH -cudi $TESTPOOL >zdbout 2>&1"
+	return 0
+}
+
+test_removal_with_operation callback
+
+log_must zpool set cachefile=none $TESTPOOL
+zdb -e -p $DISKPATH -cudi $TESTPOOL >$zdbout 2>&1
+zdbstat=$?
+log_must zpool set cachefile= $TESTPOOL
+if [[ $zdbstat != 0 ]]; then
+	log_note "Output following removal: zdb -cudi $TESTPOOL"
+	cat $zdbout
+	log_fail "zdb detected errors with exit status " \
+	    "$zdbstat following removal."
+fi
+
+log_pass "Can use zdb during removal"
diff --git a/tests/zfs-tests/tests/functional/removal/remove_mirror.ksh b/tests/zfs-tests/tests/functional/removal/remove_mirror.ksh
new file mode 100755
index 000000000..06335b703
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/remove_mirror.ksh
@@ -0,0 +1,57 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2018 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+TMPDIR=${TMPDIR:-$TEST_BASE_DIR}
+
+DISK1="$TMPDIR/dsk1"
+DISK2="$TMPDIR/dsk2"
+DISK3="$TMPDIR/dsk3"
+DISKS="$DISK1 $DISK2 $DISK3"
+
+log_must mkfile $(($MINVDEVSIZE * 2)) $DISK1
+log_must mkfile $(($MINVDEVSIZE * 2)) $DISK2
+log_must mkfile $(($MINVDEVSIZE * 2)) $DISK3
+
+function cleanup
+{
+	default_cleanup_noexit
+	log_must rm -f $DISKS
+}
+
+log_must default_setup_noexit "$DISK1 mirror $DISK2 $DISK3"
+log_onexit cleanup
+
+# Attempt to remove the non mirrored disk.
+log_must zpool remove $TESTPOOL $DISK1
+
+# Attempt to remove one of the disks in the mirror.
+log_mustnot zpool remove $TESTPOOL $DISK2
+log_must wait_for_removal $TESTPOOL
+
+# Add back the first disk.
+log_must zpool add $TESTPOOL $DISK1
+
+# Now attempt to remove the mirror.
+log_must zpool remove $TESTPOOL mirror-1
+
+log_pass "Removal will succeed for top level vdev(s)."
diff --git a/tests/zfs-tests/tests/functional/removal/remove_mirror_sanity.ksh b/tests/zfs-tests/tests/functional/removal/remove_mirror_sanity.ksh
new file mode 100755
index 000000000..fa3fb5de7
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/remove_mirror_sanity.ksh
@@ -0,0 +1,58 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2017 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+DISK1=$(echo $DISKS | awk '{print $1}')
+DISK2=$(echo $DISKS | awk '{print $2}')
+DISK3=$(echo $DISKS | awk '{print $3}')
+DISKS="$DISK1 $DISK2 $DISK3"
+
+log_must default_setup_noexit "$DISK1 mirror $DISK2 $DISK3"
+log_onexit default_cleanup_noexit
+
+WORDS_FILE1="/usr/dict/words"
+WORDS_FILE2="/usr/share/dict/words"
+FILE_CONTENTS="Leeloo Dallas mul-ti-pass."
+
+if [[ -f $WORDS_FILE1 ]]; then
+    log_must cp $WORDS_FILE1 $TESTDIR
+elif [[ -f $WORDS_FILE2 ]]; then
+    log_must cp $WORDS_FILE2 $TESTDIR
+else
+    echo $FILE_CONTENTS  >$TESTDIR/$TESTFILE0
+    log_must [ "x$(cat $TESTDIR/$TESTFILE0)" = "x$FILE_CONTENTS" ]
+fi
+
+log_must zpool remove $TESTPOOL mirror-1
+log_must wait_for_removal $TESTPOOL
+log_mustnot vdevs_in_pool $TESTPOOL mirror-1
+
+if [[ -f $WORDS_FILE1 ]]; then
+    log_must diff $WORDS_FILE1 $TESTDIR/words
+elif [[ -f $WORDS_FILE2 ]]; then
+    log_must diff $WORDS_FILE2 $TESTDIR/words
+else
+    log_must dd if=/$TESTDIR/$TESTFILE0 of=/dev/null
+    log_must [ "x$(cat $TESTDIR/$TESTFILE0)" = "x$FILE_CONTENTS" ]
+fi
+
+log_pass "Removed top-level mirror device not in use after removal."
diff --git a/tests/zfs-tests/tests/functional/removal/remove_raidz.ksh b/tests/zfs-tests/tests/functional/removal/remove_raidz.ksh
new file mode 100755
index 000000000..98d4536a1
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/removal/remove_raidz.ksh
@@ -0,0 +1,50 @@
+#! /bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/removal/removal.kshlib
+
+TMPDIR=${TMPDIR:-$TEST_BASE_DIR}
+log_must mkfile $MINVDEVSIZE $TMPDIR/dsk1
+log_must mkfile $MINVDEVSIZE $TMPDIR/dsk2
+log_must mkfile $MINVDEVSIZE $TMPDIR/dsk3
+DISKS1="$TMPDIR/dsk1"
+DISKS2="$TMPDIR/dsk2 $TMPDIR/dsk3"
+DISKS="$DISKS1 $DISKS2"
+
+function cleanup
+{
+	default_cleanup_noexit
+	log_must rm -f $DISKS
+}
+
+log_must default_setup_noexit "$DISKS1 raidz $DISKS2"
+log_onexit cleanup
+
+# Attempt to remove the non raidz disk.
+log_mustnot zpool remove $TESTPOOL $TMPDIR/dsk1
+
+# Attempt to remove one of the raidz disks.
+log_mustnot zpool remove $TESTPOOL $TMPDIR/dsk2
+
+# Attempt to remove the raidz.
+log_mustnot zpool remove $TESTPOOL raidz1-1
+
+log_pass "Removal will not succeed if there is a top level mirror."