Multi-modifier protection (MMP)

Add multihost=on|off pool property to control MMP.  When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp.  Property defaults to off.

During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock.  Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".

Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval.  The period is specified in milliseconds.  The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.

Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path.  Abbreviated
output below.

$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg   timestamp  mmp_delay   vdev_guid   vdev_label vdev_path
20468    261337  250274925   68396651780       3    /dev/sda
20468    261339  252023374   6267402363293     1    /dev/sdc
20468    261340  252000858   6698080955233     1    /dev/sdx
20468    261341  251980635   783892869810      2    /dev/sdy
20468    261342  253385953   8923255792467     3    /dev/sdd
20468    261344  253336622   042125143176      0    /dev/sdab
20468    261345  253310522   1200778101278     2    /dev/sde
20468    261346  253286429   0950576198362     2    /dev/sdt
20468    261347  253261545   96209817917       3    /dev/sds
20468    261349  253238188   8555725937673     3    /dev/sdb

Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.

When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test.  For example, the
pool is exported to run zdb and then imported again.  Add a new ztest
function, "-M", to alter ztest behavior to prevent this.

Add new tests to verify the new functionality.  Tests provided by
Giuseppe Di Natale.

Reviewed by: Matthew Ahrens <[email protected]>
Reviewed-by: Giuseppe Di Natale <[email protected]>
Reviewed-by: Ned Bass <[email protected]>
Reviewed-by: Andreas Dilger <[email protected]>
Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: Olaf Faaland <[email protected]>
Closes #745
Closes #6279

18 files changed, 991 insertions, 1 deletions

diff --git a/tests/zfs-tests/callbacks/Makefile.am b/tests/zfs-tests/callbacks/Makefile.am
index 71947f308..30e847241 100644
--- a/tests/zfs-tests/callbacks/Makefile.am
+++ b/tests/zfs-tests/callbacks/Makefile.am
@@ -1,4 +1,5 @@
 pkgdatadir = $(datadir)/@PACKAGE@/zfs-tests/callbacks
 dist_pkgdata_SCRIPTS = \
 	zfs_dbgmsg.ksh \
-	zfs_dmesg.ksh
+	zfs_dmesg.ksh \
+	zfs_mmp.ksh
diff --git a/tests/zfs-tests/callbacks/zfs_mmp.ksh b/tests/zfs-tests/callbacks/zfs_mmp.ksh
new file mode 100755
index 000000000..df2cd132d
--- /dev/null
+++ b/tests/zfs-tests/callbacks/zfs_mmp.ksh
@@ -0,0 +1,37 @@
+#!/bin/ksh -p
+
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+
+#
+# Copyright (c) 2017 by Lawrence Livermore National Security.
+# All rights reserved.
+#
+
+# $1: number of lines to output (default: 40)
+typeset lines=${1:-40}
+typeset history=$(cat /sys/module/zfs/parameters/zfs_multihost_history)
+
+if [ $history -eq 0 ]; then
+	exit
+fi
+
+for f in /proc/spl/kstat/zfs/*/multihost; do
+	echo "================================================================="
+	echo " Last $lines lines of $f"
+	echo "================================================================="
+
+	sudo tail -n $lines $f
+done
+
+echo "================================================================="
+echo " End of zfs multihost log"
+echo "================================================================="
diff --git a/tests/zfs-tests/include/commands.cfg b/tests/zfs-tests/include/commands.cfg
index 968ab3cd1..57b1bd315 100644
--- a/tests/zfs-tests/include/commands.cfg
+++ b/tests/zfs-tests/include/commands.cfg
@@ -52,6 +52,7 @@ export SYSTEM_FILES='arp
     gunzip
     gzip
     head
+    hostid
     hostname
     id
     iostat
diff --git a/tests/zfs-tests/include/libtest.shlib b/tests/zfs-tests/include/libtest.shlib
index ddfe550bf..1d1c57e7a 100644
--- a/tests/zfs-tests/include/libtest.shlib
+++ b/tests/zfs-tests/include/libtest.shlib
@@ -3428,3 +3428,83 @@ function swap_cleanup
 
 	return 0
 }
+
+#
+# Set a global system tunable (64-bit value)
+#
+# $1 tunable name
+# $2 tunable values
+#
+function set_tunable64
+{
+	set_tunable_impl "$1" "$2" Z
+}
+
+#
+# Set a global system tunable (32-bit value)
+#
+# $1 tunable name
+# $2 tunable values
+#
+function set_tunable32
+{
+	set_tunable_impl "$1" "$2" W
+}
+
+function set_tunable_impl
+{
+	typeset tunable="$1"
+	typeset value="$2"
+	typeset mdb_cmd="$3"
+	typeset module="${4:-zfs}"
+
+	[[ -z "$tunable" ]] && return 1
+	[[ -z "$value" ]] && return 1
+	[[ -z "$mdb_cmd" ]] && return 1
+
+	case "$(uname)" in
+	Linux)
+		typeset zfs_tunables="/sys/module/$module/parameters"
+		[[ -w "$zfs_tunables/$tunable" ]] || return 1
+		echo -n "$value" > "$zfs_tunables/$tunable"
+		return "$?"
+		;;
+	SunOS)
+		[[ "$module" -eq "zfs" ]] || return 1
+		echo "${tunable}/${mdb_cmd}0t${value}" | mdb -kw
+		return "$?"
+		;;
+	esac
+}
+
+#
+# Get a global system tunable
+#
+# $1 tunable name
+#
+function get_tunable
+{
+	get_tunable_impl "$1"
+}
+
+function get_tunable_impl
+{
+	typeset tunable="$1"
+	typeset module="${2:-zfs}"
+
+	[[ -z "$tunable" ]] && return 1
+
+	case "$(uname)" in
+	Linux)
+		typeset zfs_tunables="/sys/module/$module/parameters"
+		[[ -f "$zfs_tunables/$tunable" ]] || return 1
+		cat $zfs_tunables/$tunable
+		return "$?"
+		;;
+	SunOS)
+		[[ "$module" -eq "zfs" ]] || return 1
+		;;
+	esac
+
+	return 1
+}
diff --git a/tests/zfs-tests/tests/functional/Makefile.am b/tests/zfs-tests/tests/functional/Makefile.am
index 1c3f4b1a8..d68f254ef 100644
--- a/tests/zfs-tests/tests/functional/Makefile.am
+++ b/tests/zfs-tests/tests/functional/Makefile.am
@@ -29,6 +29,7 @@ SUBDIRS = \
 	link_count \
 	migration \
 	mmap \
+	mmp \
 	mount \
 	mv_files \
 	nestedfs \
diff --git a/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg b/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg
index 0ffd6f510..e1537806f 100644
--- a/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg
+++ b/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg
@@ -54,6 +54,7 @@ typeset -a properties=(
     "freeing"
     "fragmentation"
     "leaked"
+    "multihost"
     "feature@async_destroy"
     "feature@empty_bpobj"
     "feature@lz4_compress"
diff --git a/tests/zfs-tests/tests/functional/mmp/Makefile.am b/tests/zfs-tests/tests/functional/mmp/Makefile.am
new file mode 100644
index 000000000..2399a6c37
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/mmp/Makefile.am
@@ -0,0 +1,13 @@
+pkgdatadir = $(datadir)/@PACKAGE@/zfs-tests/tests/functional/mmp
+dist_pkgdata_SCRIPTS = \
+	mmp_on_thread.ksh \
+	mmp_on_uberblocks.ksh \
+	mmp_on_off.ksh \
+	mmp_interval.ksh \
+	mmp_active_import.ksh \
+	mmp_inactive_import.ksh \
+	mmp_exported_import.ksh \
+	setup.ksh \
+	cleanup.ksh \
+	mmp.kshlib \
+	mmp.cfg
diff --git a/tests/zfs-tests/tests/functional/mmp/cleanup.ksh b/tests/zfs-tests/tests/functional/mmp/cleanup.ksh
new file mode 100755
index 000000000..6e438d88d
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/mmp/cleanup.ksh
@@ -0,0 +1,28 @@
+#!/bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2017 by Lawrence Livermore National Security, LLC.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/mmp/mmp.cfg
+
+verify_runnable "global"
+
+log_must set_tunable64 zfs_multihost_history 0
+
+log_pass "mmp cleanup passed"
diff --git a/tests/zfs-tests/tests/functional/mmp/mmp.cfg b/tests/zfs-tests/tests/functional/mmp/mmp.cfg
new file mode 100644
index 000000000..f17108a87
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/mmp/mmp.cfg
@@ -0,0 +1,38 @@
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2017 by Lawrence Livermore National Security, LLC.
+#
+
+export PREV_UBER="$TEST_BASE_DIR/mmp-uber-prev.txt"
+export CURR_UBER="$TEST_BASE_DIR/mmp-uber-curr.txt"
+export DISK=${DISKS%% *}
+
+export HOSTID_FILE="/etc/hostid"
+export HOSTID1=01234567
+export HOSTID2=89abcdef
+
+export TXG_TIMEOUT_LONG=5000
+export TXG_TIMEOUT_DEFAULT=5
+
+export MMP_POOL=mmppool
+export MMP_DIR=$TEST_BASE_DIR/mmp
+export MMP_HISTORY=100
+
+export MMP_INTERVAL_DEFAULT=1000
+export MMP_INTERVAL_MIN=100
+
+export ZPOOL_IMPORT_DURATION=9
diff --git a/tests/zfs-tests/tests/functional/mmp/mmp.kshlib b/tests/zfs-tests/tests/functional/mmp/mmp.kshlib
new file mode 100644
index 000000000..a81779b0c
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/mmp/mmp.kshlib
@@ -0,0 +1,186 @@
+#
+# CDDL HEADER START
+#
+# The contents of this file are subject to the terms of the
+# Common Development and Distribution License (the "License").
+# You may not use this file except in compliance with the License.
+#
+# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+# or http://www.opensolaris.org/os/licensing.
+# See the License for the specific language governing permissions
+# and limitations under the License.
+#
+# When distributing Covered Code, include this CDDL HEADER in each
+# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+# If applicable, add the following below this CDDL HEADER, with the
+# fields enclosed by brackets "[]" replaced with your own identifying
+# information: Portions Copyright [yyyy] [name of copyright owner]
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2017 by Lawrence Livermore National Security, LLC.
+# Use is subject to license terms.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/mmp/mmp.cfg
+
+
+function check_pool_import # pool opts token keyword
+{
+	typeset pool=${1:-$MMP_POOL}
+	typeset opts=$2
+	typeset token=$3
+	typeset keyword=$4
+
+	zpool import $opts 2>&1 | \
+	    nawk -v token="$token:" '($1==token) {print $0}' | \
+	    grep -i "$keyword" > /dev/null 2>&1
+
+	return $?
+}
+
+function is_pool_imported # pool opts
+{
+	typeset pool=${1:-$MMP_POOL}
+	typeset opts=$2
+
+	check_pool_import "$pool" "$opts" "status" \
+	    "The pool is currently imported"
+	return $?
+}
+
+function wait_pool_imported # pool opts
+{
+	typeset pool=${1:-$MMP_POOL}
+	typeset opts=$2
+
+	while is_pool_imported "$pool" "$opts"; do
+		log_must sleep 5
+	done
+
+	return 0
+}
+
+function try_pool_import # pool opts message
+{
+	typeset pool=${1:-$MMP_POOL}
+	typeset opts=$2
+	typeset msg=$3
+
+	zpool import $opts $pool 2>&1 | grep -i "$msg"
+
+	return $?
+}
+
+function mmp_set_hostid
+{
+	typeset hostid=$1
+
+	a=${hostid:6:2}
+	b=${hostid:4:2}
+	c=${hostid:2:2}
+	d=${hostid:0:2}
+
+	printf "\\x$a\\x$b\\x$c\\x$d" >$HOSTID_FILE
+
+	if [ $(hostid) != "$hostid" ]; then
+		return 1
+	fi
+
+	return 0
+}
+
+function mmp_clear_hostid
+{
+	rm -f $HOSTID_FILE
+}
+
+function mmp_pool_create # pool dir
+{
+	typeset pool=${1:-$MMP_POOL}
+	typeset dir=${2:-$MMP_DIR}
+	typeset opts="-T120 -M -k0 -f $dir -E -p $pool"
+
+	log_must mkdir -p $dir
+	log_must truncate -s $MINVDEVSIZE $dir/vdev1 $dir/vdev2
+
+	log_must mmp_set_hostid $HOSTID1
+	log_must zpool create -f $pool mirror $dir/vdev1 $dir/vdev2
+	log_must zpool set multihost=on $pool
+	log_must zpool export $pool
+	log_must mmp_set_hostid $HOSTID2
+
+	log_note "Starting ztest in the background as hostid $HOSTID1"
+	log_must eval "ZFS_HOSTID=$HOSTID1 ztest $opts >/dev/null 2>&1 &"
+
+	while ! is_pool_imported "$pool" "-d $dir"; do
+		log_must pgrep ztest
+		log_must sleep 5
+	done
+}
+
+function mmp_pool_destroy # pool dir
+{
+	typeset pool=${1:-$MMP_POOL}
+	typeset dir=${2:-$MMP_DIR}
+
+	ZTESTPID=$(pgrep ztest)
+	if [ -n "$ZTESTPID" ]; then
+		log_must kill $ZTESTPID
+		wait $ZTESTPID
+	fi
+
+	if poolexists $pool; then
+		destroy_pool $pool
+        fi
+
+	rm -Rf $dir
+	mmp_clear_hostid
+}
+
+function mmp_pool_set_hostid # pool hostid
+{
+	typeset pool=$1
+	typeset hostid=$2
+
+	log_must mmp_set_hostid $hostid
+	log_must zpool export $pool
+	log_must zpool import $pool
+
+	return 0
+}
+
+function import_no_activity_check # pool opts
+{
+	typeset pool=$1
+	typeset opts=$2
+
+	SECONDS=0
+	zpool import $opts $pool
+	typeset rc=$?
+
+	if [[ $SECONDS -gt $ZPOOL_IMPORT_DURATION ]]; then
+		log_fail "unexpected activity check (${SECONDS}s)"
+	fi
+
+	return $rc
+}
+
+function import_activity_check # pool opts
+{
+	typeset pool=$1
+	typeset opts=$2
+
+	SECONDS=0
+	zpool import $opts $pool
+	typeset rc=$?
+
+	if [[ $SECONDS -le $ZPOOL_IMPORT_DURATION ]]; then
+		log_fail "expected activity check (${SECONDS}s)"
+	fi
+
+	return $rc
+}
diff --git a/tests/zfs-tests/tests/functional/mmp/mmp_active_import.ksh b/tests/zfs-tests/tests/functional/mmp/mmp_active_import.ksh
new file mode 100755
index 000000000..92eb9ce2d
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/mmp/mmp_active_import.ksh
@@ -0,0 +1,104 @@
+#!/bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2017 by Lawrence Livermore National Security, LLC.
+#
+
+# DESCRIPTION:
+#	Under no circumstances when multihost is active, should an active pool
+#	with one hostid be importable by a host with a different hostid.
+#
+# STRATEGY:
+#	 1. Simulate an active pool on another host with ztest.
+#	 2. Verify 'zpool import' reports an active pool.
+#	 3. Verify 'zpool import [-f] $MMP_POOL' cannot import the pool.
+#	 4. Kill ztest to make pool eligible for import.
+#	 5. Verify 'zpool import' fails with the expected error message.
+#	 6. Verify 'zpool import $MMP_POOL' fails with the expected message.
+#	 7. Verify 'zpool import -f $MMP_POOL' can now import the pool.
+#	 8. Verify pool may be exported/imported without -f argument.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/mmp/mmp.cfg
+. $STF_SUITE/tests/functional/mmp/mmp.kshlib
+
+verify_runnable "both"
+
+function cleanup
+{
+	mmp_pool_destroy $MMP_DIR $MMP_POOL
+	log_must set_tunable64 zfs_multihost_interval $MMP_INTERVAL_DEFAULT
+	log_must mmp_clear_hostid
+}
+
+log_assert "multihost=on|off active pool activity checks"
+log_onexit cleanup
+
+# 1. Simulate an active pool on another host with ztest.
+mmp_pool_destroy $MMP_POOL $MMP_DIR
+mmp_pool_create $MMP_POOL $MMP_DIR
+
+# 2. Verify 'zpool import' reports an active pool.
+log_must mmp_set_hostid $HOSTID2
+log_must set_tunable64 zfs_multihost_interval $MMP_INTERVAL_MIN
+log_must is_pool_imported $MMP_POOL "-d $MMP_DIR"
+
+# 3. Verify 'zpool import [-f] $MMP_POOL' cannot import the pool.
+MMP_IMPORTED_MSG="Cannot import '$MMP_POOL': pool is imported"
+log_must try_pool_import $MMP_POOL "-d $MMP_DIR" $MMP_IMPORTED_MSG
+for i in {1..10}; do
+	log_must pgrep ztest >/dev/null
+	log_must try_pool_import $MMP_POOL "-f -d $MMP_DIR" $MMP_IMPORTED_MSG
+done
+
+# 4. Kill ztest to make pool eligible for import.  Poll with 'zpool status'.
+ZTESTPID=$(pgrep ztest)
+if [ -n "$ZTESTPID" ]; then
+	log_must kill -9 $ZTESTPID
+fi
+log_must wait_pool_imported $MMP_POOL "-d $MMP_DIR"
+
+# 5. Verify 'zpool import' fails with the expected error message, when
+#    - hostid=0:        - configuration error
+#    - hostid=matches   - safe to import the pool
+#    - hostid=different - previously imported on a different system
+#
+log_must mmp_clear_hostid
+MMP_IMPORTED_MSG="Set the system hostid"
+log_must check_pool_import $MMP_POOL "-d $MMP_DIR" "action" $MMP_IMPORTED_MSG
+
+log_must mmp_set_hostid $HOSTID1
+MMP_IMPORTED_MSG="The pool can be imported"
+log_must check_pool_import $MMP_POOL "-d $MMP_DIR" "action" $MMP_IMPORTED_MSG
+
+log_must mmp_set_hostid $HOSTID2
+MMP_IMPORTED_MSG="The pool was last accessed by another system."
+log_must check_pool_import $MMP_POOL "-d $MMP_DIR" "status" $MMP_IMPORTED_MSG
+
+# 6. Verify 'zpool import $MMP_POOL' fails with the expected message.
+MMP_IMPORTED_MSG="pool was previously in use from another system."
+log_must try_pool_import $MMP_POOL "-d $MMP_DIR" $MMP_IMPORTED_MSG
+
+# 7. Verify 'zpool import -f $MMP_POOL' can now import the pool.
+log_must import_activity_check $MMP_POOL "-f -d $MMP_DIR"
+
+# 8 Verify pool may be exported/imported without -f argument.
+log_must zpool export $MMP_POOL
+log_must import_no_activity_check $MMP_POOL "-d $MMP_DIR"
+
+log_pass "multihost=on|off active pool activity checks passed"
diff --git a/tests/zfs-tests/tests/functional/mmp/mmp_exported_import.ksh b/tests/zfs-tests/tests/functional/mmp/mmp_exported_import.ksh
new file mode 100755
index 000000000..d65ca5b3d
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/mmp/mmp_exported_import.ksh
@@ -0,0 +1,104 @@
+#!/bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2017 by Lawrence Livermore National Security, LLC.
+#
+
+# DESCRIPTION:
+#	Verify import behavior for exported pool (no activity check)
+#
+# STRATEGY:
+#	1. Create a zpool
+#	2. Verify multihost=off and hostids match (no activity check)
+#	3. Verify multihost=off and hostids differ (no activity check)
+#	4. Verify multihost=off and hostid zero allowed (no activity check)
+#	5. Verify multihost=on and hostids match (no activity check)
+#	6. Verify multihost=on and hostids differ (no activity check)
+#	7. Verify multihost=on and hostid zero fails (no activity check)
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/mmp/mmp.cfg
+. $STF_SUITE/tests/functional/mmp/mmp.kshlib
+
+verify_runnable "both"
+
+function cleanup
+{
+	default_cleanup_noexit
+	log_must mmp_clear_hostid
+}
+
+log_assert "multihost=on|off activity checks exported pool"
+log_onexit cleanup
+
+# 1. Create a zpool
+log_must mmp_set_hostid $HOSTID1
+default_setup_noexit $DISK
+
+# 2. Verify multihost=off and hostids match (no activity check)
+log_must zpool set multihost=off $TESTPOOL
+
+for opt in "" "-f"; do
+	log_must zpool export $TESTPOOL
+	log_must import_no_activity_check $TESTPOOL $opt
+done
+
+# 3. Verify multihost=off and hostids differ (no activity check)
+for opt in "" "-f"; do
+	log_must mmp_pool_set_hostid $TESTPOOL $HOSTID1
+	log_must zpool export $TESTPOOL
+	log_must mmp_set_hostid $HOSTID2
+	log_must import_no_activity_check $TESTPOOL $opt
+done
+
+# 4. Verify multihost=off and hostid zero allowed (no activity check)
+log_must mmp_clear_hostid
+
+for opt in "" "-f"; do
+	log_must zpool export $TESTPOOL
+	log_must import_no_activity_check $TESTPOOL $opt
+done
+
+# 5. Verify multihost=on and hostids match (no activity check)
+log_must mmp_pool_set_hostid $TESTPOOL $HOSTID1
+log_must zpool set multihost=on $TESTPOOL
+
+for opt in "" "-f"; do
+	log_must zpool export $TESTPOOL
+	log_must import_no_activity_check $TESTPOOL $opt
+done
+
+# 6. Verify multihost=on and hostids differ (no activity check)
+for opt in "" "-f"; do
+	log_must mmp_pool_set_hostid $TESTPOOL $HOSTID1
+	log_must zpool export $TESTPOOL
+	log_must mmp_set_hostid $HOSTID2
+	log_must import_no_activity_check $TESTPOOL $opt
+done
+
+# 7. Verify multihost=on and hostid zero fails (no activity check)
+log_must zpool export $TESTPOOL
+log_must mmp_clear_hostid
+
+for opt in "" "-f"; do
+	MMP_IMPORTED_MSG="Set the system hostid"
+	log_must check_pool_import $TESTPOOL "" "action" $MMP_IMPORTED_MSG
+	log_mustnot import_no_activity_check $TESTPOOL $opt
+done
+
+log_pass "multihost=on|off exported pool activity checks passed"
diff --git a/tests/zfs-tests/tests/functional/mmp/mmp_inactive_import.ksh b/tests/zfs-tests/tests/functional/mmp/mmp_inactive_import.ksh
new file mode 100755
index 000000000..c944b6b28
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/mmp/mmp_inactive_import.ksh
@@ -0,0 +1,95 @@
+#!/bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2017 by Lawrence Livermore National Security, LLC.
+#
+
+# DESCRIPTION:
+#	Verify import behavior for inactive, but not exported, pools
+#
+# STRATEGY:
+#	1. Create a zpool
+#	2. Verify multihost=off and hostids match (no activity check)
+#	3. Verify multihost=off and hostids differ (no activity check)
+#	4. Verify multihost=off and hostid allowed (no activity check)
+#	5. Verify multihost=on and hostids match (no activity check)
+#	6. Verify multihost=on and hostids differ (activity check)
+#	7. Verify multihost=on and hostid zero fails (activity check)
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/mmp/mmp.cfg
+. $STF_SUITE/tests/functional/mmp/mmp.kshlib
+
+verify_runnable "both"
+
+function cleanup
+{
+	default_cleanup_noexit
+	log_must mmp_clear_hostid
+}
+
+log_assert "multihost=on|off inactive pool activity checks"
+log_onexit cleanup
+
+# 1. Create a zpool
+log_must mmp_set_hostid $HOSTID1
+default_setup_noexit $DISK
+
+# 2. Verify multihost=off and hostids match (no activity check)
+log_must zpool set multihost=off $TESTPOOL
+
+for opt in "" "-f"; do
+	log_must zpool export -F $TESTPOOL
+	log_must import_no_activity_check $TESTPOOL $opt
+done
+
+# 3. Verify multihost=off and hostids differ (no activity check)
+log_must zpool export -F $TESTPOOL
+log_must mmp_set_hostid $HOSTID2
+log_mustnot import_no_activity_check $TESTPOOL ""
+log_must import_no_activity_check $TESTPOOL "-f"
+
+# 4. Verify multihost=off and hostid zero allowed (no activity check)
+log_must zpool export -F $TESTPOOL
+log_must mmp_clear_hostid
+log_mustnot import_no_activity_check $TESTPOOL ""
+log_must import_no_activity_check $TESTPOOL "-f"
+
+# 5. Verify multihost=on and hostids match (no activity check)
+log_must mmp_pool_set_hostid $TESTPOOL $HOSTID1
+log_must zpool set multihost=on $TESTPOOL
+
+for opt in "" "-f"; do
+	log_must zpool export -F $TESTPOOL
+	log_must import_no_activity_check $TESTPOOL $opt
+done
+
+# 6. Verify multihost=on and hostids differ (activity check)
+log_must zpool export -F $TESTPOOL
+log_must mmp_set_hostid $HOSTID2
+log_mustnot import_activity_check $TESTPOOL ""
+log_must import_activity_check $TESTPOOL "-f"
+
+# 7. Verify multihost=on and hostid zero fails (activity check)
+log_must zpool export -F $TESTPOOL
+log_must mmp_clear_hostid
+MMP_IMPORTED_MSG="Set the system hostid"
+log_must check_pool_import $TESTPOOL "-f" "action" $MMP_IMPORTED_MSG
+log_mustnot import_activity_check $TESTPOOL "-f"
+
+log_pass "multihost=on|off inactive pool activity checks passed"
diff --git a/tests/zfs-tests/tests/functional/mmp/mmp_interval.ksh b/tests/zfs-tests/tests/functional/mmp/mmp_interval.ksh
new file mode 100755
index 000000000..fb44d6191
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/mmp/mmp_interval.ksh
@@ -0,0 +1,47 @@
+#!/bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2017 by Lawrence Livermore National Security, LLC.
+#
+
+# DESCRIPTION:
+#	zfs_multihost_interval should only accept valid values.
+#
+# STRATEGY:
+#	1. Set zfs_multihost_interval to invalid values (negative).
+#	2. Set zfs_multihost_interval to valid values.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/mmp/mmp.cfg
+. $STF_SUITE/tests/functional/mmp/mmp.kshlib
+
+verify_runnable "both"
+
+function cleanup
+{
+	log_must set_tunable64 zfs_multihost_interval $MMP_INTERVAL_DEFAULT
+}
+
+log_assert "zfs_multihost_interval cannot be set to an invalid value"
+log_onexit cleanup
+
+log_mustnot set_tunable64 zfs_multihost_interval -1
+log_must set_tunable64 zfs_multihost_interval $MMP_INTERVAL_MIN
+log_must set_tunable64 zfs_multihost_interval $MMP_INTERVAL_DEFAULT
+
+log_pass "zfs_multihost_interval cannot be set to an invalid value"
diff --git a/tests/zfs-tests/tests/functional/mmp/mmp_on_off.ksh b/tests/zfs-tests/tests/functional/mmp/mmp_on_off.ksh
new file mode 100755
index 000000000..8bef86a0f
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/mmp/mmp_on_off.ksh
@@ -0,0 +1,79 @@
+#!/bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2017 by Lawrence Livermore National Security, LLC.
+#
+
+# DESCRIPTION:
+#	When multihost=off ensure that leaf vdev uberblocks are not updated.
+#
+# STRATEGY:
+#	1. Set multihost=off (disables mmp)
+#	2. Set zfs_txg_timeout to large value
+#	3. Create a zpool
+#	4. Find the current "best" uberblock
+#	5. Sleep for enough time for uberblocks to change
+#	6. Find the current "best" uberblock
+#	7. If the uberblock changed, fail
+#	8. Set multihost=on
+#	9. Sleep for enough time for uberblocks to change
+#	10. Find the current "best" uberblock
+#	11. If uberblocks didn't change, fail
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/mmp/mmp.cfg
+. $STF_SUITE/tests/functional/mmp/mmp.kshlib
+
+verify_runnable "both"
+
+function cleanup
+{
+	default_cleanup_noexit
+	log_must set_tunable64 zfs_txg_timeout $TXG_TIMEOUT_DEFAULT
+	log_must set_tunable64 zfs_multihost_interval $MMP_INTERVAL_DEFAULT
+	log_must rm -f $PREV_UBER $CURR_UBER
+	log_must mmp_clear_hostid
+}
+
+log_assert "mmp thread won't write uberblocks with multihost=off"
+log_onexit cleanup
+
+log_must set_tunable64 zfs_multihost_interval $MMP_INTERVAL_MIN
+log_must set_tunable64 zfs_txg_timeout $TXG_TIMEOUT_LONG
+log_must mmp_set_hostid $HOSTID1
+
+default_setup_noexit $DISK
+log_must zpool set multihost=off $TESTPOOL
+
+log_must zdb -u $TESTPOOL > $PREV_UBER
+log_must sleep 5
+log_must zdb -u $TESTPOOL > $CURR_UBER
+
+if ! diff "$CURR_UBER" "$PREV_UBER"; then
+	log_fail "mmp thread has updated an uberblock"
+fi
+
+log_must zpool set multihost=on $TESTPOOL
+log_must sleep 5
+log_must zdb -u $TESTPOOL > $CURR_UBER
+
+if diff "$CURR_UBER" "$PREV_UBER"; then
+	log_fail "mmp failed to update uberblocks"
+fi
+
+log_pass "mmp thread won't write uberblocks with multihost=off passed"
diff --git a/tests/zfs-tests/tests/functional/mmp/mmp_on_thread.ksh b/tests/zfs-tests/tests/functional/mmp/mmp_on_thread.ksh
new file mode 100755
index 000000000..07384c623
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/mmp/mmp_on_thread.ksh
@@ -0,0 +1,64 @@
+#!/bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2017 by Lawrence Livermore National Security, LLC.
+#
+
+# DESCRIPTION:
+#	Ensure that the MMP thread is writing uberblocks.
+#
+# STRATEGY:
+#	1. Set zfs_txg_timeout to large value
+#	2. Create a zpool
+#	3. Find the current "best" uberblock
+#	4. Sleep for enough time for a potential uberblock update
+#	5. Find the current "best" uberblock
+#	6. If the uberblock never changed, fail
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/mmp/mmp.cfg
+. $STF_SUITE/tests/functional/mmp/mmp.kshlib
+
+verify_runnable "both"
+
+function cleanup
+{
+	default_cleanup_noexit
+	log_must set_tunable64 zfs_txg_timeout $TXG_TIMEOUT_DEFAULT
+	log_must rm -f $PREV_UBER $CURR_UBER
+	log_must mmp_clear_hostid
+}
+
+log_assert "mmp thread writes uberblocks (MMP)"
+log_onexit cleanup
+
+log_must set_tunable64 zfs_txg_timeout $TXG_TIMEOUT_LONG
+log_must mmp_set_hostid $HOSTID1
+
+default_setup_noexit $DISK
+log_must zpool set multihost=on $TESTPOOL
+
+log_must zdb -u $TESTPOOL > $PREV_UBER
+log_must sleep 5
+log_must zdb -u $TESTPOOL > $CURR_UBER
+
+if diff -u "$CURR_UBER" "$PREV_UBER"; then
+	log_fail "mmp failed to update uberblocks"
+fi
+
+log_pass "mmp thread writes uberblocks (MMP) passed"
diff --git a/tests/zfs-tests/tests/functional/mmp/mmp_on_uberblocks.ksh b/tests/zfs-tests/tests/functional/mmp/mmp_on_uberblocks.ksh
new file mode 100755
index 000000000..2e21e2fef
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/mmp/mmp_on_uberblocks.ksh
@@ -0,0 +1,79 @@
+#!/bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2017 by Lawrence Livermore National Security, LLC.
+#
+
+# DESCRIPTION:
+#	Ensure that MMP updates uberblocks at the expected intervals.
+#
+# STRATEGY:
+#	1. Set zfs_txg_timeout to large value
+#	2. Create a zpool
+#	3. Find the current "best" uberblock
+#	4. Loop for 10 seconds, increment counter for each change in UB
+#	5. If number of changes seen is less than min threshold, then fail
+#	6. If number of changes seen is more than max threshold, then fail
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/mmp/mmp.cfg
+. $STF_SUITE/tests/functional/mmp/mmp.kshlib
+
+verify_runnable "both"
+
+UBER_CHANGES=0
+
+function cleanup
+{
+	default_cleanup_noexit
+	set_tunable64 zfs_txg_timeout $TXG_TIMEOUT_DEFAULT
+	log_must rm -f $PREV_UBER $CURR_UBER
+	log_must mmp_clear_hostid
+}
+
+log_assert "Ensure MMP uberblocks update at the correct interval"
+log_onexit cleanup
+
+log_must set_tunable64 zfs_txg_timeout $TXG_TIMEOUT_LONG
+log_must mmp_set_hostid $HOSTID1
+
+default_setup_noexit $DISK
+log_must zpool set multihost=on $TESTPOOL
+
+log_must zdb -u $TESTPOOL > $PREV_UBER
+
+SECONDS=0
+while [[ $SECONDS -le 10 ]]; do
+	log_must zdb -u $TESTPOOL > $CURR_UBER
+	if ! diff -u "$CURR_UBER" "$PREV_UBER"; then
+		(( UBER_CHANGES = UBER_CHANGES + 1 ))
+		log_must mv "$CURR_UBER" "$PREV_UBER"
+	fi
+done
+
+log_note "Uberblock changed $UBER_CHANGES times"
+
+if [[ $UBER_CHANGES -lt 8 ]]; then
+	log_fail "Fewer uberblock writes occured than expected (10)"
+fi
+
+if [[ $UBER_CHANGES -gt 12 ]]; then
+	log_fail "More uberblock writes occured than expected (10)"
+fi
+
+log_pass "Ensure MMP uberblocks update at the correct interval passed"
diff --git a/tests/zfs-tests/tests/functional/mmp/setup.ksh b/tests/zfs-tests/tests/functional/mmp/setup.ksh
new file mode 100755
index 000000000..fde5e3bb7
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/mmp/setup.ksh
@@ -0,0 +1,32 @@
+#!/bin/ksh -p
+#
+# CDDL HEADER START
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# http://www.illumos.org/license/CDDL.
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2017 by Lawrence Livermore National Security, LLC.
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/mmp/mmp.cfg
+
+verify_runnable "global"
+
+if [ -e $HOSTID_FILE ]; then
+	log_unsupported "System has existing $HOSTID_FILE file"
+fi
+
+log_must set_tunable64 zfs_multihost_history $MMP_HISTORY
+
+log_pass "mmp setup pass"