MMP interval and fail_intervals in uberblock

When Multihost is enabled, and a pool is imported, uberblock writes
include ub_mmp_delay to allow an importing node to calculate the
duration of an activity test.  This value, is not enough information.

If zfs_multihost_fail_intervals > 0 on the node with the pool imported,
the safe minimum duration of the activity test is well defined, but does
not depend on ub_mmp_delay:

zfs_multihost_fail_intervals * zfs_multihost_interval

and if zfs_multihost_fail_intervals == 0 on that node, there is no such
well defined safe duration, but the importing host cannot tell whether
mmp_delay is high due to I/O delays, or due to a very large
zfs_multihost_interval setting on the host which last imported the pool.
As a result, it may use a far longer period for the activity test than
is necessary.

This patch renames ub_mmp_sequence to ub_mmp_config and uses it to
record the zfs_multihost_interval and zfs_multihost_fail_intervals
values, as well as the mmp sequence.  This allows a shorter activity
test duration to be calculated by the importing host in most situations.
These values are also added to the multihost_history kstat records.

It calculates the activity test duration differently depending on
whether the new fields are present or not; for importing pools with
only ub_mmp_delay, it uses

(zfs_multihost_interval + ub_mmp_delay) * zfs_multihost_import_intervals

Which results in an activity test duration less sensitive to the leaf
count.

In addition, it makes a few other improvements:
* It updates the "sequence" part of ub_mmp_config when MMP writes
  in between syncs occur.  This allows an importing host to detect MMP
  on the remote host sooner, when the pool is idle, as it is not limited
  to the granularity of ub_timestamp (1 second).
* It issues writes immediately when zfs_multihost_interval is changed
  so remote hosts see the updated value as soon as possible.
* It fixes a bug where setting zfs_multihost_fail_intervals = 1 results
  in immediate pool suspension.
* Update tests to verify activity check duration is based on recorded
  tunable values, not tunable values on importing host.
* Update tests to verify the expected number of uberblocks have valid
  MMP fields - fail_intervals, mmp_interval, mmp_seq (sequence number),
  that sequence number is incrementing, and that uberblock values match
  tunable settings.

Reviewed-by: Andreas Dilger <[email protected]>
Reviewed-by: Brian Behlendorf <[email protected]>
Reviewed-by: Tony Hutter <[email protected]>
Signed-off-by: Olaf Faaland <[email protected]>
Closes #7842 

1 files changed, 231 insertions, 90 deletions

diff --git a/module/zfs/mmp.c b/module/zfs/mmp.c
index 16975dd98..cd5603a1a 100644
--- a/module/zfs/mmp.c
+++ b/module/zfs/mmp.c
@@ -49,40 +49,101 @@
  * Uberblocks written by the txg_sync thread always go into the first
  * (N-MMP_BLOCKS_PER_LABEL) slots, the remaining slots are reserved for MMP.
  * They are used to hold uberblocks which are exactly the same as the last
- * synced uberblock except that the ub_timestamp is frequently updated.
- * Like all other uberblocks, the slot is written with an embedded checksum,
- * and slots with invalid checksums are ignored.  This provides the
+ * synced uberblock except that the ub_timestamp and mmp_config are frequently
+ * updated.  Like all other uberblocks, the slot is written with an embedded
+ * checksum, and slots with invalid checksums are ignored.  This provides the
  * "heartbeat", with no risk of overwriting good uberblocks that must be
  * preserved, e.g. previous txgs and associated block pointers.
  *
- * Two optional fields are added to uberblock structure: ub_mmp_magic and
- * ub_mmp_delay.  The magic field allows zfs to tell whether ub_mmp_delay is
- * valid.  The delay field is a decaying average of the amount of time between
- * completion of successive MMP writes, in nanoseconds.  It is used to predict
- * how long the import must wait to detect activity in the pool, before
- * concluding it is not in use.
+ * Three optional fields are added to uberblock structure; ub_mmp_magic,
+ * ub_mmp_config, and ub_mmp_delay.  The ub_mmp_magic value allows zfs to tell
+ * whether the other ub_mmp_* fields are valid.  The ub_mmp_config field tells
+ * the importing host the settings of zfs_multihost_interval and
+ * zfs_multihost_fail_intervals on the host which last had (or currently has)
+ * the pool imported.  These determine how long a host must wait to detect
+ * activity in the pool, before concluding the pool is not in use.  The
+ * mmp_delay field is a decaying average of the amount of time between
+ * completion of successive MMP writes, in nanoseconds.  It indicates whether
+ * MMP is enabled.
  *
  * During import an activity test may now be performed to determine if
  * the pool is in use.  The activity test is typically required if the
  * ZPOOL_CONFIG_HOSTID does not match the system hostid, the pool state is
  * POOL_STATE_ACTIVE, and the pool is not a root pool.
  *
- * The activity test finds the "best" uberblock (highest txg & timestamp),
- * waits some time, and then finds the "best" uberblock again.  If the txg
- * and timestamp in both "best" uberblocks do not match, the pool is in use
- * by another host and the import fails.  Since the granularity of the
- * timestamp is in seconds this activity test must take a bare minimum of one
- * second.  In order to assure the accuracy of the activity test, the default
- * values result in an activity test duration of 10x the mmp write interval.
- *
- * The "zpool import"  activity test can be expected to take a minimum time of
- * zfs_multihost_import_intervals * zfs_multihost_interval milliseconds.  If the
- * "best" uberblock has a valid ub_mmp_delay field, then the duration of the
- * test may take longer if MMP writes were occurring less frequently than
- * expected.  Additionally, the duration is then extended by a random 25% to
- * attempt to to detect simultaneous imports.  For example, if both partner
- * hosts are rebooted at the same time and automatically attempt to import the
- * pool.
+ * The activity test finds the "best" uberblock (highest txg, timestamp, and, if
+ * ub_mmp_magic is valid, sequence number from ub_mmp_config).  It then waits
+ * some time, and finds the "best" uberblock again.  If any of the mentioned
+ * fields have different values in the newly read uberblock, the pool is in use
+ * by another host and the import fails.  In order to assure the accuracy of the
+ * activity test, the default values result in an activity test duration of 20x
+ * the mmp write interval.
+ *
+ * The duration of the "zpool import" activity test depends on the information
+ * available in the "best" uberblock:
+ *
+ * 1) If uberblock was written by zfs-0.8 or newer and fail_intervals > 0:
+ *    ub_mmp_config.fail_intervals * ub_mmp_config.multihost_interval * 2
+ *
+ *    In this case, a weak guarantee is provided.  Since the host which last had
+ *    the pool imported will suspend the pool if no mmp writes land within
+ *    fail_intervals * multihost_interval ms, the absense of writes during that
+ *    time means either the pool is not imported, or it is imported but the pool
+ *    is suspended and no further writes will occur.
+ *
+ *    Note that resuming the suspended pool on the remote host would invalidate
+ *    this gurantee, and so it is not allowed.
+ *
+ *    The factor of 2 provides a conservative safety factor and derives from
+ *    MMP_IMPORT_SAFETY_FACTOR;
+ *
+ * 2) If uberblock was written by zfs-0.8 or newer and fail_intervals == 0:
+ *    (ub_mmp_config.multihost_interval + ub_mmp_delay) *
+ *        zfs_multihost_import_intervals
+ *
+ *    In this case no guarantee can provided.  However, as long as some devices
+ *    are healthy and connected, it is likely that at least one write will land
+ *    within (multihost_interval + mmp_delay) because multihost_interval is
+ *    enough time for a write to be attempted to each leaf vdev, and mmp_delay
+ *    is enough for one to land, based on past delays.  Multiplying by
+ *    zfs_multihost_import_intervals provides a conservative safety factor.
+ *
+ * 3) If uberblock was written by zfs-0.7:
+ *    (zfs_multihost_interval + ub_mmp_delay) * zfs_multihost_import_intervals
+ *
+ *    The same logic as case #2 applies, but we do not know remote tunables.
+ *
+ *    We use the local value for zfs_multihost_interval because the original MMP
+ *    did not record this value in the uberblock.
+ *
+ *    ub_mmp_delay >= (zfs_multihost_interval / leaves), so if the other host
+ *    has a much larger zfs_multihost_interval set, ub_mmp_delay will reflect
+ *    that.  We will have waited enough time for zfs_multihost_import_intervals
+ *    writes to be issued and all but one to land.
+ *
+ *    single device pool example delays
+ *
+ *    import_delay = (1 + 1) * 20   =  40s #defaults, no I/O delay
+ *    import_delay = (1 + 10) * 20  = 220s #defaults, 10s I/O delay
+ *    import_delay = (10 + 10) * 20 = 400s #10s multihost_interval,
+ *                                          no I/O delay
+ *    100 device pool example delays
+ *
+ *    import_delay = (1 + .01) * 20 =  20s #defaults, no I/O delay
+ *    import_delay = (1 + 10) * 20  = 220s #defaults, 10s I/O delay
+ *    import_delay = (10 + .1) * 20 = 202s #10s multihost_interval,
+ *                                          no I/O delay
+ *
+ * 4) Otherwise, this uberblock was written by a pre-MMP zfs:
+ *    zfs_multihost_import_intervals * zfs_multihost_interval
+ *
+ *    In this case local tunables are used.  By default this product = 10s, long
+ *    enough for a pool with any activity at all to write at least one
+ *    uberblock.  No guarantee can be provided.
+ *
+ * Additionally, the duration is then extended by a random 25% to attempt to to
+ * detect simultaneous imports.  For example, if both partner hosts are rebooted
+ * at the same time and automatically attempt to import the pool.
  */
 
 /*
@@ -90,10 +151,9 @@
  * 'multihost' pool property is on.  This is one factor used to determine the
  * length of the activity check during import.
  *
- * The mmp write period is zfs_multihost_interval / leaf-vdevs milliseconds.
- * This means that on average an mmp write will be issued for each leaf vdev
- * every zfs_multihost_interval milliseconds.  In practice, the observed period
- * can vary with the I/O load and this observed value is the delay which is
+ * On average an mmp write will be issued for each leaf vdev every
+ * zfs_multihost_interval milliseconds.  In practice, the observed period can
+ * vary with the I/O load and this observed value is the ub_mmp_delay which is
  * stored in the uberblock.  The minimum allowed value is 100 ms.
  */
 ulong_t zfs_multihost_interval = MMP_DEFAULT_INTERVAL;
@@ -108,19 +168,21 @@ ulong_t zfs_multihost_interval = MMP_DEFAULT_INTERVAL;
 uint_t zfs_multihost_import_intervals = MMP_DEFAULT_IMPORT_INTERVALS;
 
 /*
- * Controls the behavior of the pool when mmp write failures are detected.
+ * Controls the behavior of the pool when mmp write failures or delays are
+ * detected.
  *
- * When zfs_multihost_fail_intervals = 0 then mmp write failures are ignored.
- * The failures will still be reported to the ZED which depending on its
- * configuration may take action such as suspending the pool or taking a
+ * When zfs_multihost_fail_intervals = 0, mmp write failures or delays are
+ * ignored.  The failures will still be reported to the ZED which depending on
+ * its configuration may take action such as suspending the pool or taking a
  * device offline.
  *
- * When zfs_multihost_fail_intervals > 0 then sequential mmp write failures will
- * cause the pool to be suspended.  This occurs when
- * zfs_multihost_fail_intervals * zfs_multihost_interval milliseconds have
- * passed since the last successful mmp write.  This guarantees the activity
- * test will see mmp writes if the
- * pool is imported.
+ * When zfs_multihost_fail_intervals > 0, the pool will be suspended if
+ * zfs_multihost_fail_intervals * zfs_multihost_interval milliseconds pass
+ * without a successful mmp write.  This guarantees the activity test will see
+ * mmp writes if the pool is imported.  A value of 1 is ignored and treated as
+ * if it was set to 2, because a single leaf vdev pool will issue a write once
+ * per multihost_interval and thus any variation in latency would cause the
+ * pool to be suspended.
  */
 uint_t zfs_multihost_fail_intervals = MMP_DEFAULT_FAIL_INTERVALS;
 
@@ -136,6 +198,14 @@ mmp_init(spa_t *spa)
 	cv_init(&mmp->mmp_thread_cv, NULL, CV_DEFAULT, NULL);
 	mutex_init(&mmp->mmp_io_lock, NULL, MUTEX_DEFAULT, NULL);
 	mmp->mmp_kstat_id = 1;
+
+	/*
+	 * mmp_write_done() calculates mmp_delay based on prior mmp_delay and
+	 * the elapsed time since the last write.  For the first mmp write,
+	 * there is no "last write", so we start with fake non-zero values.
+	 */
+	mmp->mmp_last_write = gethrtime();
+	mmp->mmp_delay = MSEC2NSEC(MMP_INTERVAL_OK(zfs_multihost_interval));
 }
 
 void
@@ -173,10 +243,10 @@ mmp_thread_start(spa_t *spa)
 	if (spa_writeable(spa)) {
 		mutex_enter(&mmp->mmp_thread_lock);
 		if (!mmp->mmp_thread) {
-			dprintf("mmp_thread_start pool %s\n",
-			    spa->spa_name);
 			mmp->mmp_thread = thread_create(NULL, 0, mmp_thread,
 			    spa, 0, &p0, TS_RUN, defclsyspri);
+			zfs_dbgmsg("MMP thread started pool '%s' "
+			    "gethrtime %llu", spa_name(spa), gethrtime());
 		}
 		mutex_exit(&mmp->mmp_thread_lock);
 	}
@@ -195,6 +265,8 @@ mmp_thread_stop(spa_t *spa)
 		cv_wait(&mmp->mmp_thread_cv, &mmp->mmp_thread_lock);
 	}
 	mutex_exit(&mmp->mmp_thread_lock);
+	zfs_dbgmsg("MMP thread stopped pool '%s' gethrtime %llu",
+	    spa_name(spa), gethrtime());
 
 	ASSERT(mmp->mmp_thread == NULL);
 	mmp->mmp_thread_exiting = 0;
@@ -304,7 +376,8 @@ mmp_delay_update(spa_t *spa, boolean_t write_completed)
 	 * strictly less than, in case delay was changed above.
 	 */
 	if (delay < mts->mmp_delay) {
-		hrtime_t min_delay = MSEC2NSEC(zfs_multihost_interval) /
+		hrtime_t min_delay =
+		    MSEC2NSEC(MMP_INTERVAL_OK(zfs_multihost_interval)) /
 		    MAX(1, vdev_count_leaves(spa));
 		mts->mmp_delay = MAX(((delay + mts->mmp_delay * 127) / 128),
 		    min_delay);
@@ -348,6 +421,7 @@ mmp_update_uberblock(spa_t *spa, uberblock_t *ub)
 
 	mutex_enter(&mmp->mmp_io_lock);
 	mmp->mmp_ub = *ub;
+	mmp->mmp_seq = 1;
 	mmp->mmp_ub.ub_timestamp = gethrestime_sec();
 	mmp_delay_update(spa, B_TRUE);
 	mutex_exit(&mmp->mmp_io_lock);
@@ -372,8 +446,9 @@ mmp_write_uberblock(spa_t *spa)
 	spa_config_enter(spa, SCL_STATE, mmp_tag, RW_READER);
 	lock_acquire_time = gethrtime() - lock_acquire_time;
 	if (lock_acquire_time > (MSEC2NSEC(MMP_MIN_INTERVAL) / 10))
-		zfs_dbgmsg("SCL_STATE acquisition took %llu ns\n",
-		    (u_longlong_t)lock_acquire_time);
+		zfs_dbgmsg("MMP SCL_STATE acquisition pool '%s' took %llu ns "
+		    "gethrtime %llu", spa_name(spa), lock_acquire_time,
+		    gethrtime());
 
 	mutex_enter(&mmp->mmp_io_lock);
 
@@ -396,6 +471,9 @@ mmp_write_uberblock(spa_t *spa)
 			spa_mmp_history_add(spa, mmp->mmp_ub.ub_txg,
 			    gethrestime_sec(), mmp->mmp_delay, NULL, 0,
 			    mmp->mmp_kstat_id++, error);
+			zfs_dbgmsg("MMP error choosing leaf pool '%s' "
+			    "gethrtime %llu fail_mask %#x", spa_name(spa),
+			    gethrtime(), error);
 		}
 		mutex_exit(&mmp->mmp_io_lock);
 		spa_config_exit(spa, SCL_STATE, mmp_tag);
@@ -403,16 +481,34 @@ mmp_write_uberblock(spa_t *spa)
 	}
 
 	vd = spa->spa_mmp.mmp_last_leaf;
-	mmp->mmp_skip_error = 0;
+	if (mmp->mmp_skip_error != 0) {
+		mmp->mmp_skip_error = 0;
+		zfs_dbgmsg("MMP write after skipping due to unavailable "
+		    "leaves, pool '%s' gethrtime %llu leaf %#llu",
+		    spa_name(spa), gethrtime(), vd->vdev_guid);
+	}
 
 	if (mmp->mmp_zio_root == NULL)
 		mmp->mmp_zio_root = zio_root(spa, NULL, NULL,
 		    flags | ZIO_FLAG_GODFATHER);
 
+	if (mmp->mmp_ub.ub_timestamp != gethrestime_sec()) {
+		/*
+		 * Want to reset mmp_seq when timestamp advances because after
+		 * an mmp_seq wrap new values will not be chosen by
+		 * uberblock_compare() as the "best".
+		 */
+		mmp->mmp_ub.ub_timestamp = gethrestime_sec();
+		mmp->mmp_seq = 1;
+	}
+
 	ub = &mmp->mmp_ub;
-	ub->ub_timestamp = gethrestime_sec();
 	ub->ub_mmp_magic = MMP_MAGIC;
 	ub->ub_mmp_delay = mmp->mmp_delay;
+	ub->ub_mmp_config = MMP_SEQ_SET(mmp->mmp_seq) |
+	    MMP_INTERVAL_SET(MMP_INTERVAL_OK(zfs_multihost_interval)) |
+	    MMP_FAIL_INT_SET(MMP_FAIL_INTVS_OK(
+	    zfs_multihost_fail_intervals));
 	vd->vdev_mmp_pending = gethrtime();
 	vd->vdev_mmp_kstat_id = mmp->mmp_kstat_id;
 
@@ -421,6 +517,7 @@ mmp_write_uberblock(spa_t *spa)
 	abd_zero(ub_abd, VDEV_UBERBLOCK_SIZE(vd));
 	abd_copy_from_buf(ub_abd, ub, sizeof (uberblock_t));
 
+	mmp->mmp_seq++;
 	mmp->mmp_kstat_id++;
 	mutex_exit(&mmp->mmp_io_lock);
 
@@ -443,38 +540,75 @@ mmp_thread(void *arg)
 {
 	spa_t *spa = (spa_t *)arg;
 	mmp_thread_t *mmp = &spa->spa_mmp;
-	boolean_t last_spa_suspended = spa_suspended(spa);
-	boolean_t last_spa_multihost = spa_multihost(spa);
+	boolean_t suspended = spa_suspended(spa);
+	boolean_t multihost = spa_multihost(spa);
+	uint64_t mmp_interval = MSEC2NSEC(MMP_INTERVAL_OK(
+	    zfs_multihost_interval));
+	uint32_t mmp_fail_intervals = MMP_FAIL_INTVS_OK(
+	    zfs_multihost_fail_intervals);
+	hrtime_t mmp_fail_ns = mmp_fail_intervals * mmp_interval;
+	boolean_t last_spa_suspended = suspended;
+	boolean_t last_spa_multihost = multihost;
+	uint64_t last_mmp_interval = mmp_interval;
+	uint32_t last_mmp_fail_intervals = mmp_fail_intervals;
+	hrtime_t last_mmp_fail_ns = mmp_fail_ns;
 	callb_cpr_t cpr;
-	hrtime_t max_fail_ns = zfs_multihost_fail_intervals *
-	    MSEC2NSEC(MAX(zfs_multihost_interval, MMP_MIN_INTERVAL));
+	int skip_wait = 0;
 
 	mmp_thread_enter(mmp, &cpr);
 
-	/*
-	 * The mmp_write_done() function calculates mmp_delay based on the
-	 * prior value of mmp_delay and the elapsed time since the last write.
-	 * For the first mmp write, there is no "last write", so we start
-	 * with fake, but reasonable, default non-zero values.
-	 */
-	mmp->mmp_delay = MSEC2NSEC(MAX(zfs_multihost_interval,
-	    MMP_MIN_INTERVAL)) / MAX(vdev_count_leaves(spa), 1);
-	mmp->mmp_last_write = gethrtime() - mmp->mmp_delay;
-
 	while (!mmp->mmp_thread_exiting) {
-		uint64_t mmp_fail_intervals = zfs_multihost_fail_intervals;
-		uint64_t mmp_interval = MSEC2NSEC(
-		    MAX(zfs_multihost_interval, MMP_MIN_INTERVAL));
-		boolean_t suspended = spa_suspended(spa);
-		boolean_t multihost = spa_multihost(spa);
-		hrtime_t next_time;
+		hrtime_t next_time = gethrtime() +
+		    MSEC2NSEC(MMP_DEFAULT_INTERVAL);
+		int leaves = MAX(vdev_count_leaves(spa), 1);
+
+		/* Detect changes in tunables or state */
+
+		last_spa_suspended = suspended;
+		last_spa_multihost = multihost;
+		suspended = spa_suspended(spa);
+		multihost = spa_multihost(spa);
+
+		last_mmp_interval = mmp_interval;
+		last_mmp_fail_intervals = mmp_fail_intervals;
+		last_mmp_fail_ns = mmp_fail_ns;
+		mmp_interval = MSEC2NSEC(MMP_INTERVAL_OK(
+		    zfs_multihost_interval));
+		mmp_fail_intervals = MMP_FAIL_INTVS_OK(
+		    zfs_multihost_fail_intervals);
+
+		/* Smooth so pool is not suspended when reducing tunables */
+		if (mmp_fail_intervals * mmp_interval < mmp_fail_ns) {
+			mmp_fail_ns = (mmp_fail_ns * 31 +
+			    mmp_fail_intervals * mmp_interval) / 32;
+		} else {
+			mmp_fail_ns = mmp_fail_intervals *
+			    mmp_interval;
+		}
+
+		if (mmp_interval != last_mmp_interval ||
+		    mmp_fail_intervals != last_mmp_fail_intervals) {
+			/*
+			 * We want other hosts to see new tunables as quickly as
+			 * possible.  Write out at higher frequency than usual.
+			 */
+			skip_wait += leaves;
+		}
 
 		if (multihost)
-			next_time = gethrtime() + mmp_interval /
-			    MAX(vdev_count_leaves(spa), 1);
-		else
-			next_time = gethrtime() +
-			    MSEC2NSEC(MMP_DEFAULT_INTERVAL);
+			next_time = gethrtime() + mmp_interval / leaves;
+
+		if (mmp_fail_ns != last_mmp_fail_ns) {
+			zfs_dbgmsg("MMP interval change pool '%s' "
+			    "gethrtime %llu last_mmp_interval %llu "
+			    "mmp_interval %llu last_mmp_fail_intervals %u "
+			    "mmp_fail_intervals %u mmp_fail_ns %llu "
+			    "skip_wait %d leaves %d next_time %llu",
+			    spa_name(spa), gethrtime(), last_mmp_interval,
+			    mmp_interval, last_mmp_fail_intervals,
+			    mmp_fail_intervals, mmp_fail_ns, skip_wait, leaves,
+			    next_time);
+		}
 
 		/*
 		 * MMP off => on, or suspended => !suspended:
@@ -483,8 +617,14 @@ mmp_thread(void *arg)
 		 */
 		if ((!last_spa_multihost && multihost) ||
 		    (last_spa_suspended && !suspended)) {
+			zfs_dbgmsg("MMP state change pool '%s': gethrtime %llu "
+			    "last_spa_multihost %u multihost %u "
+			    "last_spa_suspended %u suspended %u",
+			    spa_name(spa), last_spa_multihost, multihost,
+			    last_spa_suspended, suspended);
 			mutex_enter(&mmp->mmp_io_lock);
 			mmp->mmp_last_write = gethrtime();
+			mmp->mmp_delay = mmp_interval;
 			mutex_exit(&mmp->mmp_io_lock);
 		}
 
@@ -497,38 +637,39 @@ mmp_thread(void *arg)
 			mmp->mmp_delay = 0;
 			mutex_exit(&mmp->mmp_io_lock);
 		}
-		last_spa_multihost = multihost;
-		last_spa_suspended = suspended;
-
-		/*
-		 * Smooth max_fail_ns when its factors are decreased, because
-		 * making (max_fail_ns < mmp_interval) results in the pool being
-		 * immediately suspended before writes can occur at the new
-		 * higher frequency.
-		 */
-		if ((mmp_interval * mmp_fail_intervals) < max_fail_ns) {
-			max_fail_ns = ((31 * max_fail_ns) + (mmp_interval *
-			    mmp_fail_intervals)) / 32;
-		} else {
-			max_fail_ns = mmp_interval * mmp_fail_intervals;
-		}
 
 		/*
 		 * Suspend the pool if no MMP write has succeeded in over
 		 * mmp_interval * mmp_fail_intervals nanoseconds.
 		 */
-		if (!suspended && mmp_fail_intervals && multihost &&
-		    (gethrtime() - mmp->mmp_last_write) > max_fail_ns) {
+		if (multihost && !suspended && mmp_fail_intervals &&
+		    (gethrtime() - mmp->mmp_last_write) > mmp_fail_ns) {
+			zfs_dbgmsg("MMP suspending pool '%s': gethrtime %llu "
+			    "mmp_last_write %llu mmp_interval %llu "
+			    "mmp_fail_intervals %llu mmp_fail_ns %llu",
+			    spa_name(spa), (u_longlong_t)gethrtime(),
+			    (u_longlong_t)mmp->mmp_last_write,
+			    (u_longlong_t)mmp_interval,
+			    (u_longlong_t)mmp_fail_intervals,
+			    (u_longlong_t)mmp_fail_ns);
 			cmn_err(CE_WARN, "MMP writes to pool '%s' have not "
-			    "succeeded in over %llus; suspending pool",
+			    "succeeded in over %llu ms; suspending pool. "
+			    "Hrtime %llu",
 			    spa_name(spa),
-			    NSEC2SEC(gethrtime() - mmp->mmp_last_write));
+			    NSEC2MSEC(gethrtime() - mmp->mmp_last_write),
+			    gethrtime());
 			zio_suspend(spa, NULL, ZIO_SUSPEND_MMP);
 		}
 
 		if (multihost && !suspended)
 			mmp_write_uberblock(spa);
 
+		if (skip_wait > 0) {
+			next_time = gethrtime() + MSEC2NSEC(MMP_MIN_INTERVAL) /
+			    leaves;
+			skip_wait--;
+		}
+
 		CALLB_CPR_SAFE_BEGIN(&cpr);
 		(void) cv_timedwait_sig_hires(&mmp->mmp_thread_cv,
 		    &mmp->mmp_thread_lock, next_time, USEC2NSEC(1),