Illumos 4976-4984 - metaslab improvements

4976 zfs should only avoid writing to a failing non-redundant top-level vdev
4978 ztest fails in get_metaslab_refcount()
4979 extend free space histogram to device and pool
4980 metaslabs should have a fragmentation metric
4981 remove fragmented ops vector from block allocator
4982 space_map object should proactively upgrade when feature is enabled
4983 need to collect metaslab information via mdb
4984 device selection should use fragmentation metric
Reviewed by: Matthew Ahrens <[email protected]>
Reviewed by: Adam Leventhal <[email protected]>
Reviewed by: Christopher Siden <[email protected]>
Approved by: Garrett D'Amore <[email protected]>

References:
  https://www.illumos.org/issues/4976
  https://www.illumos.org/issues/4978
  https://www.illumos.org/issues/4979
  https://www.illumos.org/issues/4980
  https://www.illumos.org/issues/4981
  https://www.illumos.org/issues/4982
  https://www.illumos.org/issues/4983
  https://www.illumos.org/issues/4984
  https://github.com/illumos/illumos-gate/commit/2e4c998

Notes:
    The "zdb -M" option has been re-tasked to display the new metaslab
    fragmentation metric and the new "zdb -I" option is used to control
    the maximum number of in-flight I/Os.

    The new fragmentation metric is derived from the space map histogram
    which has been rolled up to the vdev and pool level and is presented
    to the user via "zpool list".

    Add a number of module parameters related to the new metaslab weighting
    logic.

Ported by: Tim Chase <[email protected]>
Signed-off-by: Brian Behlendorf <[email protected]>
Closes #2595

3 files changed, 121 insertions, 19 deletions

diff --git a/man/man5/zfs-module-parameters.5 b/man/man5/zfs-module-parameters.5
index 2f87f114c..d23e8022a 100644
--- a/man/man5/zfs-module-parameters.5
+++ b/man/man5/zfs-module-parameters.5
@@ -137,6 +137,18 @@ Default value: \fB8,388,608\fR.
 .sp
 .ne 2
 .na
+\fBmetaslab_bias_enabled\fR (int)
+.ad
+.RS 12n
+Enable metaslab group biasing based on its vdev's over- or under-utilization
+relative to the pool.
+.sp
+Use \fB1\fR for yes (default) and \fB0\fR for no.
+.RE
+
+.sp
+.ne 2
+.na
 \fBmetaslab_debug_load\fR (int)
 .ad
 .RS 12n
@@ -159,6 +171,41 @@ Use \fB1\fR for yes and \fB0\fR for no (default).
 .sp
 .ne 2
 .na
+\fBmetaslab_fragmentation_factor_enabled\fR (int)
+.ad
+.RS 12n
+Enable use of the fragmentation metric in computing metaslab weights.
+.sp
+Use \fB1\fR for yes (default) and \fB0\fR for no.
+.RE
+
+.sp
+.ne 2
+.na
+\fBmetaslab_preload_enabled\fR (int)
+.ad
+.RS 12n
+Enable metaslab group preloading.
+.sp
+Use \fB1\fR for yes (default) and \fB0\fR for no.
+.RE
+
+.sp
+.ne 2
+.na
+\fBmetaslab_lba_weighting_enabled\fR (int)
+.ad
+.RS 12n
+Give more weight to metaslabs with lower LBAs, assuming they have
+greater bandwidth as is typically the case on a modern constant
+angular velocity disk drive.
+.sp
+Use \fB1\fR for yes (default) and \fB0\fR for no.
+.RE
+
+.sp
+.ne 2
+.na
 \fBspa_config_path\fR (charp)
 .ad
 .RS 12n
@@ -769,6 +816,35 @@ Use \fB1\fR for yes and \fB0\fR for no (default).
 .sp
 .ne 2
 .na
+\fBzfs_metaslab_fragmentation_threshold\fR (int)
+.ad
+.RS 12n
+Allow metaslabs to keep their active state as long as their fragmentation
+percentage is less than or equal to this value. An active metaslab that
+exceeds this threshold will no longer keep its active status allowing
+better metaslabs to be selected.
+.sp
+Default value: \fB70\fR.
+.RE
+
+.sp
+.ne 2
+.na
+\fBzfs_mg_fragmentation_threshold\fR (int)
+.ad
+.RS 12n
+Metaslab groups are considered eligible for allocations if their
+fragmenation metric (measured as a percentage) is less than or equal to
+this value. If a metaslab group exceeds this threshold then it will be
+skipped unless all metaslab groups within the metaslab class have also
+crossed this threshold.
+.sp
+Default value: \fB85\fR.
+.RE
+
+.sp
+.ne 2
+.na
 \fBzfs_mg_noalloc_threshold\fR (int)
 .ad
 .RS 12n
diff --git a/man/man8/zdb.8 b/man/man8/zdb.8
index adde85f4a..48710c5cd 100644
--- a/man/man8/zdb.8
+++ b/man/man8/zdb.8
@@ -19,16 +19,16 @@
 \fBzdb\fR - Display zpool debugging and consistency information
 
 .SH "SYNOPSIS"
-\fBzdb\fR [-CumdibcsDvhLVXFPA] [-e [-p \fIpath\fR...]] [-t \fItxg\fR]
-    [-U \fIcache\fR] [-M \fIinflight I/Os\fR] [\fIpoolname\fR
-    [\fIobject\fR ...]]
+\fBzdb\fR [-CumdibcsDvhLMXFPA] [-e [-p \fIpath\fR...]] [-t \fItxg\fR]
+    [-U \fIcache\fR] [-I \fIinflight I/Os\fR]
+    [\fIpoolname\fR [\fIobject\fR ...]]
 
 .P
 \fBzdb\fR [-divPA] [-e [-p \fIpath\fR...]] [-U \fIcache\fR]
     \fIdataset\fR [\fIobject\fR ...]
 
 .P
-\fBzdb\fR -m [-LXFPA] [-t \fItxg\fR] [-e [-p \fIpath\fR...]] [-U \fIcache\fR]
+\fBzdb\fR -m [-MLXFPA] [-t \fItxg\fR] [-e [-p \fIpath\fR...]] [-U \fIcache\fR]
     \fIpoolname\fR [\fIvdev\fR [\fImetaslab\fR ...]]
 
 .P
@@ -194,6 +194,21 @@ verifies that all non-free blocks are referenced, which can be very expensive.
 .sp .6
 .RS 4n
 Display the offset, spacemap, and free space of each metaslab.
+When specified twice, also display information about the on-disk free
+space histogram associated with each metaslab. When specified three time,
+display the maximum contiguous free space, the in-core free space histogram,
+and the percentage of free space in each space map.  When specified
+four times display every spacemap record.
+.RE
+
+.sp
+.ne 2
+.na
+\fB-M\fR
+.ad
+.sp .6
+.RS 4n
+Display the offset, spacemap, and free space of each metaslab.
 When specified twice, also display information about the maximum contiguous
 free space and the percentage of free space in each space map.  When specified
 three times display every spacemap record.
@@ -366,7 +381,7 @@ transactions.
 .sp
 .ne 2
 .na
-\fB-M \fIinflight I/Os\fR \fR
+\fB-I \fIinflight I/Os\fR \fR
 .ad
 .sp .6
 .RS 4n
diff --git a/man/man8/zpool.8 b/man/man8/zpool.8
index 0b57bf6bb..4f81a229a 100644
--- a/man/man8/zpool.8
+++ b/man/man8/zpool.8
@@ -1,7 +1,7 @@
 '\" te
 .\" Copyright (c) 2007, Sun Microsystems, Inc. All Rights Reserved.
 .\" Copyright 2011 Nexenta Systems, Inc.  All rights reserved.
-.\" Copyright (c) 2012 by Delphix. All Rights Reserved.
+.\" Copyright (c) 2013 by Delphix. All rights reserved.
 .\" Copyright (c) 2012 Cyril Plisko. All Rights Reserved.
 .\" The contents of this file are subject to the terms of the Common Development
 .\" and Distribution License (the "License"). You may not use this file except
@@ -506,6 +506,17 @@ any space on an EFI labeled vdev which has not been brought online
 .ne 2
 .mk
 .na
+\fB\fBfragmentation\fR\fR
+.ad
+.RS 20n
+.rt
+The amount of fragmentation in the pool.
+.RE
+
+.sp
+.ne 2
+.mk
+.na
 \fB\fBfree\fR\fR
 .ad
 .RS 20n
@@ -1587,7 +1598,7 @@ Specify \fBu\fR for a printed representation of the internal representation of t
 .ad
 .RS 12n
 .rt  
-Comma-separated list of properties to display. See the "Properties" section for a list of valid properties. The default list is "name, size, used, available, expandsize, capacity, dedupratio, health, altroot"
+Comma-separated list of properties to display. See the "Properties" section for a list of valid properties. The default list is "name, size, used, available, fragmentation, expandsize, capacity, dedupratio, health, altroot"
 .RE
 
 .sp
@@ -2002,10 +2013,10 @@ The results from this command are similar to the following:
 .in +2
 .nf
 # \fBzpool list\fR
-     NAME    SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
-     rpool  19.9G  8.43G  11.4G         -    42%  1.00x  ONLINE  -
-     tank   61.5G  20.0G  41.5G         -    32%  1.00x  ONLINE  -
-     zion       -      -      -         -      -      -  FAULTED -
+NAME    SIZE  ALLOC   FREE   FRAG  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
+rpool  19.9G  8.43G  11.4G    33%         -    42%  1.00x  ONLINE  -
+tank   61.5G  20.0G  41.5G    48%         -    32%  1.00x  ONLINE  -
+zion       -      -      -      -         -      -      -  FAULTED -
 .fi
 .in -2
 .sp
@@ -2212,9 +2223,9 @@ The command to remove the mirrored log \fBmirror-2\fR is:
 \fBExample 15 \fRDisplaying expanded space on a device
 .sp
 .LP
-The following command dipslays the detailed information for the \fIdata\fR
+The following command displays the detailed information for the \fIdata\fR
 pool. This pool is comprised of a single \fIraidz\fR vdev where one of its
-devices increased its capacity by 1GB. In this example, the pool will not 
+devices increased its capacity by 10GB. In this example, the pool will not 
 be able to utilized this extra capacity until all the devices under the
 \fIraidz\fR vdev have been expanded.
 
@@ -2222,12 +2233,12 @@ be able to utilized this extra capacity until all the devices under the
 .in +2
 .nf
 # \fBzpool list -v data\fR
-   NAME         SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
-   data        17.9G   174K  17.9G         -     0%  1.00x  ONLINE  -
-     raidz1    17.9G   174K  17.9G         -
-       c4t2d0      -      -      -        1G
-       c4t3d0      -      -      -         -
-       c4t4d0      -      -      -         - 
+NAME         SIZE  ALLOC   FREE   FRAG  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
+data        23.9G  14.6G  9.30G    48%         -    61%  1.00x  ONLINE  -
+  raidz1    23.9G  14.6G  9.30G    48%         -
+    c1t1d0      -      -      -      -         -
+    c1t2d0      -      -      -      -       10G
+    c1t3d0      -      -      -      -         -
 .fi
 .in -2