Add device rebuild feature

The device_rebuild feature enables sequential reconstruction when
resilvering.  Mirror vdevs can be rebuilt in LBA order which may
more quickly restore redundancy depending on the pools average block
size, overall fragmentation and the performance characteristics
of the devices.  However, block checksums cannot be verified
as part of the rebuild thus a scrub is automatically started after
the sequential resilver completes.

The new '-s' option has been added to the `zpool attach` and
`zpool replace` command to request sequential reconstruction
instead of healing reconstruction when resilvering.

    zpool attach -s <pool> <existing vdev> <new vdev>
    zpool replace -s <pool> <old vdev> <new vdev>

The `zpool status` output has been updated to report the progress
of sequential resilvering in the same way as healing resilvering.
The one notable difference is that multiple sequential resilvers
may be in progress as long as they're operating on different
top-level vdevs.

The `zpool wait -t resilver` command was extended to wait on
sequential resilvers.  From this perspective they are no different
than healing resilvers.

Sequential resilvers cannot be supported for RAIDZ, but are
compatible with the dRAID feature being developed.

As part of this change the resilver_restart_* tests were moved
in to the functional/replacement directory.  Additionally, the
replacement tests were renamed and extended to verify both
resilvering and rebuilding.

Original-patch-by: Isaac Huang <[email protected]>
Reviewed-by: Tony Hutter <[email protected]>
Reviewed-by: John Poduska <[email protected]>
Co-authored-by: Mark Maybee <[email protected]>
Signed-off-by: Brian Behlendorf <[email protected]>
Closes #10349 

5 files changed, 93 insertions, 13 deletions

diff --git a/man/man5/zfs-module-parameters.5 b/man/man5/zfs-module-parameters.5
index 687b85d0b..3fbd3c67f 100644
--- a/man/man5/zfs-module-parameters.5
+++ b/man/man5/zfs-module-parameters.5
@@ -1865,6 +1865,30 @@ Default value: \fB1,000\fR.
 .sp
 .ne 2
 .na
+\fBzfs_vdev_rebuild_max_active\fR (int)
+.ad
+.RS 12n
+Maximum sequential resilver I/Os active to each device.
+See the section "ZFS I/O SCHEDULER".
+.sp
+Default value: \fB3\fR.
+.RE
+
+.sp
+.ne 2
+.na
+\fBzfs_vdev_rebuild_min_active\fR (int)
+.ad
+.RS 12n
+Minimum sequential resilver I/Os active to each device.
+See the section "ZFS I/O SCHEDULER".
+.sp
+Default value: \fB1\fR.
+.RE
+
+.sp
+.ne 2
+.na
 \fBzfs_vdev_removal_max_active\fR (int)
 .ad
 .RS 12n
@@ -2710,6 +2734,18 @@ Use \fB1\fR for yes and \fB0\fR for no (default).
 .sp
 .ne 2
 .na
+\fBzfs_rebuild_max_segment\fR (ulong)
+.ad
+.RS 12n
+Maximum read segment size to issue when sequentially resilvering a
+top-level vdev.
+.sp
+Default value: \fB1,048,576\fR.
+.RE
+
+.sp
+.ne 2
+.na
 \fBzfs_reconstruct_indirect_combinations_max\fR (int)
 .ad
 .RS 12na
diff --git a/man/man5/zpool-features.5 b/man/man5/zpool-features.5
index e7a61957f..3f690c334 100644
--- a/man/man5/zpool-features.5
+++ b/man/man5/zpool-features.5
@@ -258,6 +258,35 @@ returned to the \fBenabled\fR state when all bookmarks with these fields are des
 .sp
 .ne 2
 .na
+\fBdevice_rebuild\fR
+.ad
+.RS 4n
+.TS
+l l .
+GUID	org.openzfs:device_rebuild
+READ\-ONLY COMPATIBLE	yes
+DEPENDENCIES	none
+.TE
+
+This feature enables the ability for the \fBzpool attach\fR and \fBzpool
+replace\fR subcommands to perform sequential reconstruction (instead of
+healing reconstruction) when resilvering.
+
+Sequential reconstruction resilvers a device in LBA order without immediately
+verifying the checksums.  Once complete a scrub is started which then verifies
+the checksums.  This approach allows full redundancy to be restored to the pool
+in the minimum amount of time.  This two phase approach will take longer than a
+healing resilver when the time to verify the checksums is included.  However,
+unless there is additional pool damage no checksum errors should be reported
+by the scrub.  This feature is incompatible with raidz configurations.
+
+This feature becomes \fBactive\fR while a sequential resilver is in progress,
+and returns to \fBenabled\fR when the resilver completes.
+.RE
+
+.sp
+.ne 2
+.na
 \fBdevice_removal\fR
 .ad
 .RS 4n
diff --git a/man/man8/zpool-attach.8 b/man/man8/zpool-attach.8
index be0be4e07..585357b96 100644
--- a/man/man8/zpool-attach.8
+++ b/man/man8/zpool-attach.8
@@ -27,7 +27,7 @@
 .\" Copyright 2017 Nexenta Systems, Inc.
 .\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved.
 .\"
-.Dd August 9, 2019
+.Dd May 15, 2020
 .Dt ZPOOL-ATTACH 8
 .Os Linux
 .Sh NAME
@@ -36,7 +36,7 @@
 .Sh SYNOPSIS
 .Nm
 .Cm attach
-.Op Fl fw
+.Op Fl fsw
 .Oo Fl o Ar property Ns = Ns Ar value Oc
 .Ar pool device new_device
 .Sh DESCRIPTION
@@ -44,7 +44,7 @@
 .It Xo
 .Nm
 .Cm attach
-.Op Fl fw
+.Op Fl fsw
 .Oo Fl o Ar property Ns = Ns Ar value Oc
 .Ar pool device new_device
 .Xc
@@ -68,22 +68,29 @@ is part of a two-way mirror, attaching
 creates a three-way mirror, and so on.
 In either case,
 .Ar new_device
-begins to resilver immediately.
+begins to resilver immediately and any running scrub is cancelled.
 .Bl -tag -width Ds
 .It Fl f
 Forces use of
 .Ar new_device ,
 even if it appears to be in use.
 Not all devices can be overridden in this manner.
-.It Fl w
-Waits until
-.Ar new_device
-has finished resilvering before returning.
 .It Fl o Ar property Ns = Ns Ar value
 Sets the given pool properties. See the
 .Xr zpoolprops 8
 manual page for a list of valid properties that can be set. The only property
 supported at the moment is ashift.
+.It Fl s
+The
+.Ar new_device
+is reconstructed sequentially to restore redundancy as quickly as possible.
+Checksums are not verfied during sequential reconstruction so a scrub is
+started when the resilver completes.
+Sequential reconstruction is not supported for raidz configurations.
+.It Fl w
+Waits until
+.Ar new_device
+has finished resilvering before returning.
 .El
 .El
 .Sh SEE ALSO
diff --git a/man/man8/zpool-replace.8 b/man/man8/zpool-replace.8
index 933fb4ae9..5e639feaf 100644
--- a/man/man8/zpool-replace.8
+++ b/man/man8/zpool-replace.8
@@ -27,7 +27,7 @@
 .\" Copyright 2017 Nexenta Systems, Inc.
 .\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved.
 .\"
-.Dd August 9, 2019
+.Dd May 15, 2020
 .Dt ZPOOL-REPLACE 8
 .Os Linux
 .Sh NAME
@@ -36,7 +36,7 @@
 .Sh SYNOPSIS
 .Nm
 .Cm replace
-.Op Fl fw
+.Op Fl fsw
 .Oo Fl o Ar property Ns = Ns Ar value Oc
 .Ar pool Ar device Op Ar new_device
 .Sh DESCRIPTION
@@ -44,7 +44,7 @@
 .It Xo
 .Nm
 .Cm replace
-.Op Fl fw
+.Op Fl fsw
 .Op Fl o Ar property Ns = Ns Ar value
 .Ar pool Ar device Op Ar new_device
 .Xc
@@ -56,6 +56,7 @@ This is equivalent to attaching
 .Ar new_device ,
 waiting for it to resilver, and then detaching
 .Ar old_device .
+Any in progress scrub will be cancelled.
 .Pp
 The size of
 .Ar new_device
@@ -86,6 +87,13 @@ Sets the given pool properties. See the
 manual page for a list of valid properties that can be set.
 The only property supported at the moment is
 .Sy ashift .
+.It Fl s
+The
+.Ar new_device
+is reconstructed sequentially to restore redundancy as quickly as possible.
+Checksums are not verfied during sequential reconstruction so a scrub is
+started when the resilver completes.
+Sequential reconstruction is not supported for raidz configurations.
 .It Fl w
 Waits until the replacement has completed before returning.
 .El
diff --git a/man/man8/zpool-status.8 b/man/man8/zpool-status.8
index 7364bf635..66e335995 100644
--- a/man/man8/zpool-status.8
+++ b/man/man8/zpool-status.8
@@ -27,7 +27,7 @@
 .\" Copyright 2017 Nexenta Systems, Inc.
 .\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved.
 .\"
-.Dd August 9, 2019
+.Dd May 15, 2020
 .Dt ZPOOL-STATUS 8
 .Os Linux
 .Sh NAME
@@ -59,7 +59,7 @@ is specified, then the status of each pool in the system is displayed.
 For more information on pool and device health, see the
 .Em Device Failure and Recovery
 section of
-.Xr zpoolconcepts 8.
+.Xr zpoolconcepts 8 .
 .Pp
 If a scrub or resilver is in progress, this command reports the percentage done
 and the estimated time to completion.