Improve zpool status output, list all affected datasets

Currently, determining which datasets are affected by corruption is
a manual process.

The primary difficulty in reporting the list of affected snapshots is
that since the error was initially found, the snapshot where the error
originally occurred in, may have been deleted. To solve this issue, we
add the ID of the head dataset of the original snapshot which the error
was detected in, to the stored error report. Then any time a filesystem
is deleted, the errors associated with it are deleted as well. Any time
a clone promote occurs, we modify reports associated with the original
head to refer to the new head. The stored error reports are identified
by this head ID, the birth time of the block which the error occurred
in, as well as some information about the error itself are also stored.

Once this information is stored, we can find the set of datasets
affected by an error by walking back the list of snapshots in the given
head until we find one with the appropriate birth txg, and then traverse
through the snapshots of the clone family, terminating a branch if the
block was replaced in a given snapshot. Then we report this information
back to libzfs, and to the zpool status command, where it is displayed
as follows:

 pool: test
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 00:00:00 with 800 errors on Fri Dec  3
08:27:57 2021
config:

        NAME        STATE     READ WRITE CKSUM
        test        ONLINE       0     0     0
          sdb       ONLINE       0     0 1.58K

errors: Permanent errors have been detected in the following files:

        test@1:/test.0.0
        /test/test.0.0
        /test/1clone/test.0.0

A new feature flag is introduced to mark the presence of this change, as
well as promotion and backwards compatibility logic. This is an updated
version of #9175. Rebase required fixing the tests, updating the ABI of
libzfs, updating the man pages, fixing bugs, fixing the error returns,
and updating the old on-disk error logs to the new format when
activating the feature.

Reviewed-by: Matthew Ahrens <[email protected]>
Reviewed-by: Brian Behlendorf <[email protected]>
Reviewed-by: Mark Maybee <[email protected]>
Reviewed-by: Tony Hutter <[email protected]>
Co-authored-by: TulsiJain <[email protected]>
Signed-off-by: George Amanakis <[email protected]>
Closes #9175
Closes #12812

5 files changed, 156 insertions, 1 deletions

diff --git a/tests/runfiles/common.run b/tests/runfiles/common.run
index 1cd439c43..b8310264a 100644
--- a/tests/runfiles/common.run
+++ b/tests/runfiles/common.run
@@ -488,6 +488,7 @@ tags = ['functional', 'cli_root', 'zpool_split']
 
 [tests/functional/cli_root/zpool_status]
 tests = ['zpool_status_001_pos', 'zpool_status_002_pos',
+    'zpool_status_003_pos', 'zpool_status_004_pos',
     'zpool_status_features_001_pos']
 tags = ['functional', 'cli_root', 'zpool_status']
 
diff --git a/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg b/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg
index fac96e26e..4ea5725e0 100644
--- a/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg
+++ b/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg
@@ -98,5 +98,6 @@ if is_linux || is_freebsd; then
 	    "feature@livelist"
 	    "feature@zstd_compress"
 	    "feature@zilsaxattr"
+	    "feature@head_errlog"
 	)
-fi
\ No newline at end of file
+fi
diff --git a/tests/zfs-tests/tests/functional/cli_root/zpool_status/Makefile.am b/tests/zfs-tests/tests/functional/cli_root/zpool_status/Makefile.am
index 5553061c6..538c5d6c1 100644
--- a/tests/zfs-tests/tests/functional/cli_root/zpool_status/Makefile.am
+++ b/tests/zfs-tests/tests/functional/cli_root/zpool_status/Makefile.am
@@ -4,4 +4,6 @@ dist_pkgdata_SCRIPTS = \
 	cleanup.ksh \
 	zpool_status_001_pos.ksh \
 	zpool_status_002_pos.ksh \
+	zpool_status_003_pos.ksh \
+	zpool_status_004_pos.ksh \
 	zpool_status_features_001_pos.ksh
diff --git a/tests/zfs-tests/tests/functional/cli_root/zpool_status/zpool_status_003_pos.ksh b/tests/zfs-tests/tests/functional/cli_root/zpool_status/zpool_status_003_pos.ksh
new file mode 100755
index 000000000..e0c2ed669
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/cli_root/zpool_status/zpool_status_003_pos.ksh
@@ -0,0 +1,70 @@
+#!/bin/ksh -p
+#
+# CDDL HEADER START
+#
+# The contents of this file are subject to the terms of the
+# Common Development and Distribution License (the "License").
+# You may not use this file except in compliance with the License.
+#
+# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+# or http://www.opensolaris.org/os/licensing.
+# See the License for the specific language governing permissions
+# and limitations under the License.
+#
+# When distributing Covered Code, include this CDDL HEADER in each
+# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+# If applicable, add the following below this CDDL HEADER, with the
+# fields enclosed by brackets "[]" replaced with your own identifying
+# information: Portions Copyright [yyyy] [name of copyright owner]
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2019, Delphix. All rights reserved.
+# Copyright (c) 2021, George Amanakis. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+
+#
+# DESCRIPTION:
+# Verify correct output with 'zpool status -v' after corrupting a file
+#
+# STRATEGY:
+# 1. Create a pool and a file
+# 2. zinject checksum errors
+# 3. Read the file
+# 4. Take a snapshot and make a clone
+# 5. Verify we see "snapshot, clone and filesystem" output in 'zpool status -v'
+
+function cleanup
+{
+	log_must zinject -c all
+	datasetexists $TESTPOOL2 && log_must zpool destroy $TESTPOOL2
+	rm -f $TESTDIR/vdev_a
+}
+
+verify_runnable "both"
+
+log_assert "Verify correct 'zpool status -v' output with a corrupted file"
+log_onexit cleanup
+
+truncate -s $MINVDEVSIZE $TESTDIR/vdev_a
+log_must zpool create -f $TESTPOOL2 $TESTDIR/vdev_a
+
+log_must fio --rw=write --name=job --size=10M --filename=/$TESTPOOL2/10m_file
+log_must zinject -t data -e checksum -f 100 -am /$TESTPOOL2/10m_file
+
+# Try to read the 2nd megabyte of 10m_file
+dd if=/$TESTPOOL2/10m_file bs=1M || true
+
+log_must zfs snapshot $TESTPOOL2@snap
+log_must zfs clone $TESTPOOL2@snap $TESTPOOL2/clone
+
+# Look to see that snapshot, clone and filesystem our files report errors
+log_must eval "zpool status -v | grep '$TESTPOOL2@snap:/10m_file'"
+log_must eval "zpool status -v | grep '$TESTPOOL2/clone/10m_file'"
+log_must eval "zpool status -v | grep '$TESTPOOL2/10m_file'"
+
+log_pass "'zpool status -v' outputs affected filesystem, snapshot & clone"
diff --git a/tests/zfs-tests/tests/functional/cli_root/zpool_status/zpool_status_004_pos.ksh b/tests/zfs-tests/tests/functional/cli_root/zpool_status/zpool_status_004_pos.ksh
new file mode 100755
index 000000000..6d8571950
--- /dev/null
+++ b/tests/zfs-tests/tests/functional/cli_root/zpool_status/zpool_status_004_pos.ksh
@@ -0,0 +1,81 @@
+#!/bin/ksh -p
+#
+# CDDL HEADER START
+#
+# The contents of this file are subject to the terms of the
+# Common Development and Distribution License (the "License").
+# You may not use this file except in compliance with the License.
+#
+# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+# or http://www.opensolaris.org/os/licensing.
+# See the License for the specific language governing permissions
+# and limitations under the License.
+#
+# When distributing Covered Code, include this CDDL HEADER in each
+# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+# If applicable, add the following below this CDDL HEADER, with the
+# fields enclosed by brackets "[]" replaced with your own identifying
+# information: Portions Copyright [yyyy] [name of copyright owner]
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2019, by Delphix. All rights reserved.
+# Copyright (c) 2021, George Amanakis. All rights reserved.
+#
+
+. $STF_SUITE/include/libtest.shlib
+
+#
+# DESCRIPTION:
+# Verify feature@head_errlog=disabled works.
+#
+# STRATEGY:
+# 1. Create a pool with feature@head_errlog=disabled and a file
+# 2. zinject checksum errors
+# 3. Read the file
+# 4. Take a snapshot and make a clone
+# 5. Verify that zpool status displays the old behaviour.
+
+function cleanup
+{
+	log_must zinject -c all
+	datasetexists $TESTPOOL2 && log_must zpool destroy $TESTPOOL2
+	rm -f $TESTDIR/vdev_a
+}
+
+verify_runnable "both"
+
+log_assert "Verify 'zpool status -v' with feature@head_errlog=disabled works"
+log_onexit cleanup
+
+truncate -s $MINVDEVSIZE $TESTDIR/vdev_a
+log_must zpool create -f -o feature@head_errlog=disabled $TESTPOOL2 $TESTDIR/vdev_a
+
+state=$(zpool list -Ho feature@head_errlog $TESTPOOL2)
+if [[ "$state" != "disabled" ]]; then
+	log_fail "head_errlog has state $state"
+fi
+
+log_must fio --rw=write --name=job --size=10M --filename=/$TESTPOOL2/10m_file
+log_must zinject -t data -e checksum -f 100 -am /$TESTPOOL2/10m_file
+
+# Try to read the file
+dd if=/$TESTPOOL2/10m_file bs=1M || true
+
+log_must zfs snapshot $TESTPOOL2@snap
+log_must zfs clone $TESTPOOL2@snap $TESTPOOL2/clone
+
+# Check that snapshot and clone do not report the error.
+log_mustnot eval "zpool status -v | grep '$TESTPOOL2@snap:/10m_file'"
+log_mustnot eval "zpool status -v | grep '$TESTPOOL2/clone/10m_file'"
+log_must eval "zpool status -v | grep '$TESTPOOL2/10m_file'"
+
+# Check that enabling the feature reports the error properly.
+log_must zpool set feature@head_errlog=enabled $TESTPOOL2
+log_must eval "zpool status -v | grep '$TESTPOOL2@snap:/10m_file'"
+log_must eval "zpool status -v | grep '$TESTPOOL2/clone/10m_file'"
+log_must eval "zpool status -v | grep '$TESTPOOL2/10m_file'"
+
+log_pass "'zpool status -v' with feature@head_errlog=disabled works"