Split functional testings via github action matrix

This commit changes the workflow of the github actions.

We split the workflow into different parts:

1) build zfs modules for Ubuntu 20.04 and 22.04 (~25m)
2) 2x zloop test (~10m) + 2x sanity test (~25m)
3) functional testings in parts 1..5 (each ~1h)
   - these could be triggered, when sanity tests are ok
   - currently I just start them all in the same time
4) cleanup and create summary

When everything is fine, the full run with all testings
should be done in around 2 hours.

The codeql.yml and checkstyle.yml are not part in this circle.

The testings are also modified a bit:
- report info about CPU and checksum benchmarks
- reset the debugging logs for each test
  - when some error occurred, we call dmesg with -c to get
    only the log output for the last failed test
  - we empty also the dbgsys

Reviewed-by: George Melikov <[email protected]>
Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: Tino Reichardt <[email protected]>
Closes #14078 

10 files changed, 471 insertions, 256 deletions

diff --git a/.github/workflows/README.md b/.github/workflows/README.md
new file mode 100644
index 000000000..8255dd210
--- /dev/null
+++ b/.github/workflows/README.md
@@ -0,0 +1,51 @@
+
+## The testings are done this way
+
+```mermaid
+flowchart TB
+subgraph CleanUp and Summary
+  Part1-20.04-->CleanUp+nice+Summary
+  Part2-20.04-->CleanUp+nice+Summary
+  PartN-20.04-->CleanUp+nice+Summary
+  Part1-22.04-->CleanUp+nice+Summary
+  Part2-22.04-->CleanUp+nice+Summary
+  PartN-22.04-->CleanUp+nice+Summary
+end
+
+subgraph Functional Testings
+  functional-testing-20.04-->Part1-20.04
+  functional-testing-20.04-->Part2-20.04
+  functional-testing-20.04-->PartN-20.04
+  functional-testing-22.04-->Part1-22.04
+  functional-testing-22.04-->Part2-22.04
+  functional-testing-22.04-->PartN-22.04
+end
+
+subgraph Sanity and zloop Testings
+  sanity-checks-20.04-->functional-testing-20.04
+  sanity-checks-22.04-->functional-testing-22.04
+  zloop-checks-20.04-->functional
+  zloop-checks-22.04-->functional
+end
+
+subgraph Code Checking + Building
+  codeql.yml
+  checkstyle.yml
+  Build-Ubuntu-20.04-->sanity-checks-20.04
+  Build-Ubuntu-22.04-->sanity-checks-22.04
+  Build-Ubuntu-20.04-->zloop-checks-20.04
+  Build-Ubuntu-22.04-->zloop-checks-22.04
+end
+```
+
+
+1) build zfs modules for Ubuntu 20.04 and 22.04 (~15m)
+2) 2x zloop test (~10m) + 2x sanity test (~25m)
+3) functional testings in parts 1..5 (each ~1h)
+4) cleanup and create summary
+   - content of summary depends on the results of the steps
+
+When everything runs fine, the full run should be done in
+about 2 hours.
+
+The codeql.yml and checkstyle.yml are not part in this circle.
diff --git a/.github/workflows/scripts/generate-summary.sh b/.github/workflows/scripts/generate-summary.sh
new file mode 100755
index 000000000..cd5ea3421
--- /dev/null
+++ b/.github/workflows/scripts/generate-summary.sh
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+
+# for runtime reasons we split functional testings into N parts
+# - use a define to check for missing tarfiles
+FUNCTIONAL_PARTS="4"
+
+ZTS_REPORT="tests/test-runner/bin/zts-report.py"
+chmod +x $ZTS_REPORT
+
+function output() {
+  echo -e $* >> Summary.md
+}
+
+function error() {
+  output ":bangbang: $* :bangbang:\n"
+}
+
+# this function generates the real summary
+# - expects a logfile "log" in current directory
+function generate() {
+  # we issued some error already
+  test ! -s log && return
+
+  # for overview and zts-report
+  cat log | grep '^Test' > list
+
+  # error details
+  awk '/\[FAIL\]|\[KILLED\]/{ show=1; print; next; }
+    /\[SKIP\]|\[PASS\]/{ show=0; } show' log > err
+
+  # summary of errors
+  if [ -s err ]; then
+    output "<pre>"
+    $ZTS_REPORT --no-maybes ./list >> Summary.md
+    output "</pre>"
+
+    # generate seperate error logfile
+    ERRLOGS=$((ERRLOGS+1))
+    errfile="err-$ERRLOGS.md"
+    echo -e "\n## $headline (debugging)\n" >> $errfile
+    echo "<details><summary>Error Listing - with dmesg and dbgmsg</summary><pre>" >> $errfile
+    dd if=err bs=999k count=1 >> $errfile
+    echo "</pre></details>" >> $errfile
+  else
+    output "All tests passed :thumbsup:"
+  fi
+
+  output "<details><summary>Full Listing</summary><pre>"
+  cat list >> Summary.md
+  output "</pre></details>"
+
+  # remove tmp files
+  rm -f err list log
+}
+
+# check tarfiles and untar
+function check_tarfile() {
+  if [ -f "$1" ]; then
+    tar xf "$1" || error "Tarfile $1 returns some error"
+  else
+    error "Tarfile $1 not found"
+  fi
+}
+
+# check logfile and concatenate test results
+function check_logfile() {
+  if [ -f "$1" ]; then
+    cat "$1" >> log
+  else
+    error "Logfile $1 not found"
+  fi
+}
+
+# sanity
+function summarize_s() {
+  headline="$1"
+  output "\n## $headline\n"
+  rm -rf testfiles
+  check_tarfile "$2/sanity.tar"
+  check_logfile "testfiles/log"
+  generate
+}
+
+# functional
+function summarize_f() {
+  headline="$1"
+  output "\n## $headline\n"
+  rm -rf testfiles
+  for i in $(seq 1 $FUNCTIONAL_PARTS); do
+    tarfile="$2/part$i.tar"
+    check_tarfile "$tarfile"
+    check_logfile "testfiles/log"
+  done
+  generate
+}
+
+# https://docs.github.com/en/[email protected]/actions/using-workflows/workflow-commands-for-github-actions#step-isolation-and-limits
+# Job summaries are isolated between steps and each step is restricted to a maximum size of 1MiB.
+# [ ] can not show all error findings here
+# [x] split files into smaller ones and create additional steps
+
+ERRLOGS=0
+if [ ! -f Summary/Summary.md ]; then
+  # first call, we do the default summary (~500k)
+  echo -n > Summary.md
+  summarize_s "Sanity Tests Ubuntu 20.04" Logs-20.04-sanity
+  summarize_s "Sanity Tests Ubuntu 22.04" Logs-22.04-sanity
+  summarize_f "Functional Tests Ubuntu 20.04" Logs-20.04-functional
+  summarize_f "Functional Tests Ubuntu 22.04" Logs-22.04-functional
+
+  cat Summary.md >> $GITHUB_STEP_SUMMARY
+  mkdir -p Summary
+  mv *.md Summary
+else
+  # here we get, when errors where returned in first call
+  test -f Summary/err-$1.md && cat Summary/err-$1.md >> $GITHUB_STEP_SUMMARY
+fi
+
+exit 0
diff --git a/.github/workflows/scripts/reclaim_disk_space.sh b/.github/workflows/scripts/reclaim_disk_space.sh
deleted file mode 100755
index ed23ce31d..000000000
--- a/.github/workflows/scripts/reclaim_disk_space.sh
+++ /dev/null
@@ -1,23 +0,0 @@
-#!/bin/sh
-
-set -eu
-
-# remove 4GiB of images
-sudo systemd-run docker system prune --force --all --volumes
-
-# remove unused software
-sudo systemd-run --wait rm -rf \
-  "$AGENT_TOOLSDIRECTORY" \
-  /opt/* \
-  /usr/local/* \
-  /usr/share/az* \
-  /usr/share/dotnet \
-  /usr/share/gradle* \
-  /usr/share/miniconda \
-  /usr/share/swift \
-  /var/lib/gems \
-  /var/lib/mysql \
-  /var/lib/snapd
-
-# trim the cleaned space
-sudo fstrim /
diff --git a/.github/workflows/scripts/setup-dependencies.sh b/.github/workflows/scripts/setup-dependencies.sh
new file mode 100755
index 000000000..c788f0afa
--- /dev/null
+++ b/.github/workflows/scripts/setup-dependencies.sh
@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+
+set -eu
+
+function prerun() {
+  echo "::group::Install build dependencies"
+  # remove snap things, update+upgrade will be faster then
+  for x in lxd core20 snapd; do sudo snap remove $x; done
+  sudo apt-get purge snapd google-chrome-stable firefox
+  # https://github.com/orgs/community/discussions/47863
+  sudo apt-get remove grub-efi-amd64-bin grub-efi-amd64-signed shim-signed --allow-remove-essential
+  sudo apt-get update
+  sudo apt upgrade
+  sudo xargs --arg-file=.github/workflows/build-dependencies.txt apt-get install -qq
+  sudo apt-get clean
+  sudo dmesg -c > /var/tmp/dmesg-prerun
+  echo "::endgroup::"
+}
+
+function mod_build() {
+  echo "::group::Generate debian packages"
+  ./autogen.sh
+  ./configure --enable-debug --enable-debuginfo --enable-asan --enable-ubsan
+  make --no-print-directory --silent native-deb-utils native-deb-kmod
+  mv ../*.deb .
+  rm ./openzfs-zfs-dracut*.deb ./openzfs-zfs-dkms*.deb
+  echo "$ImageOS-$ImageVersion" > tests/ImageOS.txt
+  echo "::endgroup::"
+}
+
+function mod_install() {
+  # install the pre-built module only on the same runner image
+  MOD=`cat tests/ImageOS.txt`
+  if [ "$MOD" != "$ImageOS-$ImageVersion" ]; then
+    rm -f *.deb
+    mod_build
+  fi
+
+  echo "::group::Install and load modules"
+  # delete kernel-shipped zfs modules, be sure about correct modules
+  sudo sed -i.bak 's/updates/extra updates/' /etc/depmod.d/ubuntu.conf
+  sudo apt-get install --fix-missing ./*.deb
+
+  # Native Debian packages enable and start the services
+  # Stop zfs-zed daemon, as it may interfere with some ZTS test cases
+  sudo systemctl stop zfs-zed
+  sudo depmod -a
+  sudo modprobe zfs
+  sudo dmesg
+  sudo dmesg -c > /var/tmp/dmesg-module-load
+  echo "::endgroup::"
+
+  echo "::group::Report CPU information"
+  lscpu
+  cat /proc/spl/kstat/zfs/chksum_bench
+  echo "::endgroup::"
+
+  echo "::group::Reclaim and report disk space"
+  # remove 4GiB of images
+  sudo systemd-run docker system prune --force --all --volumes
+
+  # remove unused software
+  sudo systemd-run --wait rm -rf \
+    "$AGENT_TOOLSDIRECTORY" \
+    /opt/* \
+    /usr/local/* \
+    /usr/share/az* \
+    /usr/share/dotnet \
+    /usr/share/gradle* \
+    /usr/share/miniconda \
+    /usr/share/swift \
+    /var/lib/gems \
+    /var/lib/mysql \
+    /var/lib/snapd
+
+  # trim the cleaned space
+  sudo fstrim /
+
+  # disk usage afterwards
+  df -h /
+  echo "::endgroup::"
+}
+
+case "$1" in
+  build)
+    prerun
+    mod_build
+    ;;
+  tests)
+    prerun
+    mod_install
+    ;;
+esac
diff --git a/.github/workflows/scripts/setup-functional.sh b/.github/workflows/scripts/setup-functional.sh
new file mode 100755
index 000000000..08c4d872a
--- /dev/null
+++ b/.github/workflows/scripts/setup-functional.sh
@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+
+set -eu
+
+TDIR="/usr/share/zfs/zfs-tests/tests/functional"
+echo -n "TODO="
+case "$1" in
+  part1)
+    # ~1h 20m
+    echo "cli_root"
+    ;;
+  part2)
+    # ~1h
+    ls $TDIR|grep '^[a-m]'|grep -v "cli_root"|xargs|tr -s ' ' ','
+    ;;
+  part3)
+    # ~1h
+    ls $TDIR|grep '^[n-qs-z]'|xargs|tr -s ' ' ','
+    ;;
+  part4)
+    # ~1h
+    ls $TDIR|grep '^r'|xargs|tr -s ' ' ','
+    ;;
+esac
diff --git a/.github/workflows/zfs-linux-tests.yml b/.github/workflows/zfs-linux-tests.yml
new file mode 100644
index 000000000..b76c634e7
--- /dev/null
+++ b/.github/workflows/zfs-linux-tests.yml
@@ -0,0 +1,113 @@
+name: zfs-linux-tests
+
+on:
+  workflow_call:
+    inputs:
+      os:
+        description: 'The ubuntu version: 20.02 or 22.04'
+        required: true
+        type: string
+
+jobs:
+
+  zloop:
+    runs-on: ubuntu-${{ inputs.os }}
+    steps:
+    - uses: actions/download-artifact@v3
+      with:
+        name: modules-${{ inputs.os }}
+    - name: Install modules
+      run: |
+        tar xzf modules-${{ inputs.os }}.tgz
+        .github/workflows/scripts/setup-dependencies.sh tests
+    - name: Tests
+      timeout-minutes: 30
+      run: |
+        sudo mkdir -p /var/tmp/zloop
+        # run for 10 minutes or at most 2 iterations for a maximum runner
+        #   time of 20 minutes.
+        sudo /usr/share/zfs/zloop.sh -t 600 -I 2 -l -m1 -- -T 120 -P 60
+    - name: Prepare artifacts
+      if: failure()
+      run: |
+        sudo chmod +r -R /var/tmp/zloop/
+    - uses: actions/upload-artifact@v3
+      if: failure()
+      with:
+        name: Zpool-logs-${{ inputs.os }}
+        path: |
+          /var/tmp/zloop/*/
+          !/var/tmp/zloop/*/vdev/
+        if-no-files-found: ignore
+    - uses: actions/upload-artifact@v3
+      if: failure()
+      with:
+        name: Zpool-files-${{ inputs.os }}
+        path: |
+          /var/tmp/zloop/*/vdev/
+        if-no-files-found: ignore
+
+  sanity:
+    runs-on: ubuntu-${{ inputs.os }}
+    steps:
+    - uses: actions/download-artifact@v3
+      with:
+        name: modules-${{ inputs.os }}
+    - name: Install modules
+      run: |
+        tar xzf modules-${{ inputs.os }}.tgz
+        .github/workflows/scripts/setup-dependencies.sh tests
+    - name: Tests
+      timeout-minutes: 60
+      shell: bash
+      run: |
+        set -o pipefail
+        /usr/share/zfs/zfs-tests.sh -vKR -s 3G -r sanity | scripts/zfs-tests-color.sh
+    - name: Prepare artifacts
+      if: success() || failure()
+      run: |
+        RESPATH="/var/tmp/test_results"
+        mv -f $RESPATH/current $RESPATH/testfiles
+        tar cf $RESPATH/sanity.tar -h -C $RESPATH testfiles
+    - uses: actions/upload-artifact@v3
+      if: success() || failure()
+      with:
+        name: Logs-${{ inputs.os }}-sanity
+        path: /var/tmp/test_results/sanity.tar
+        if-no-files-found: ignore
+
+  functional:
+    runs-on: ubuntu-${{ inputs.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        tests: [ part1, part2, part3, part4 ]
+    steps:
+    - uses: actions/download-artifact@v3
+      with:
+        name: modules-${{ inputs.os }}
+    - name: Install modules
+      run: |
+        tar xzf modules-${{ inputs.os }}.tgz
+        .github/workflows/scripts/setup-dependencies.sh tests
+    - name: Setup tests
+      run: |
+        .github/workflows/scripts/setup-functional.sh ${{ matrix.tests }} >> $GITHUB_ENV
+    - name: Tests
+      timeout-minutes: 120
+      shell: bash
+      run: |
+        set -o pipefail
+        /usr/share/zfs/zfs-tests.sh -vKR -s 3G -T ${{ env.TODO }} | scripts/zfs-tests-color.sh
+    - name: Prepare artifacts
+      if: success() || failure()
+      run: |
+        RESPATH="/var/tmp/test_results"
+        mv -f $RESPATH/current $RESPATH/testfiles
+        tar cf $RESPATH/${{ matrix.tests }}.tar -h -C $RESPATH testfiles
+    - uses: actions/upload-artifact@v3
+      if: success() || failure()
+      with:
+        name: Logs-${{ inputs.os }}-functional
+        path: /var/tmp/test_results/${{ matrix.tests }}.tar
+        if-no-files-found: ignore
diff --git a/.github/workflows/zfs-linux.yml b/.github/workflows/zfs-linux.yml
new file mode 100644
index 000000000..4681bea50
--- /dev/null
+++ b/.github/workflows/zfs-linux.yml
@@ -0,0 +1,71 @@
+name: zfs-linux
+
+on:
+  push:
+  pull_request:
+
+jobs:
+
+  build:
+    name: Build
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [20.04, 22.04]
+    runs-on: ubuntu-${{ matrix.os }}
+    steps:
+    - uses: actions/checkout@v3
+      with:
+        ref: ${{ github.event.pull_request.head.sha }}
+    - name: Build modules
+      run: .github/workflows/scripts/setup-dependencies.sh build
+    - name: Prepare modules upload
+      run: tar czf modules-${{ matrix.os }}.tgz *.deb .github scripts/zfs-tests-color.sh tests/test-runner tests/ImageOS.txt
+    - uses: actions/upload-artifact@v3
+      with:
+        name: modules-${{ matrix.os }}
+        path: modules-${{ matrix.os }}.tgz
+    - name: Prepare scripts upload
+      run: tar czf scripts.tgz .github tests/test-runner
+    - uses: actions/upload-artifact@v3
+      with:
+        name: scripts
+        path: scripts.tgz
+
+  testings:
+    name: Testing
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [20.04, 22.04]
+    needs: build
+    uses: ./.github/workflows/zfs-linux-tests.yml
+    with:
+      os: ${{ matrix.os }}
+
+  cleanup:
+    if: always()
+    name: Cleanup
+    runs-on: ubuntu-22.04
+    needs: testings
+    steps:
+    - uses: actions/download-artifact@v3
+    - name: Generating summary
+      run: |
+        tar xzf scripts/scripts.tgz .github tests
+        .github/workflows/scripts/generate-summary.sh
+    # up to 4 steps, each can have 1 MiB output (for debugging log files)
+    - run: .github/workflows/scripts/generate-summary.sh 1
+    - run: .github/workflows/scripts/generate-summary.sh 2
+    - run: .github/workflows/scripts/generate-summary.sh 3
+    - run: .github/workflows/scripts/generate-summary.sh 4
+    - uses: actions/upload-artifact@v3
+      with:
+        name: Summary Files
+        path: Summary/
+    - uses: geekyeggo/delete-artifact@v2
+      with:
+        name: modules-20.04
+    - uses: geekyeggo/delete-artifact@v2
+      with:
+        name: modules-22.04
diff --git a/.github/workflows/zfs-tests-functional.yml b/.github/workflows/zfs-tests-functional.yml
deleted file mode 100644
index 237a0cb73..000000000
--- a/.github/workflows/zfs-tests-functional.yml
+++ /dev/null
@@ -1,86 +0,0 @@
-name: zfs-tests-functional
-
-on:
-  push:
-  pull_request:
-
-jobs:
-  tests-functional-ubuntu:
-    strategy:
-      fail-fast: false
-      matrix:
-        os: [20.04, 22.04]
-    runs-on: ubuntu-${{ matrix.os }}
-    steps:
-    - uses: actions/checkout@v3
-      with:
-        ref: ${{ github.event.pull_request.head.sha }}
-    - name: Install dependencies
-      run: |
-        # https://github.com/orgs/community/discussions/47863
-        sudo apt-mark hold grub-efi-amd64-signed
-        sudo apt-get update --fix-missing
-        sudo apt-get upgrade
-        sudo xargs --arg-file=${{ github.workspace }}/.github/workflows/build-dependencies.txt apt-get install -qq
-        sudo apt-get clean
-    - name: Autogen.sh
-      run: |
-        ./autogen.sh
-    - name: Configure
-      run: |
-        ./configure --enable-debug --enable-debuginfo --enable-asan --enable-ubsan
-    - name: Make
-      run: |
-        make --no-print-directory --silent native-deb-utils native-deb-kmod
-        mv ../*.deb .
-        rm ./openzfs-zfs-dkms*.deb ./openzfs-zfs-dracut*.deb
-    - name: Install
-      run: |
-        # Update order of directories to search for modules, otherwise
-        #   Ubuntu will load kernel-shipped ones.
-        sudo sed -i.bak 's/updates/extra updates/' /etc/depmod.d/ubuntu.conf
-        sudo dpkg -i *.deb
-        # Native Debian packages enable and start the services
-        # Stop zfs-zed daemon, as it may interfere with some ZTS test cases
-        sudo systemctl stop zfs-zed
-        # Workaround for cloud-init bug
-        # see https://github.com/openzfs/zfs/issues/12644
-        FILE=/lib/udev/rules.d/10-cloud-init-hook-hotplug.rules
-        if [ -r "${FILE}" ]; then
-          HASH=$(md5sum "${FILE}" | awk '{ print $1 }')
-          if [ "${HASH}" = "121ff0ef1936cd2ef65aec0458a35772" ]; then
-            # Just shove a zd* exclusion right above the hotplug hook...
-            sudo sed -i -e s/'LABEL="cloudinit_hook"'/'KERNEL=="zd*", GOTO="cloudinit_end"\n&'/ "${FILE}"
-            sudo udevadm control --reload-rules
-          fi
-        fi
-    - name: Clear the kernel ring buffer
-      run: |
-        sudo dmesg -c >/var/tmp/dmesg-prerun
-    - name: Reclaim and report disk space
-      run: |
-        ${{ github.workspace }}/.github/workflows/scripts/reclaim_disk_space.sh
-        df -h /
-    - name: Tests
-      run: |
-        set -o pipefail
-        /usr/share/zfs/zfs-tests.sh -vKR -s 3G | scripts/zfs-tests-color.sh
-      shell: bash
-      timeout-minutes: 330
-    - name: Prepare artifacts
-      if: failure()
-      run: |
-        RESULTS_PATH=$(readlink -f /var/tmp/test_results/current)
-        sudo dmesg > $RESULTS_PATH/dmesg
-        sudo cp /var/log/syslog /var/tmp/dmesg-prerun $RESULTS_PATH/
-        sudo chmod +r $RESULTS_PATH/*
-        # Replace ':' in dir names, actions/upload-artifact doesn't support it
-        for f in $(find /var/tmp/test_results -name '*:*'); do mv "$f" "${f//:/__}"; done
-    - uses: actions/upload-artifact@v3
-      if: failure()
-      with:
-        name: Test logs Ubuntu-${{ matrix.os }}
-        path: |
-          /var/tmp/test_results/*
-          !/var/tmp/test_results/current
-        if-no-files-found: ignore
diff --git a/.github/workflows/zfs-tests-sanity.yml b/.github/workflows/zfs-tests-sanity.yml
deleted file mode 100644
index ef2c0a869..000000000
--- a/.github/workflows/zfs-tests-sanity.yml
+++ /dev/null
@@ -1,82 +0,0 @@
-name: zfs-tests-sanity
-
-on:
-  push:
-  pull_request:
-
-jobs:
-  tests:
-    runs-on: ubuntu-22.04
-    steps:
-    - uses: actions/checkout@v3
-      with:
-        ref: ${{ github.event.pull_request.head.sha }}
-    - name: Install dependencies
-      run: |
-        # https://github.com/orgs/community/discussions/47863
-        sudo apt-mark hold grub-efi-amd64-signed
-        sudo apt-get update --fix-missing
-        sudo apt-get upgrade
-        sudo xargs --arg-file=${{ github.workspace }}/.github/workflows/build-dependencies.txt apt-get install -qq
-        sudo apt-get clean
-    - name: Autogen.sh
-      run: |
-        ./autogen.sh
-    - name: Configure
-      run: |
-        ./configure --enable-debug --enable-debuginfo --enable-asan --enable-ubsan
-    - name: Make
-      run: |
-        make --no-print-directory --silent native-deb-utils native-deb-kmod
-        mv ../*.deb .
-        rm ./openzfs-zfs-dkms*.deb ./openzfs-zfs-dracut*.deb
-    - name: Install
-      run: |
-        # Update order of directories to search for modules, otherwise
-        #   Ubuntu will load kernel-shipped ones.
-        sudo sed -i.bak 's/updates/extra updates/' /etc/depmod.d/ubuntu.conf
-        sudo dpkg -i *.deb
-        # Native Debian packages enable and start the services
-        # Stop zfs-zed daemon, as it may interfere with some ZTS test cases
-        sudo systemctl stop zfs-zed
-        # Workaround for cloud-init bug
-        # see https://github.com/openzfs/zfs/issues/12644
-        FILE=/lib/udev/rules.d/10-cloud-init-hook-hotplug.rules
-        if [ -r "${FILE}" ]; then
-          HASH=$(md5sum "${FILE}" | awk '{ print $1 }')
-          if [ "${HASH}" = "121ff0ef1936cd2ef65aec0458a35772" ]; then
-            # Just shove a zd* exclusion right above the hotplug hook...
-            sudo sed -i -e s/'LABEL="cloudinit_hook"'/'KERNEL=="zd*", GOTO="cloudinit_end"\n&'/ "${FILE}"
-            sudo udevadm control --reload-rules
-          fi
-        fi
-    - name: Clear the kernel ring buffer
-      run: |
-        sudo dmesg -c >/var/tmp/dmesg-prerun
-    - name: Reclaim and report disk space
-      run: |
-        ${{ github.workspace }}/.github/workflows/scripts/reclaim_disk_space.sh
-        df -h /
-    - name: Tests
-      run: |
-        set -o pipefail
-        /usr/share/zfs/zfs-tests.sh -vKR -s 3G -r sanity | scripts/zfs-tests-color.sh
-      shell: bash
-      timeout-minutes: 330
-    - name: Prepare artifacts
-      if: failure()
-      run: |
-        RESULTS_PATH=$(readlink -f /var/tmp/test_results/current)
-        sudo dmesg > $RESULTS_PATH/dmesg
-        sudo cp /var/log/syslog /var/tmp/dmesg-prerun $RESULTS_PATH/
-        sudo chmod +r $RESULTS_PATH/*
-        # Replace ':' in dir names, actions/upload-artifact doesn't support it
-        for f in $(find /var/tmp/test_results -name '*:*'); do mv "$f" "${f//:/__}"; done
-    - uses: actions/upload-artifact@v3
-      if: failure()
-      with:
-        name: Test logs Ubuntu-${{ matrix.os }}
-        path: |
-          /var/tmp/test_results/*
-          !/var/tmp/test_results/current
-        if-no-files-found: ignore
diff --git a/.github/workflows/zloop.yml b/.github/workflows/zloop.yml
deleted file mode 100644
index 8333a88de..000000000
--- a/.github/workflows/zloop.yml
+++ /dev/null
@@ -1,65 +0,0 @@
-name: zloop
-
-on:
-  push:
-  pull_request:
-
-jobs:
-  tests:
-    runs-on: ubuntu-22.04
-    env:
-      TEST_DIR: /var/tmp/zloop
-    steps:
-    - uses: actions/checkout@v3
-      with:
-        ref: ${{ github.event.pull_request.head.sha }}
-    - name: Install dependencies
-      run: |
-        # https://github.com/orgs/community/discussions/47863
-        sudo apt-mark hold grub-efi-amd64-signed
-        sudo apt-get update --fix-missing
-        sudo apt-get upgrade
-        sudo xargs --arg-file=${{ github.workspace }}/.github/workflows/build-dependencies.txt apt-get install -qq
-        sudo apt-get clean
-    - name: Autogen.sh
-      run: |
-        ./autogen.sh
-    - name: Configure
-      run: |
-        ./configure --enable-debug --enable-debuginfo --enable-asan --enable-ubsan
-    - name: Make
-      run: |
-        make -j$(nproc) --no-print-directory --silent pkg-utils pkg-kmod
-    - name: Install
-      run: |
-        sudo dpkg -i *.deb
-        # Update order of directories to search for modules, otherwise
-        #   Ubuntu will load kernel-shipped ones.
-        sudo sed -i.bak 's/updates/extra updates/' /etc/depmod.d/ubuntu.conf
-        sudo depmod
-        sudo modprobe zfs
-    - name: Tests
-      run: |
-        sudo mkdir -p $TEST_DIR
-        # run for 10 minutes or at most 2 iterations for a maximum runner
-        #   time of 20 minutes.
-        sudo /usr/share/zfs/zloop.sh -t 600 -I 2 -l -m1 -- -T 120 -P 60
-    - name: Prepare artifacts
-      if: failure()
-      run: |
-        sudo chmod +r -R $TEST_DIR/
-    - uses: actions/upload-artifact@v3
-      if: failure()
-      with:
-        name: Logs
-        path: |
-          /var/tmp/zloop/*/
-          !/var/tmp/zloop/*/vdev/
-        if-no-files-found: ignore
-    - uses: actions/upload-artifact@v3
-      if: failure()
-      with:
-        name: Pool files
-        path: |
-          /var/tmp/zloop/*/vdev/
-        if-no-files-found: ignore