aboutsummaryrefslogtreecommitdiffstats
path: root/scripts
Commit message (Collapse)AuthorAgeFilesLines
* zpool_influxdb: move to libexec dirPavel Snajdr2020-11-283-1/+4
| | | | | | | | | | | | Move the zpool_influxdb command to /usr/libexec/zfs, and include the /usr/libexec/zfs path in the system search directory when running the test suite. Reviewed-by: Richard Elling <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Pavel Snajdr <[email protected]> Closes #11156 Closes #11160 Closes #11224
* Distributed Spare (dRAID) FeatureBrian Behlendorf2020-11-133-10/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds a new top-level vdev type called dRAID, which stands for Distributed parity RAID. This pool configuration allows all dRAID vdevs to participate when rebuilding to a distributed hot spare device. This can substantially reduce the total time required to restore full parity to pool with a failed device. A dRAID pool can be created using the new top-level `draid` type. Like `raidz`, the desired redundancy is specified after the type: `draid[1,2,3]`. No additional information is required to create the pool and reasonable default values will be chosen based on the number of child vdevs in the dRAID vdev. zpool create <pool> draid[1,2,3] <vdevs...> Unlike raidz, additional optional dRAID configuration values can be provided as part of the draid type as colon separated values. This allows administrators to fully specify a layout for either performance or capacity reasons. The supported options include: zpool create <pool> \ draid[<parity>][:<data>d][:<children>c][:<spares>s] \ <vdevs...> - draid[parity] - Parity level (default 1) - draid[:<data>d] - Data devices per group (default 8) - draid[:<children>c] - Expected number of child vdevs - draid[:<spares>s] - Distributed hot spares (default 0) Abbreviated example `zpool status` output for a 68 disk dRAID pool with two distributed spares using special allocation classes. ``` pool: tank state: ONLINE config: NAME STATE READ WRITE CKSUM slag7 ONLINE 0 0 0 draid2:8d:68c:2s-0 ONLINE 0 0 0 L0 ONLINE 0 0 0 L1 ONLINE 0 0 0 ... U25 ONLINE 0 0 0 U26 ONLINE 0 0 0 spare-53 ONLINE 0 0 0 U27 ONLINE 0 0 0 draid2-0-0 ONLINE 0 0 0 U28 ONLINE 0 0 0 U29 ONLINE 0 0 0 ... U42 ONLINE 0 0 0 U43 ONLINE 0 0 0 special mirror-1 ONLINE 0 0 0 L5 ONLINE 0 0 0 U5 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 L6 ONLINE 0 0 0 U6 ONLINE 0 0 0 spares draid2-0-0 INUSE currently in use draid2-0-1 AVAIL ``` When adding test coverage for the new dRAID vdev type the following options were added to the ztest command. These options are leverages by zloop.sh to test a wide range of dRAID configurations. -K draid|raidz|random - kind of RAID to test -D <value> - dRAID data drives per group -S <value> - dRAID distributed hot spares -R <value> - RAID parity (raidz or dRAID) The zpool_create, zpool_import, redundancy, replacement and fault test groups have all been updated provide test coverage for the dRAID feature. Co-authored-by: Isaac Huang <[email protected]> Co-authored-by: Mark Maybee <[email protected]> Co-authored-by: Don Brady <[email protected]> Co-authored-by: Matthew Ahrens <[email protected]> Co-authored-by: Brian Behlendorf <[email protected]> Reviewed-by: Mark Maybee <[email protected]> Reviewed-by: Matt Ahrens <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #10102
* Fix commitcheck on FreeBSDRyan Moeller2020-10-201-44/+10
| | | | | | | | | Convert from bash to sh, avoid Perl regexes and \s, prune unused functions. Reviewed-by: Mateusz Piotrowski <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #11070
* Add zfs.sh module unload error messageAdam D. Moss2020-10-131-0/+3
| | | | | | | | If modules fail to unload because of outstanding users, don't consider this a success. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Adam Moss <[email protected]> Closes #11042
* Replace ZFS on Linux references with OpenZFSBrian Behlendorf2020-10-081-4/+4
| | | | | | | | | | | | | This change updates the documentation to refer to the project as OpenZFS instead ZFS on Linux. Web links have been updated to refer to https://github.com/openzfs/zfs. The extraneous zfsonlinux.org web links in the ZED and SPL sources have been dropped. Reviewed-by: George Melikov <[email protected]> Reviewed-by: Richard Laager <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #11007
* Update issue templates, commitcheck and Contributing.mdKjeld Schouten-Lebbing2020-09-231-59/+0
| | | | | | | | | | | | - Removes OpenZFS ports from commit check - Removes OpenZFS ports from CONTRIBUTING.md - Adds mailings lists and IRC to issue template selector - Remove blank issue option from issue creator Reviewed-by: George Melikov <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Co-authored-by: George Melikov <[email protected]> Signed-off-by: Kjeld Schouten-Lebbing <[email protected]> Closes #10965
* Add a missing option prefix `-` in zfs-tests.sh usage()Peter Dave Hello2020-09-081-1/+1
| | | | | | Reviewed-by: Giuseppe Di Natale <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Peter Dave Hello <[email protected]> Closes #10893
* ZTS: Remove leftover variable namesRyan Moeller2020-08-221-3/+0
| | | | | | | | | These were overlooked when use of `local` was removed to satisfy checkbashisms. Reviewed-by: Richard Laager <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10762
* Silence 'make checkbashisms'Brian Behlendorf2020-08-202-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Commit d2bce6d03 added the 'make checkbashisms' target but did not resolve all of the bashisms in the scripts. This commit doesn't resolve them all either but it does fix up a few, and it excludes the others so 'make checkstyle' no longer prints warnings. It's a small step in the right direction. * Dracut is Linux specific and itself depends on bash. Therefore all dracut support scripts can be bash specific, update their shebang accordingly. * zed-functions.sh, zfs-import, zfs-mount, zfs-zed, smart paxcheck.sh, make_gitrev.sh - these scripts were excuded from the check until they can be updated and properly tested. * zfsunlock - only whole values for sleep are allowed. * vdev_id - removed unneeded locals; use && instead of -a. * dkms.mkconf, dkms.postbuil - use || instead of -o. Reviewed-by: InsanePrawn <[email protected]> Reviewed-by: Gabriel A. Devenyi <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: George Melikov <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #10755
* Add zstd support to zfsMichael Niewöhner2020-08-203-5/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This PR adds two new compression types, based on ZStandard: - zstd: A basic ZStandard compression algorithm Available compression. Levels for zstd are zstd-1 through zstd-19, where the compression increases with every level, but speed decreases. - zstd-fast: A faster version of the ZStandard compression algorithm zstd-fast is basically a "negative" level of zstd. The compression decreases with every level, but speed increases. Available compression levels for zstd-fast: - zstd-fast-1 through zstd-fast-10 - zstd-fast-20 through zstd-fast-100 (in increments of 10) - zstd-fast-500 and zstd-fast-1000 For more information check the man page. Implementation details: Rather than treat each level of zstd as a different algorithm (as was done historically with gzip), the block pointer `enum zio_compress` value is simply zstd for all levels, including zstd-fast, since they all use the same decompression function. The compress= property (a 64bit unsigned integer) uses the lower 7 bits to store the compression algorithm (matching the number of bits used in a block pointer, as the 8th bit was borrowed for embedded block pointers). The upper bits are used to store the compression level. It is necessary to be able to determine what compression level was used when later reading a block back, so the concept used in LZ4, where the first 32bits of the on-disk value are the size of the compressed data (since the allocation is rounded up to the nearest ashift), was extended, and we store the version of ZSTD and the level as well as the compressed size. This value is returned when decompressing a block, so that if the block needs to be recompressed (L2ARC, nop-write, etc), that the same parameters will be used to result in the matching checksum. All of the internal ZFS code ( `arc_buf_hdr_t`, `objset_t`, `zio_prop_t`, etc.) uses the separated _compress and _complevel variables. Only the properties ZAP contains the combined/bit-shifted value. The combined value is split when the compression_changed_cb() callback is called, and sets both objset members (os_compress and os_complevel). The userspace tools all use the combined/bit-shifted value. Additional notes: zdb can now also decode the ZSTD compression header (flag -Z) and inspect the size, version and compression level saved in that header. For each record, if it is ZSTD compressed, the parameters of the decoded compression header get printed. ZSTD is included with all current tests and new tests are added as-needed. Per-dataset feature flags now get activated when the property is set. If a compression algorithm requires a feature flag, zfs activates the feature when the property is set, rather than waiting for the first block to be born. This is currently only used by zstd but can be extended as needed. Portions-Sponsored-By: The FreeBSD Foundation Co-authored-by: Allan Jude <[email protected]> Co-authored-by: Brian Behlendorf <[email protected]> Co-authored-by: Sebastian Gottschall <[email protected]> Co-authored-by: Kjeld Schouten-Lebbing <[email protected]> Co-authored-by: Michael Niewöhner <[email protected]> Signed-off-by: Allan Jude <[email protected]> Signed-off-by: Allan Jude <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Signed-off-by: Sebastian Gottschall <[email protected]> Signed-off-by: Kjeld Schouten-Lebbing <[email protected]> Signed-off-by: Michael Niewöhner <[email protected]> Closes #6247 Closes #9024 Closes #10277 Closes #10278
* cstyle.pl: echo commands for github workflowGeorge Melikov2020-08-131-4/+9
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: George Melikov <[email protected]> Closes #10705
* ZTS: Remove bashisms from zfs-tests.shRyan Moeller2020-08-071-65/+52
| | | | | | | | Bring zfs-tests.sh in to compliance with the other scripts by converting it /bin/sh for to avoid a dependency on bash. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10640
* Add zfs_gitrev.h to the distributed sourcesArvind Sankar2020-07-221-1/+29
| | | | | | | | | | | | | | | | | | | | | | | | | Commit 109d2c931020 ("Move zfs_gitrev.h to build directory") stopped distributing zfs_gitrev.h, as it is a generated file. Add it back, with some changes in behavior. Change the logic for gitrev as follows - if the source tree is a git repository, the behavior for build is unchanged. For make dist, append -dist to the git tag in the distributed version of zfs_gitrev.h. - otherwise, check if the source tree contains zfs_gitrev.h, and use it if so, falling back to "unknown" if it doesn't exist. - clean it only in make maintainer-clean, so we don't remove it from the source tree on make clean or make distclean. This allows disted sources to track what git tag they originally came from, with the -dist suffix indicating that the code wasn't built directly from git and so might contain additional changes beyond the git tag. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Eli Schwartz <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10595
* Restore scripts/make_gitrev.shArvind Sankar2020-07-222-0/+51
| | | | | | | | | | | | | | | | Commit 109d2c931020 ("Move zfs_gitrev.h to build directory") removed scripts/make_gitrev.sh, putting the logic into the Makefile itself. However, at least the Arch Linux packager wants the script so that the file can be generated without having to run configure first, for DKMS packaging purposes. So move the make recipe back into the script. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Eli Schwartz <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10595
* Centralize variable substitutionArvind Sankar2020-07-141-4/+0
| | | | | | | | | | | | A bunch of places need to edit files to incorporate the configured paths i.e. bindir, sbindir etc. Move this logic into a common file. Create arc_summary by copying arc_summary[23] as appropriate at build time instead of install time. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10559
* Drop unnecessary srcdir pathsArvind Sankar2020-06-241-5/+5
| | | | | | | | | | There's no need to specify the srcdir explicitly in _HEADERS and EXTRA_DIST. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10493
* Move zfs_gitrev.h to build directoryArvind Sankar2020-06-242-46/+0
| | | | | | | | | | | | | | | | | | | | | | Currently an out-of-tree build does not work with read-only source directory because zfs_gitrev.h can't be created. Move this file to the build directory, which is more appropriate for a generated file, and drop the dist-hook for zfs_gitrev.h. There is no need to distribute this file since it will be regenerated as part of the compilation in any case. scripts/make_gitrev.sh tries to avoid updating zfs_gitrev.h if there has been no change, however this doesn't cover the case when the source directory is not in git: in that case zfs_gitrev.h gets overwritten even though it's always "unknown". Simplify the logic to always write out a new version of zfs_gitrev.h, compare against the old and overwrite only if different. This is now simple enough to just include in the Makefile, so drop the script. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10493
* Add FreeBSD support to OpenZFSMatthew Macy2020-04-141-1/+1
| | | | | | | | | | | | | | | | | | Add the FreeBSD platform code to the OpenZFS repository. As of this commit the source can be compiled and tested on FreeBSD 11 and 12. Subsequent commits are now required to compile on FreeBSD and Linux. Additionally, they must pass the ZFS Test Suite on FreeBSD which is being run by the CI. As of this commit 1230 tests pass on FreeBSD and there are no unexpected failures. Reviewed-by: Sean Eric Fagan <[email protected]> Reviewed-by: Jorgen Lundman <[email protected]> Reviewed-by: Richard Laager <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Co-authored-by: Ryan Moeller <[email protected]> Signed-off-by: Matt Macy <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #898 Closes #8987
* zloop.sh should call ZDB with pool nameOlaf Faaland2020-03-111-1/+1
| | | | | | | | | | | Commit 54007c79 introduced an error, changing the final argument to $ZDB from ztest to $ZTEST. This argument indicates the pool name, not the script, and so should not have been changed. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Olaf Faaland <[email protected]> Closes #10118
* bash scripts: use /usr/bin/env for bash shebangsGraham Christensen2020-02-103-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Not all systems / distros have a `/bin/bash`, and these scripts are more difficult to run at development time. For example, my system is NixOS which doesn't have a /bin/bash. This is not a problem for NixOS building ZFS as a package: the build environment automatically replaces these shebangs with corrected paths. The problem is much more annoying at development time: either the scripts don't run, or I correct them for my local machine and deal with a perpetually dirty work tree. Before committing this patch I confirmed there are existing scripts which use `/usr/bin/env` to locate bash, so I am thinking this is a safe transformation. There are a handful of other shebangs in this repository which don't work on my system. This patch is useful on its own specifically for `commitcheck.sh`, otherwise I can't validate my commits before submission. Here are the remaining shebangs which NixOS systems won't have: 1274 #!/bin/ksh -p 91 #!/bin/ksh 89 #! /bin/ksh -p 2 #!/bin/sed -f 1 #!/usr/bin/perl -w 1 #!/usr/bin/ksh 1 #!/bin/nawk -f plus this which will create an invalid shebang in `tests/zfs-tests/tests/functional/mv_files/mv_files_common.kshlib`: echo "#!/bin/ksh" > $TEST_BASE_DIR/exitsZero.ksh I chose to leave those alone for now, and gauge the interest in this much smaller patch first. The fixes for these are easy enough by simply using `/usr/bin/env ksh`: 91 #!/bin/ksh 1 #!/usr/bin/ksh The fix for the other set is much trickier. Quoting the GNU coreutils manual: Most operating systems (e.g. GNU/Linux, BSDs) treat all text after the first space as a single argument. When using env in a script it is thus not possible to specify multiple arguments. and not all `env`'s support arguments. Mine (GNU Coreutils 8.31) does, though this feature is new since April 2018, GNU Coreutils 8.30: https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=668306ed86c8c79b0af0db8b9c882654ebb66db2 and worse, requires the -S argument: -S, --split-string=S process and split S into separate arguments; used to pass multiple arguments on shebang lines Example: $ seq 1 2 | $(nix-build '<nixpkgs>' -A coreutils)/bin/env "sort -nr" /nix/[...]-coreutils-8.31/bin/env: ‘sort -nr’: No such file or directory /nix/[...]-coreutils-8.31/bin/env: use -[v]S to pass options in shebang lines $ seq 1 2 | $(nix-build '<nixpkgs>' -A coreutils)/bin/env "-S sort -nr" 2 1 GNU Coreutils says FreeBSD's `env` does, though I wonder if FreeBSD's would be unhappy with the `-S`: https://www.gnu.org/software/coreutils/manual/html_node/env-invocation.html#env-invocation BusyBox v1.30.1 does not, and does not have a `-S`-like option: $ seq 1 2 | $(nix-build '<nixpkgs>' -A busybox)/bin/env "sort -nr" env: can't execute 'sort -nr': No such file or directory Toybox 0.8.1 also does not, and also does not have a `-S` option: $ seq 1 2 | $(nix-build '<nixpkgs>' -A toybox)/bin/env "sort -nr" env: exec sort -nr: No such file or directory --- At any rate, if this patch merges and the remaining ~1,500 are updated, the much larger patch should probably include a checkstyle-like test asserting all new shebangs use `/usr/bin/env`. I also don't mind dealing with NixOS weirdness if the project would prefer that. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Graham Christensen <[email protected]> Closes #9893
* ZTS: Reverse constrained path lookup orderRyan Moeller2020-01-291-2/+5
| | | | | | | | | | | | FreeBSD base system zfs utils are in /sbin. ZoF utils install to /usr/local/sbin. Ensure we link to the ZoF utils not the base utils when searching for utils to constrain paths to for the tests. Reviewed-by: John Kennedy <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #9906
* scripts/zfs-test.sh: example for -tChristian Schwarz2020-01-231-0/+3
| | | | | | | | Add an example for running a single test case. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Christian Schwarz <[email protected]> Closes #9878
* Add FreeBSD core handling in zloop.shRyan Moeller2020-01-022-15/+31
| | | | | | | And use the correct path to libtool and ztest. Reviewed-By: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #9790
* Update zfs.sh work on FreeBSDRyan Moeller2019-12-192-16/+54
| | | | | | | | Extend the zfs.sh script to load and unload zfs kmods on FreeBSD. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Kjeld Schouten <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #9746
* Update ZTS to work on FreeBSDMatthew Macy2019-12-181-45/+121
| | | | | | | | | | | | Update the common ZTS scripts and individual test cases as needed in order to allow them to be run on FreeBSD. The high level goal is to provide compatibility wrappers whenever possible to minimize changes to individual test cases. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: John Kennedy <[email protected]> Signed-off-by: Matt Macy <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #9692
* Move platform independent tests to a shared runfileRyan Moeller2019-10-091-20/+28
| | | | | | | | | | | | | | | | | | | Tests that aren't limited to running on Linux can be moved to a common runfile to be shared with other platforms. The test runner and wrapper script are enhanced to allow specifying multiple runfiles as a comma-separated list. The default runfiles are now "common.run,PLATFORM.run" where PLATFORM is determined at run time. Sections in runfiles that share a path with another runfile can append a colon separator and an identifier to the path in the section name, ie `[tests/functional/atime:Linux]`, to avoid overriding the tests specified by other runfiles. Reviewed-by: Jorgen Lundman <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: John Kennedy <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #9391
* kmodtool: depmod pathBrian Behlendorf2019-09-111-7/+19
| | | | | | | | | | | | | | | Determine the location of depmod on the system, either /sbin/depmod or /usr/sbin/depmod. Then use that path when generating the specfile. Additionally, update the Requires lines to reference the package which provides depmod rather than the binary itself. For CentOS/RHEL 7+8 and all supported Fedora releases this is the kmod package, and for CentOS/RHEL 6 it is the module-init-tools package. Reviewed-by: Minh Diep <[email protected]> Signed-off-by: Olaf Faaland <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #8724 Closes #9310
* Fix typosAndrea Gelmini2019-09-021-3/+3
| | | | | | | Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Richard Laager <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Andrea Gelmini <[email protected]> Closes #9251
* Split argument list, satisfy shellcheck SC2086Ryan Moeller2019-08-251-6/+13
| | | | | | | | | | | | | | | | | | | Split the arguments for ${TEST_RUNNER} across multiple lines for clarity. Also added quotes in the message to match the invoked command. Unquoted variables in argument lists are subject to splitting. In this particular case we can't quote the variable because it is an optional argument. Use the method suggested in the description linked below, instead. The technique is to use an unquoted variable with an alternate value. https://github.com/koalaman/shellcheck/wiki/SC2086 Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Giuseppe Di Natale <[email protected]> Reviewed-by: John Kennedy <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #9212
* Fix out-of-tree build failuresBrian Behlendorf2019-06-241-2/+3
| | | | | | | | | | | | | | | | | | | | Resolve the incorrect use of srcdir and builddir references for various files in the build system. These have crept in over time and went unnoticed because when building in the top level directory srcdir and builddir are identical. With this change it's again possible to build in a subdirectory. $ mkdir obj $ cd obj $ ../configure $ make Reviewed-by: loli10K <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Don Brady <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #8921 Closes #8943
* Fix %post and %postun generation in kmodtoolSamuel VERSCHELDE2019-06-101-2/+2
| | | | | | | | | | | | During zfs-kmod RPM build, $(uname -r) gets unintentionally evaluated on the build host, once and for all. It should be evaluated during the execution of the scriptlets on the installation host. Escaping the $ character avoids evaluating it during build. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Olaf Faaland <[email protected]> Reviewed-by: Neal Gompa <[email protected]> Signed-off-by: Samuel Verschelde <[email protected]> Closes #8866
* Avoid updating zfs_gitrev.h when rev is unchangedJorgen Lundman2019-06-061-0/+4
| | | | | | | | | | Build process would always re-compile spa_history.c due to touching zfs_gitrev.h - avoid if no change in gitrev. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Chris Dunlop <[email protected]> Reviewed-by: Allan Jude <[email protected]> Signed-off-by: Jorgen Lundman <[email protected]> Closes #8860
* Add zol2zfs-patch.sed to Makefile.am and sortRichard Laager2019-05-081-2/+3
| | | | | | | | | | | In adding man-dates.sh, I noticed that zol2zfs-patch.sed was missing, even though zfs2zol-patch.sed was present. Also, the list was not sorted, so I sorted it. Reviewed-by: Olaf Faaland <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Richard Laager <[email protected]> Closes #8710
* Correct man page datesRichard Laager2019-05-082-0/+13
| | | | | | | | | | | Various changes (many by me) have been made to the man pages without bumping their dates. I have now corrected them based on the last commit to each file. I also added the script I used to make these changes. Reviewed-by: Olaf Faaland <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Richard Laager <[email protected]> Closes #8710
* OpenZFS 9185 - Enable testing over NFS in ZFS performance testsAhmed Ghanem2019-02-041-1/+8
| | | | | | | | | | | | | | | | | | | | This change makes additions to the ZFS test suite that allows the performance tests to run over NFS. The test is run and performance data collected from the server side, while IO is generated on the NFS client. This has been tested with Linux and illumos NFS clients. Authored by: Ahmed Ghanem <[email protected]> Reviewed by: Dan Kimmel <[email protected]> Reviewed by: John Kennedy <[email protected]> Reviewed by: Kevin Greene <[email protected]> Reviewed-by: Richard Elling <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Ported-by: John Kennedy <[email protected]> Signed-off-by: John Kennedy <[email protected]> OpenZFS-issue: https://www.illumos.org/issues/9185 Closes #8367
* shellcheck passbunder20152019-02-041-2/+2
| | | | | | | | | | note: which is non-standard. Use builtin 'command -v' instead. [SC2230] note: Use -n instead of ! -z. [SC2236] Reviewed-by: George Melikov <[email protected]> Reviewed-by: Giuseppe Di Natale <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: bunder2015 <[email protected]> Closes #8367
* dkms: Enable debuginfo option to be set with zfs sysconfig fileNeal Gompa (ニール・ゴンパ)2019-01-181-0/+4
| | | | | | | | | | | | | | | | On some Linux distributions, the kernel module build will not default to building with debuginfo symbols, which can make it difficult for debugging and testing. For this case, we provide a flag to override the build to force debuginfo to be produced for the kernel module build. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Co-authored-by: Neal Gompa <[email protected]> Co-authored-by: Simon Watson <[email protected]> Signed-off-by: Neal Gompa <[email protected]> Signed-off-by: Simon Watson <[email protected]> Closes #8304
* Bump commit subject length to 72 charactersNeal Gompa (ニール・ゴンパ)2019-01-081-3/+3
| | | | | | | | | | | | | There's not really a reason to keep the subject length so short, since the reason to make it this short was for making nice renders of a summary list of the git log. With 72 characters, this still works out fine, so let's just raise it to that so that it's easier to give slightly more descriptive change summaries. Reviewed by: Matt Ahrens <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Neal Gompa <[email protected]> Closes #8250
* arc_summary: consolidate test caseBrian Behlendorf2019-01-061-0/+4
| | | | | | | | | | | | | | Since we're only installing one version of arc_summary we only need one test case. Update the test to determine which version is available and then test its supported flags. Remove files for misc tests which should have been cleaned up. Reviewed-by: John Ramsden <[email protected]> Reviewed-by: Neal Gompa <[email protected]> Reviewed-by: loli10K <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #8096
* Do not enable stack tracer for ZFS performance testTony Nguyen2018-12-072-6/+21
| | | | | | | | | | | | | | Linux ZFS test suite runs with /proc/sys/kernel/stack_tracer_enabled=1, via zfs.sh script, which has negative performance impact, up to 40%. Since large stack is a rare issue now, preferred behavior would be: - making stack tracer an opt-in feature for zfs.sh - zfs-test.sh enables stack tracer only when requested Reviewed-by: George Melikov <[email protected]> Reviewed-by: Richard Elling <[email protected]> Reviewed-by: John Kennedy <[email protected]> Signed-off-by: Tony Nguyen <[email protected]> #8173
* Make gitrev more reliableMatthew Ahrens2018-10-221-3/+1
| | | | | | | | | | | | | | | | In some build methods, the gitrev is unnecessarily set to "unknown". We can improve this by changing the gitrev to use `git describe --always --long --dirty`. This gets the revision even when no tag matches (--always). It prints the hash even when it exactly matches a tag (--long). And if there are uncommitted changes, it appends "-dirty", rather than failing (--dirty). Reviewed-by: George Melikov <[email protected]> Reviewed by: Pavel Zakharov <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Matthew Thode <[email protected]> Signed-off-by: Matthew Ahrens <[email protected]> Closes #8034
* Allow copy-builtin to work with modified sourcesMatthew Thode2018-10-171-24/+13
| | | | | | | | | | | | | | `scripts/make_gitrev.sh` had 'set -e' so if any command failed it would fail and cause copy-builtin to fail (copy-builtin also has `set -e`. This commit also simplifies scripts/make_gitrev.sh to always write a file by using a cleanup function. It also simplifies other areas of the script as well (making it much shorter). Reviewed-by: John Kennedy <[email protected]> Reviewed-by: George Melikov <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Matthew Thode <[email protected]> Closes #8022 Closes #8025
* OpenZFS 9689 - zfs range lock code should not be zpl-specificMatt Ahrens2018-10-111-5/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The ZFS range locking code in zfs_rlock.c/h depends on ZPL-specific data structures, specifically znode_t. However, it's also used by the ZVOL code, which uses a "dummy" znode_t to pass to the range locking code. We should clean this up so that the range locking code is generic and can be used equally by ZPL and ZVOL, and also can be used by future consumers that may need to run in userland (libzpool) as well as the kernel. Porting notes: * Added missing sys/avl.h include to sys/zfs_rlock.h. * Removed 'dbuf is within the locked range' ASSERTs from dmu_sync(). This was needed because ztest does not yet use a locked_range_t. * Removed "Approved by:" tag requirement from OpenZFS commit check to prevent needless warnings when integrating changes which has not been merged to illumos. * Reverted free_list range lock changes which were originally needed to defer the cv_destroy() which was called immediately after cv_broadcast(). With d2733258 this should be safe but if not we may need to reintroduce this logic. * Reverts: The following two commits were reverted and squashed in to this change in order to make it easier to apply OpenZFS 9689. - d88895a0, which removed the dummy znode from zvol_state - e3a07cd0, which updated ztest to use range locks * Preserved optimized rangelock comparison function. Preserved the rangelock free list. The cv_destroy() function will block waiting for all processes in cv_wait() to be scheduled and drop their reference. This is done to ensure it's safe to free the condition variable. However, blocking while holding the rl->rl_lock mutex can result in a deadlock on Linux. A free list is introduced to defer the cv_destroy() and kmem_free() until after the mutex is released. Authored by: Matthew Ahrens <[email protected]> Reviewed by: Brian Behlendorf <[email protected]> Reviewed by: Serapheim Dimitropoulos <[email protected]> Reviewed by: George Wilson <[email protected]> Reviewed by: Brad Lewis <[email protected]> Ported-by: Brian Behlendorf <[email protected]> OpenZFS-issue: https://illumos.org/issues/9689 OpenZFS-commit: https://github.com/openzfs/openzfs/pull/680 External-issue: DLPX-58662 Closes #7980
* Create /proc/sys/kernel/spl/gitrev with git hashMatthew Ahrens2018-10-082-0/+55
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The existing mechanisms for determining what code is running in the kernel do not always correctly report the git hash. The versions reported there do not reflect changes made since `configure` was run (i.e. incremental builds do not update the version) and they are misleading if git tags are not set up properly. This applies to `modinfo zfs`, `dmesg`, and `/sys/module/zfs/version`. There are complicated requirements on how the existing version is generated. Therefore we are leaving that alone, and adding a new mechanism to record and retrieve the git hash: `cat /proc/sys/kernel/spl/gitrev` The gitrev is re-generated at compile time, when running `make` (including for incremental builds). The value is the output of `git describe` (or "unknown" if not in a git repo or there are uncommitted changes). We're also removing /proc/sys/kernel/spl/version, which was never very useful. Reviewed by: Pavel Zakharov <[email protected]> Reviewed by: Brian Behlendorf <[email protected]> Reviewed by: Tim Chase <[email protected]> Signed-off-by: Matthew Ahrens <[email protected]> Closes #7931 Closes #7965
* Direct IO supportBrian Behlendorf2018-08-271-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Direct IO via the O_DIRECT flag was originally introduced in XFS by IRIX for database workloads. Its purpose was to allow the database to bypass the page and buffer caches to prevent unnecessary IO operations (e.g. readahead) while preventing contention for system memory between the database and kernel caches. On Illumos, there is a library function called directio(3C) that allows user space to provide a hint to the file system that Direct IO is useful, but the file system is free to ignore it. The semantics are also entirely a file system decision. Those that do not implement it return ENOTTY. Since the semantics were never defined in any standard, O_DIRECT is implemented such that it conforms to the behavior described in the Linux open(2) man page as follows. 1. Minimize cache effects of the I/O. By design the ARC is already scan-resistant which helps mitigate the need for special O_DIRECT handling. Data which is only accessed once will be the first to be evicted from the cache. This behavior is in consistent with Illumos and FreeBSD. Future performance work may wish to investigate the benefits of immediately evicting data from the cache which has been read or written with the O_DIRECT flag. Functionally this behavior is very similar to applying the 'primarycache=metadata' property per open file. 2. O_DIRECT _MAY_ impose restrictions on IO alignment and length. No additional alignment or length restrictions are imposed. 3. O_DIRECT _MAY_ perform unbuffered IO operations directly between user memory and block device. No unbuffered IO operations are currently supported. In order to support features such as transparent compression, encryption, and checksumming a copy must be made to transform the data. 4. O_DIRECT _MAY_ imply O_DSYNC (XFS). O_DIRECT does not imply O_DSYNC for ZFS. Callers must provide O_DSYNC to request synchronous semantics. 5. O_DIRECT _MAY_ disable file locking that serializes IO operations. Applications should avoid mixing O_DIRECT and normal IO or mmap(2) IO to the same file. This is particularly true for overlapping regions. All I/O in ZFS is locked for correctness and this locking is not disabled by O_DIRECT. However, concurrently mixing O_DIRECT, mmap(2), and normal I/O on the same file is not recommended. This change is implemented by layering the aops->direct_IO operations on the existing AIO operations. Code already existed in ZFS on Linux for bypassing the page cache when O_DIRECT is specified. References: * http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide/tmp/en-US/html/ch02s09.html * https://blogs.oracle.com/roch/entry/zfs_and_directio * https://ext4.wiki.kernel.org/index.php/Clarifying_Direct_IO's_Semantics * https://illumos.org/man/3c/directio Reviewed-by: Richard Elling <[email protected]> Signed-off-by: Richard Yao <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #224 Closes #7823
* OpenZFS 9330 - stack overflow when creating a deeply nested datasetSerapheim Dimitropoulos2018-07-091-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Datasets that are deeply nested (~100 levels) are impractical. We just put a limit of 50 levels to newly created datasets. Existing datasets should work without a problem. The problem can be seen by attempting to create a dataset using the -p option with many levels: panic[cpu0]/thread=ffffff01cd282c20: BAD TRAP: type=8 (#df Double fault) rp=ffffffff fffffffffbc3aa60 unix:die+100 () fffffffffbc3ab70 unix:trap+157d () ffffff00083d7020 unix:_patch_xrstorq_rbx+196 () ffffff00083d7050 zfs:dbuf_rele+2e () ... ffffff00083d7080 zfs:dsl_dir_close+32 () ffffff00083d70b0 zfs:dsl_dir_evict+30 () ffffff00083d70d0 zfs:dbuf_evict_user+4a () ffffff00083d7100 zfs:dbuf_rele_and_unlock+87 () ffffff00083d7130 zfs:dbuf_rele+2e () ... The block above repeats once per directory in the ... ... create -p command, working towards the root ... ffffff00083db9f0 zfs:dsl_dataset_drop_ref+19 () ffffff00083dba20 zfs:dsl_dataset_rele+42 () ffffff00083dba70 zfs:dmu_objset_prefetch+e4 () ffffff00083dbaa0 zfs:findfunc+23 () ffffff00083dbb80 zfs:dmu_objset_find_spa+38c () ffffff00083dbbc0 zfs:dmu_objset_find+40 () ffffff00083dbc20 zfs:zfs_ioc_snapshot_list_next+4b () ffffff00083dbcc0 zfs:zfsdev_ioctl+347 () ffffff00083dbd00 genunix:cdev_ioctl+45 () ffffff00083dbd40 specfs:spec_ioctl+5a () ffffff00083dbdc0 genunix:fop_ioctl+7b () ffffff00083dbec0 genunix:ioctl+18e () ffffff00083dbf10 unix:brand_sys_sysenter+1c9 () Porting notes: * Added zfs_max_dataset_nesting module option with documentation. * Updated zfs_rename_014_neg.ksh for Linux. * Increase the zfs.sh stack warning to 15K. Enough time has passed that 16K can be reasonably assumed to be the default value. It was increased in the 3.15 kernel released in June of 2014. Authored by: Serapheim Dimitropoulos <[email protected]> Reviewed by: John Kennedy <[email protected]> Reviewed by: Matt Ahrens <[email protected]> Ported-by: Brian Behlendorf <[email protected]> Approved by: Garrett D'Amore <[email protected]> OpenZFS-issue: https://www.illumos.org/issues/9330 OpenZFS-commit: https://github.com/openzfs/openzfs/commit/757a75a Closes #7681
* ZTS: Adopt OpenZFS test analysis scriptBrian Behlendorf2018-06-202-2/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Adopt and extend the OpenZFS ZTS results analysis script for use with ZFS on Linux. This allows for automatic analysis of tests which may be skipped for a variety or reasons or which are not entirely reliable. In addition to the list of 'known' failures, which have been updated for ZFS on Linux, there in a new 'maybe' section. This mapping include tests which might be correctly skipped depending on the test environment. This may be because of a missing dependency or lack of required kernel support. This list also includes tests which normally pass but might on occasion fail for a harmless reason. The script was also extended include a reason for why a given test might be skipped or may fail. The reason will be included after the test in the "results other than PASS that are expected" section. For failures it is preferable to set the reason to the GitHub issue number and for skipped tests several generic reasons are available. You may also specify a custom reason if needed. All tests were added back in to the linux.run file even if they are expected to failed. There is value in running tests which may not pass, the expected results for these tests has been encoded in the new analysis script. All tests which were disabled because they ran more slowly on a 32-bit system have been re-enabled. Developers working on 32-bit systems should assess what it reasonable for their environment. The unnecessary dependency on physical block devices was removed for the checksum, grow_pool, and grow_replicas test groups so they are no longer skipped. Updated the filetest_001_pos test case to run properly now that it is enabled and moved the grow tests in to a single directory. Reviewed-by: Prakash Surya <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #7638
* Update build system and packagingBrian Behlendorf2018-05-295-86/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Minimal changes required to integrate the SPL sources in to the ZFS repository build infrastructure and packaging. Build system and packaging: * Renamed SPL_* autoconf m4 macros to ZFS_*. * Removed redundant SPL_* autoconf m4 macros. * Updated the RPM spec files to remove SPL package dependency. * The zfs package obsoletes the spl package, and the zfs-kmod package obsoletes the spl-kmod package. * The zfs-kmod-devel* packages were updated to add compatibility symlinks under /usr/src/spl-x.y.z until all dependent packages can be updated. They will be removed in a future release. * Updated copy-builtin script for in-kernel builds. * Updated DKMS package to include the spl.ko. * Updated stale AUTHORS file to include all contributors. * Updated stale COPYRIGHT and included the SPL as an exception. * Renamed README.markdown to README.md * Renamed OPENSOLARIS.LICENSE to LICENSE. * Renamed DISCLAIMER to NOTICE. Required code changes: * Removed redundant HAVE_SPL macro. * Removed _BOOT from nvpairs since it doesn't apply for Linux. * Initial header cleanup (removal of empty headers, refactoring). * Remove SPL repository clone/build from zimport.sh. * Use of DEFINE_RATELIMIT_STATE and DEFINE_SPINLOCK removed due to build issues when forcing C99 compilation. * Replaced legacy ACCESS_ONCE with READ_ONCE. * Include needed headers for `current` and `EXPORT_SYMBOL`. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Olaf Faaland <[email protected]> Reviewed-by: Matthew Ahrens <[email protected]> Reviewed-by: Pavel Zakharov <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> TEST_ZIMPORT_SKIP="yes" Closes #7556
* Support Debian DKMS buildsAntonio Russo2018-05-261-1/+16
| | | | | | | | | | | | | | | scripts/dkms.mkconf calls configure with `--with-linux=${kernel_source_dir}`, but Debian puts it kernel source at `/lib/modules/<version>/source`. This patch adds the same logic to the DKMS file produced by `scripts/dkms.mkconf` that Debian has shipped in its official ZFS packaging: at DKMS build time, it checks if the system is a Debian system, and adjusts the path accordingly. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: George Melikov <[email protected]> Signed-off-by: Antonio Russo <[email protected]> Closes #7358 Closes #7540 Closes #7554
* Adopt pyzfs from ClusterHQloli10K2018-05-012-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit introduces several changes: * Update LICENSE and project information * Give a good PEP8 talk to existing Python source code * Add RPM/DEB packaging for pyzfs * Fix some outstanding issues with the existing pyzfs code caused by changes in the ABI since the last time the code was updated * Integrate pyzfs Python unittest with the ZFS Test Suite * Add missing libzfs_core functions: lzc_change_key, lzc_channel_program, lzc_channel_program_nosync, lzc_load_key, lzc_receive_one, lzc_receive_resumable, lzc_receive_with_cmdprops, lzc_receive_with_header, lzc_reopen, lzc_send_resume, lzc_sync, lzc_unload_key, lzc_remap Note: this commit slightly changes zfs_ioc_unload_key() ABI. This allow to differentiate the case where we tried to unload a key on a non-existing dataset (ENOENT) from the situation where a dataset has no key loaded: this is consistent with the "change" case where trying to zfs_ioc_change_key() from a dataset with no key results in EACCES. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: loli10K <[email protected]> Closes #7230
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-18 16:41:50 +0000