aboutsummaryrefslogtreecommitdiffstats
path: root/scripts
Commit message (Collapse)AuthorAgeFilesLines
* Add ShellCheck's `--enable=all` inside `scripts/`Damian Szuberski2022-01-1310-81/+105
| | | | | | | | Strengthen static code analysis for shell scripts. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: John Kennedy <[email protected]> Signed-off-by: szubersk <[email protected]> Closes #12914
* Add zfs-test facility to automatically rerun failing testsPaul Dagnelie2021-12-011-3/+35
| | | | | | | | | | | | This was a project proposed as part of the Quality theme for the hackthon for the 2021 OpenZFS Developer Summit. The idea is to improve the usability of the automated tests that get run when a PR is created by having failing tests automatically rerun in order to make flaky tests less impactful. Reviewed-by: John Kennedy <[email protected]> Reviewed-by: Tony Nguyen <[email protected]> Signed-off-by: Paul Dagnelie <[email protected]> Closes #12740
* Remove REMAKE_INITRDjokersus2021-11-301-1/+0
| | | | | | | | | The option has been deprecated in dkms and will break packaging in future versions. See https://github.com/dell/dkms/commit/7114c62 Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: George Amanakis <[email protected]> Signed-off-by: jokersus <[email protected]> Closes #12781
* Pass `--enable=all` to shellcheck within contrib/Damian Szuberski2021-11-302-2/+1
| | | | | | | | | | | | | | | - Remove `SHELLCHECK_IGNORE` in favor of inline suppressions and more general `SHELLCHECK_OPTS`. - Exclude `SC2250` (turned on by `--enable=all`) globally - Pass `--enable=all` to shellcheck for scripts in contrib/: it's very important to catch errors early in areas that are not easily testable. Reviewed-by: George Melikov <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: szubersk <[email protected]> Closes #12760
* Remove basename(1). Clean up/shorten some coreutils pipelinesнаб2021-11-114-8/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Basenames that remain, in cmd/zed/zed.d/statechange-led.sh: dev=$(basename "$(echo "$therest" | awk '{print $(NF-1)}')") vdev=$(basename "$ZEVENT_VDEV_PATH") I don't wanna interfere with #11988 scripts/zfs-tests.sh: SINGLETESTFILE=$(basename "$SINGLETEST") tests/zfs-tests/tests/functional/cli_user/zfs_list/zfs_list.kshlib: ACTUAL=$(basename $dataset) ACTUAL=$(basename $dataset) tests/zfs-tests/tests/functional/cli_user/zpool_iostat/ zpool_iostat_-c_homedir.ksh: typeset USER_SCRIPT=$(basename "$USER_SCRIPT_FULL") tests/zfs-tests/tests/functional/cli_user/zpool_iostat/ zpool_iostat_-c_searchpath.ksh: typeset CMD_1=$(basename "$SCRIPT_1") typeset CMD_2=$(basename "$SCRIPT_2") tests/zfs-tests/tests/functional/cli_user/zpool_status/ zpool_status_-c_homedir.ksh: typeset USER_SCRIPT=$(basename "$USER_SCRIPT_FULL") tests/zfs-tests/tests/functional/cli_user/zpool_status/ zpool_status_-c_searchpath.ksh typeset CMD_1=$(basename "$SCRIPT_1") typeset CMD_2=$(basename "$SCRIPT_2") tests/zfs-tests/tests/functional/migration/migration.cfg: export BNAME=`basename $TESTFILE` tests/zfs-tests/tests/perf/perf.shlib: typeset logbase="$(get_perf_output_dir)/$(basename \ tests/zfs-tests/tests/perf/perf.shlib: typeset logbase="$(get_perf_output_dir)/$(basename \ These are potentially Of Directories, where basename is actually useful Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: John Kennedy <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12652
* Update `checkstyle` workflow env to ubuntu-20.04Damian Szuberski2021-11-021-5/+15
| | | | | | | | | - `checkstyle` workflow uses ubuntu-20.04 environment - improved `mancheck.sh` readability Reviewed-by: Matthew Ahrens <[email protected]> Reviewed-by: John Kennedy <[email protected]> Signed-off-by: szubersk <[email protected]> Closes #12713
* Add zfs.sh -r flag to reload modulesRyan Moeller2021-08-131-2/+10
| | | | | | | | | | | | | zfs.sh already can load and unload, so why not both? This is convenient when developing changes to the module and you want to rapidly make some changes, rebuild the module, reload the module, and test the changes. Reviewed-by: George Melikov <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #12450
* Assorted fixes for the performance testsJohn Wren Kennedy2021-07-261-8/+19
| | | | | | | | | | | | | | | | | | | | | - Bail out early if we're running the perf tests and forget to specify disks. - Allow perf tests to run with any number of disks. - Remove weekly vs. nightly settings - Move variables with common values to perf.shlib - Use zinject to clear the ARC over export/import - Fix dbuf cache size calculation When the meaning of `dbuf_cache_max_bytes` changed, the performance test that covers the dbuf cache started to fail. The test would try to write files for the test using the max possible size of the cache, inevitably filling the pool and failing. This change uses `dbuf_cache_shift` to correctly calculate the dbuf cache size. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Tony Nguyen <[email protected]> Reviewed-by: George Melikov <[email protected]> Signed-off-by: John Kennedy <[email protected]> Closes #12408
* zloop: Add a max iterations option, use default run/pass timesRyan Moeller2021-07-221-27/+33
| | | | | | | | | | | | | | | | | | | | | | | It is useful to have control over the number of iterations of zloop so we can easily produce "x core dumps found *in y iterations*" metrics. Using random values for run/pass times doesn't improve coverage in a meaningful way. Randomizing run time could be seen as a compromise between running a greater variety of shorter tests versus a smaller variety of longer tests within a fixed time span. However, it is not desirable when running a fixed number of iterations. Pass time already incorporates randomness within ztest. Either parameter can be passed to ztest explicitly if the defaults are not satisfactory. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: George Melikov <[email protected]> Reviewed-by: John Kennedy <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #12411
* scripts/commitcheck.sh: fix false positive for signed commitsнаб2021-06-111-6/+6
| | | | | | | | Reviewed-by: John Kennedy <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12105
* Move properties, parameters, events, and concepts around manual sectionsнаб2021-06-091-1/+1
| | | | | | | | | | | | | | | | | | | The pages moved as follows: zpool-features.{5 => 7} spl{-module-parameters.5 => .4} zfs{-module-parameters.5 => .4} zfs-events.5 => into zpool-events.8 zfsconcepts.{8 => 7} zfsprops.{8 => 7} zpoolconcepts.{8 => 7} zpoolprops.{8 => 7} Reviewed-by: Richard Laager <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Co-authored-by: Daniel Ebdrup Jensen <[email protected]> Closes #12149 Closes #12212
* Fix minor shellcheck 0.7.2 warningsBrian Behlendorf2021-06-092-2/+2
| | | | | | | | | | | | | | | | | The first warning of a misspelling is a false positive, so we annotate the script accordingly. As for the x-prefix warnings update the check to use the conventional '[ -z <string> ]' syntax. all-syslog.sh:46:47: warning: Possible misspelling: ZEVENT_ZIO_OBJECT may not be assigned, but ZEVENT_ZIO_OBJSET is. [SC2153] make_gitrev.sh:53:6: note: Avoid x-prefix in comparisons as it no longer serves a purpose [SC2268] man-dates.sh:10:7: note: Avoid x-prefix in comparisons as it no longer serves a purpose [SC2268] Reviewed-by: Ahelenia Ziemiańska <[email protected]> Reviewed-by: John Kennedy <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #12208
* Modernise/fix/rewrite unlinted manpagesнаб2021-06-071-1/+1
| | | | | | | | | | | | | | | | | | | zpool-destroy.8: flatten, fix description zfs-wait.8: flatten, fix description, use list for events zpool-reguid.8: flatten, fix description zpool-history.8: flatten, fix description zpool-export.8: flatten, fix description, remove -f "unmount" reference AFAICT no such command exists even in Illumos (as of today, anyway), and we definitely don't call it zpool-labelclear.8: flatten, fix description zpool-features.5: modernise spl-module-parameters.5: modernise zfs-mount-generator.8: rewrite zfs-module-parameters.5: modernise Reviewed-by: Richard Laager <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12169
* mancheck: accept lints, accept lint overridesнаб2021-06-042-1/+45
| | | | | | | | | Reviewed-by: Richard Laager <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Tony Nguyen <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12129
* Turn checkbashisms into a make targetнаб2021-06-012-4/+3
| | | | | | | | | make_gitrev.sh actually breaks checkbashisms' parser, which /insists/ that the end-of-line " is actually a string start Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12101
* Turn shellcheck into a normal make target. Fix new files it caughtнаб2021-06-014-19/+32
| | | | | | | | | | | This checks every file it checked (and a few more), but explicitly instead of "if it works it works" best-effort (which wasn't that good anyway) Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #10512 Closes #12101
* Trim excess shellcheck annotations. Widen to all non-Korn scriptsнаб2021-05-203-14/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Before, make shellcheck checked scripts/{commitcheck,make_gitrev,man-dates,paxcheck,zfs-helpers,zfs, zfs-tests,zimport,zloop}.sh cmd/zed/zed.d/{{all-debug,all-syslog,data-notify,generic-notify, resilver_finish-start-scrub,scrub_finish-notify, statechange-led,statechange-notify,trim_finish-notify, zed-functions}.sh,history_event-zfs-list-cacher.sh.in} cmd/zpool/zpool.d/{dm-deps,iostat,lsblk,media,ses,smart,upath} now it also checks contrib/dracut/{02zfsexpandknowledge/module-setup, 90zfs/{export-zfs,parse-zfs,zfs-needshutdown, zfs-load-key,zfs-lib,module-setup, mount-zfs,zfs-generator}}.sh.in cmd/zed/zed.d/{pool_import-led,vdev_attach-led, resilver_finish-notify,vdev_clear-led}.sh contrib/initramfs/{zfsunlock,hooks/zfs.in,scripts/local-top/zfs} tests/zfs-tests/tests/perf/scripts/prefetch_io.sh scripts/common.sh.in contrib/bpftrace/zfs-trace.sh autogen.sh Reviewed-by: John Kennedy <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12042
* Fix SC2181 ("[ $?") outside tests/наб2021-05-204-15/+6
| | | | | | | Reviewed-by: John Kennedy <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12042
* Replace ZoL with OpenZFS where applicableнаб2021-05-071-3/+3
| | | | | | | | | | | | | | | | Afterward, git grep ZoL matches: * README.md: * [ZoL Site](https://zfsonlinux.org) - Correct * etc/default/zfs.in:# ZoL userland configuration. - Changing this would induce a needless upgrade-check, if the user has modified the configuration; this can be updated the next time the defaults change * module/zfs/dmu_send.c: * ZoL < 0.7 does not handle [...] - Before 0.7 is ZoL, so fair enough Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Issue #11956
* Add kmodtool fix to detect different System.map locationMarcin Skarbek2021-04-071-2/+14
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Marcin Skarbek <[email protected]> Closes #7807 Closes #11836
* Fix error message when zfs module are already unloadedAdam D. Moss2021-02-201-1/+1
| | | | | | | | | Using zfs-sh -u on linux will fail with inaccurate message when the zfs modules are already unloaded. Deal with the case where a module is already unloaded; its USE_COUNT will be the empty string Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Adam Moss <[email protected]> Closes #11627
* Add "compatibility" property for zpool feature setsColm2021-02-171-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Property to allow sets of features to be specified; for compatibility with specific versions / releases / external systems. Influences the behavior of 'zpool upgrade' and 'zpool create'. Initial man page changes and test cases included. Brief synopsis: zpool create -o compatibility=off|legacy|file[,file...] pool vdev... compatibility = off : disable compatibility mode (enable all features) compatibility = legacy : request that no features be enabled compatibility = file[,file...] : read features from specified files. Only features present in *all* files will be enabled on the resulting pool. Filenames may be absolute, or relative to /etc/zfs/compatibility.d or /usr/share/zfs/compatibility.d (/etc checked first). Only affects zpool create, zpool upgrade and zpool status. ABI changes in libzfs: * New function "zpool_load_compat" to load and parse compat sets. * Add "zpool_compat_status_t" typedef for compatibility parse status. * Add ZPOOL_PROP_COMPATIBILITY to the pool properties enum * Add ZPOOL_STATUS_COMPATIBILITY_ERR to the pool status enum An initial set of base compatibility sets are included in cmd/zpool/compatibility.d, and the Makefile for cmd/zpool is modified to install these in $pkgdatadir/compatibility.d and to create symbolic links to a reasonable set of aliases. Reviewed-by: ericloewe Reviewed-by: Matthew Ahrens <[email protected]> Reviewed-by: Richard Laager <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Colm Buckley <[email protected]> Closes #11468
* DKMS: Disable weak modulesgregory-lee-bartholomew2020-12-151-0/+1
| | | | | | | | | | | | Fedora does not guarantee a stable kABI, so weak modules should be dis- abled. See the dkms man page for a more detailed explanation of the weak module feature. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Gregory Bartholomew <[email protected]> Closes #9891 Closes #11128 Closes #11242 Closes #11335
* Remove draid.d symlink from zfs_helpers.shBrian Behlendorf2020-12-111-3/+0
| | | | | | | | | | In an earlier revision of dRAID there existed an /etc/zfs/draid.d directory. This was removed before the final version was integrated but a little bit was accidentally overlooked in the zfs_helpers.sh script. Remove this remnant. Reviewed-by: George Melikov <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #11326
* Add compatibility for busybox mktempqzdanis2020-12-031-5/+5
| | | | | | | | | | | Busybox's mktemp requires at least six X's in the template, causing the current sed --in-place check to fail because the file does not exist. This change adds additional X's to mktemp templates that do not already have at least six X's in them. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Quentin Zdanis <[email protected]> Closes #11269
* zpool_influxdb: move to libexec dirPavel Snajdr2020-11-283-1/+4
| | | | | | | | | | | | Move the zpool_influxdb command to /usr/libexec/zfs, and include the /usr/libexec/zfs path in the system search directory when running the test suite. Reviewed-by: Richard Elling <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Pavel Snajdr <[email protected]> Closes #11156 Closes #11160 Closes #11224
* Distributed Spare (dRAID) FeatureBrian Behlendorf2020-11-133-10/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch adds a new top-level vdev type called dRAID, which stands for Distributed parity RAID. This pool configuration allows all dRAID vdevs to participate when rebuilding to a distributed hot spare device. This can substantially reduce the total time required to restore full parity to pool with a failed device. A dRAID pool can be created using the new top-level `draid` type. Like `raidz`, the desired redundancy is specified after the type: `draid[1,2,3]`. No additional information is required to create the pool and reasonable default values will be chosen based on the number of child vdevs in the dRAID vdev. zpool create <pool> draid[1,2,3] <vdevs...> Unlike raidz, additional optional dRAID configuration values can be provided as part of the draid type as colon separated values. This allows administrators to fully specify a layout for either performance or capacity reasons. The supported options include: zpool create <pool> \ draid[<parity>][:<data>d][:<children>c][:<spares>s] \ <vdevs...> - draid[parity] - Parity level (default 1) - draid[:<data>d] - Data devices per group (default 8) - draid[:<children>c] - Expected number of child vdevs - draid[:<spares>s] - Distributed hot spares (default 0) Abbreviated example `zpool status` output for a 68 disk dRAID pool with two distributed spares using special allocation classes. ``` pool: tank state: ONLINE config: NAME STATE READ WRITE CKSUM slag7 ONLINE 0 0 0 draid2:8d:68c:2s-0 ONLINE 0 0 0 L0 ONLINE 0 0 0 L1 ONLINE 0 0 0 ... U25 ONLINE 0 0 0 U26 ONLINE 0 0 0 spare-53 ONLINE 0 0 0 U27 ONLINE 0 0 0 draid2-0-0 ONLINE 0 0 0 U28 ONLINE 0 0 0 U29 ONLINE 0 0 0 ... U42 ONLINE 0 0 0 U43 ONLINE 0 0 0 special mirror-1 ONLINE 0 0 0 L5 ONLINE 0 0 0 U5 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 L6 ONLINE 0 0 0 U6 ONLINE 0 0 0 spares draid2-0-0 INUSE currently in use draid2-0-1 AVAIL ``` When adding test coverage for the new dRAID vdev type the following options were added to the ztest command. These options are leverages by zloop.sh to test a wide range of dRAID configurations. -K draid|raidz|random - kind of RAID to test -D <value> - dRAID data drives per group -S <value> - dRAID distributed hot spares -R <value> - RAID parity (raidz or dRAID) The zpool_create, zpool_import, redundancy, replacement and fault test groups have all been updated provide test coverage for the dRAID feature. Co-authored-by: Isaac Huang <[email protected]> Co-authored-by: Mark Maybee <[email protected]> Co-authored-by: Don Brady <[email protected]> Co-authored-by: Matthew Ahrens <[email protected]> Co-authored-by: Brian Behlendorf <[email protected]> Reviewed-by: Mark Maybee <[email protected]> Reviewed-by: Matt Ahrens <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #10102
* Fix commitcheck on FreeBSDRyan Moeller2020-10-201-44/+10
| | | | | | | | | Convert from bash to sh, avoid Perl regexes and \s, prune unused functions. Reviewed-by: Mateusz Piotrowski <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #11070
* Add zfs.sh module unload error messageAdam D. Moss2020-10-131-0/+3
| | | | | | | | If modules fail to unload because of outstanding users, don't consider this a success. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Adam Moss <[email protected]> Closes #11042
* Replace ZFS on Linux references with OpenZFSBrian Behlendorf2020-10-081-4/+4
| | | | | | | | | | | | | This change updates the documentation to refer to the project as OpenZFS instead ZFS on Linux. Web links have been updated to refer to https://github.com/openzfs/zfs. The extraneous zfsonlinux.org web links in the ZED and SPL sources have been dropped. Reviewed-by: George Melikov <[email protected]> Reviewed-by: Richard Laager <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #11007
* Update issue templates, commitcheck and Contributing.mdKjeld Schouten-Lebbing2020-09-231-59/+0
| | | | | | | | | | | | - Removes OpenZFS ports from commit check - Removes OpenZFS ports from CONTRIBUTING.md - Adds mailings lists and IRC to issue template selector - Remove blank issue option from issue creator Reviewed-by: George Melikov <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Co-authored-by: George Melikov <[email protected]> Signed-off-by: Kjeld Schouten-Lebbing <[email protected]> Closes #10965
* Add a missing option prefix `-` in zfs-tests.sh usage()Peter Dave Hello2020-09-081-1/+1
| | | | | | Reviewed-by: Giuseppe Di Natale <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Peter Dave Hello <[email protected]> Closes #10893
* ZTS: Remove leftover variable namesRyan Moeller2020-08-221-3/+0
| | | | | | | | | These were overlooked when use of `local` was removed to satisfy checkbashisms. Reviewed-by: Richard Laager <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10762
* Silence 'make checkbashisms'Brian Behlendorf2020-08-202-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Commit d2bce6d03 added the 'make checkbashisms' target but did not resolve all of the bashisms in the scripts. This commit doesn't resolve them all either but it does fix up a few, and it excludes the others so 'make checkstyle' no longer prints warnings. It's a small step in the right direction. * Dracut is Linux specific and itself depends on bash. Therefore all dracut support scripts can be bash specific, update their shebang accordingly. * zed-functions.sh, zfs-import, zfs-mount, zfs-zed, smart paxcheck.sh, make_gitrev.sh - these scripts were excuded from the check until they can be updated and properly tested. * zfsunlock - only whole values for sleep are allowed. * vdev_id - removed unneeded locals; use && instead of -a. * dkms.mkconf, dkms.postbuil - use || instead of -o. Reviewed-by: InsanePrawn <[email protected]> Reviewed-by: Gabriel A. Devenyi <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: George Melikov <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #10755
* Add zstd support to zfsMichael Niewöhner2020-08-203-5/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This PR adds two new compression types, based on ZStandard: - zstd: A basic ZStandard compression algorithm Available compression. Levels for zstd are zstd-1 through zstd-19, where the compression increases with every level, but speed decreases. - zstd-fast: A faster version of the ZStandard compression algorithm zstd-fast is basically a "negative" level of zstd. The compression decreases with every level, but speed increases. Available compression levels for zstd-fast: - zstd-fast-1 through zstd-fast-10 - zstd-fast-20 through zstd-fast-100 (in increments of 10) - zstd-fast-500 and zstd-fast-1000 For more information check the man page. Implementation details: Rather than treat each level of zstd as a different algorithm (as was done historically with gzip), the block pointer `enum zio_compress` value is simply zstd for all levels, including zstd-fast, since they all use the same decompression function. The compress= property (a 64bit unsigned integer) uses the lower 7 bits to store the compression algorithm (matching the number of bits used in a block pointer, as the 8th bit was borrowed for embedded block pointers). The upper bits are used to store the compression level. It is necessary to be able to determine what compression level was used when later reading a block back, so the concept used in LZ4, where the first 32bits of the on-disk value are the size of the compressed data (since the allocation is rounded up to the nearest ashift), was extended, and we store the version of ZSTD and the level as well as the compressed size. This value is returned when decompressing a block, so that if the block needs to be recompressed (L2ARC, nop-write, etc), that the same parameters will be used to result in the matching checksum. All of the internal ZFS code ( `arc_buf_hdr_t`, `objset_t`, `zio_prop_t`, etc.) uses the separated _compress and _complevel variables. Only the properties ZAP contains the combined/bit-shifted value. The combined value is split when the compression_changed_cb() callback is called, and sets both objset members (os_compress and os_complevel). The userspace tools all use the combined/bit-shifted value. Additional notes: zdb can now also decode the ZSTD compression header (flag -Z) and inspect the size, version and compression level saved in that header. For each record, if it is ZSTD compressed, the parameters of the decoded compression header get printed. ZSTD is included with all current tests and new tests are added as-needed. Per-dataset feature flags now get activated when the property is set. If a compression algorithm requires a feature flag, zfs activates the feature when the property is set, rather than waiting for the first block to be born. This is currently only used by zstd but can be extended as needed. Portions-Sponsored-By: The FreeBSD Foundation Co-authored-by: Allan Jude <[email protected]> Co-authored-by: Brian Behlendorf <[email protected]> Co-authored-by: Sebastian Gottschall <[email protected]> Co-authored-by: Kjeld Schouten-Lebbing <[email protected]> Co-authored-by: Michael Niewöhner <[email protected]> Signed-off-by: Allan Jude <[email protected]> Signed-off-by: Allan Jude <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Signed-off-by: Sebastian Gottschall <[email protected]> Signed-off-by: Kjeld Schouten-Lebbing <[email protected]> Signed-off-by: Michael Niewöhner <[email protected]> Closes #6247 Closes #9024 Closes #10277 Closes #10278
* cstyle.pl: echo commands for github workflowGeorge Melikov2020-08-131-4/+9
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: George Melikov <[email protected]> Closes #10705
* ZTS: Remove bashisms from zfs-tests.shRyan Moeller2020-08-071-65/+52
| | | | | | | | Bring zfs-tests.sh in to compliance with the other scripts by converting it /bin/sh for to avoid a dependency on bash. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10640
* Add zfs_gitrev.h to the distributed sourcesArvind Sankar2020-07-221-1/+29
| | | | | | | | | | | | | | | | | | | | | | | | | Commit 109d2c931020 ("Move zfs_gitrev.h to build directory") stopped distributing zfs_gitrev.h, as it is a generated file. Add it back, with some changes in behavior. Change the logic for gitrev as follows - if the source tree is a git repository, the behavior for build is unchanged. For make dist, append -dist to the git tag in the distributed version of zfs_gitrev.h. - otherwise, check if the source tree contains zfs_gitrev.h, and use it if so, falling back to "unknown" if it doesn't exist. - clean it only in make maintainer-clean, so we don't remove it from the source tree on make clean or make distclean. This allows disted sources to track what git tag they originally came from, with the -dist suffix indicating that the code wasn't built directly from git and so might contain additional changes beyond the git tag. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Eli Schwartz <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10595
* Restore scripts/make_gitrev.shArvind Sankar2020-07-222-0/+51
| | | | | | | | | | | | | | | | Commit 109d2c931020 ("Move zfs_gitrev.h to build directory") removed scripts/make_gitrev.sh, putting the logic into the Makefile itself. However, at least the Arch Linux packager wants the script so that the file can be generated without having to run configure first, for DKMS packaging purposes. So move the make recipe back into the script. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Eli Schwartz <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10595
* Centralize variable substitutionArvind Sankar2020-07-141-4/+0
| | | | | | | | | | | | A bunch of places need to edit files to incorporate the configured paths i.e. bindir, sbindir etc. Move this logic into a common file. Create arc_summary by copying arc_summary[23] as appropriate at build time instead of install time. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10559
* Drop unnecessary srcdir pathsArvind Sankar2020-06-241-5/+5
| | | | | | | | | | There's no need to specify the srcdir explicitly in _HEADERS and EXTRA_DIST. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10493
* Move zfs_gitrev.h to build directoryArvind Sankar2020-06-242-46/+0
| | | | | | | | | | | | | | | | | | | | | | Currently an out-of-tree build does not work with read-only source directory because zfs_gitrev.h can't be created. Move this file to the build directory, which is more appropriate for a generated file, and drop the dist-hook for zfs_gitrev.h. There is no need to distribute this file since it will be regenerated as part of the compilation in any case. scripts/make_gitrev.sh tries to avoid updating zfs_gitrev.h if there has been no change, however this doesn't cover the case when the source directory is not in git: in that case zfs_gitrev.h gets overwritten even though it's always "unknown". Simplify the logic to always write out a new version of zfs_gitrev.h, compare against the old and overwrite only if different. This is now simple enough to just include in the Makefile, so drop the script. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10493
* Add FreeBSD support to OpenZFSMatthew Macy2020-04-141-1/+1
| | | | | | | | | | | | | | | | | | Add the FreeBSD platform code to the OpenZFS repository. As of this commit the source can be compiled and tested on FreeBSD 11 and 12. Subsequent commits are now required to compile on FreeBSD and Linux. Additionally, they must pass the ZFS Test Suite on FreeBSD which is being run by the CI. As of this commit 1230 tests pass on FreeBSD and there are no unexpected failures. Reviewed-by: Sean Eric Fagan <[email protected]> Reviewed-by: Jorgen Lundman <[email protected]> Reviewed-by: Richard Laager <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Co-authored-by: Ryan Moeller <[email protected]> Signed-off-by: Matt Macy <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #898 Closes #8987
* zloop.sh should call ZDB with pool nameOlaf Faaland2020-03-111-1/+1
| | | | | | | | | | | Commit 54007c79 introduced an error, changing the final argument to $ZDB from ztest to $ZTEST. This argument indicates the pool name, not the script, and so should not have been changed. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Olaf Faaland <[email protected]> Closes #10118
* bash scripts: use /usr/bin/env for bash shebangsGraham Christensen2020-02-103-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Not all systems / distros have a `/bin/bash`, and these scripts are more difficult to run at development time. For example, my system is NixOS which doesn't have a /bin/bash. This is not a problem for NixOS building ZFS as a package: the build environment automatically replaces these shebangs with corrected paths. The problem is much more annoying at development time: either the scripts don't run, or I correct them for my local machine and deal with a perpetually dirty work tree. Before committing this patch I confirmed there are existing scripts which use `/usr/bin/env` to locate bash, so I am thinking this is a safe transformation. There are a handful of other shebangs in this repository which don't work on my system. This patch is useful on its own specifically for `commitcheck.sh`, otherwise I can't validate my commits before submission. Here are the remaining shebangs which NixOS systems won't have: 1274 #!/bin/ksh -p 91 #!/bin/ksh 89 #! /bin/ksh -p 2 #!/bin/sed -f 1 #!/usr/bin/perl -w 1 #!/usr/bin/ksh 1 #!/bin/nawk -f plus this which will create an invalid shebang in `tests/zfs-tests/tests/functional/mv_files/mv_files_common.kshlib`: echo "#!/bin/ksh" > $TEST_BASE_DIR/exitsZero.ksh I chose to leave those alone for now, and gauge the interest in this much smaller patch first. The fixes for these are easy enough by simply using `/usr/bin/env ksh`: 91 #!/bin/ksh 1 #!/usr/bin/ksh The fix for the other set is much trickier. Quoting the GNU coreutils manual: Most operating systems (e.g. GNU/Linux, BSDs) treat all text after the first space as a single argument. When using env in a script it is thus not possible to specify multiple arguments. and not all `env`'s support arguments. Mine (GNU Coreutils 8.31) does, though this feature is new since April 2018, GNU Coreutils 8.30: https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=668306ed86c8c79b0af0db8b9c882654ebb66db2 and worse, requires the -S argument: -S, --split-string=S process and split S into separate arguments; used to pass multiple arguments on shebang lines Example: $ seq 1 2 | $(nix-build '<nixpkgs>' -A coreutils)/bin/env "sort -nr" /nix/[...]-coreutils-8.31/bin/env: ‘sort -nr’: No such file or directory /nix/[...]-coreutils-8.31/bin/env: use -[v]S to pass options in shebang lines $ seq 1 2 | $(nix-build '<nixpkgs>' -A coreutils)/bin/env "-S sort -nr" 2 1 GNU Coreutils says FreeBSD's `env` does, though I wonder if FreeBSD's would be unhappy with the `-S`: https://www.gnu.org/software/coreutils/manual/html_node/env-invocation.html#env-invocation BusyBox v1.30.1 does not, and does not have a `-S`-like option: $ seq 1 2 | $(nix-build '<nixpkgs>' -A busybox)/bin/env "sort -nr" env: can't execute 'sort -nr': No such file or directory Toybox 0.8.1 also does not, and also does not have a `-S` option: $ seq 1 2 | $(nix-build '<nixpkgs>' -A toybox)/bin/env "sort -nr" env: exec sort -nr: No such file or directory --- At any rate, if this patch merges and the remaining ~1,500 are updated, the much larger patch should probably include a checkstyle-like test asserting all new shebangs use `/usr/bin/env`. I also don't mind dealing with NixOS weirdness if the project would prefer that. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Graham Christensen <[email protected]> Closes #9893
* ZTS: Reverse constrained path lookup orderRyan Moeller2020-01-291-2/+5
| | | | | | | | | | | | FreeBSD base system zfs utils are in /sbin. ZoF utils install to /usr/local/sbin. Ensure we link to the ZoF utils not the base utils when searching for utils to constrain paths to for the tests. Reviewed-by: John Kennedy <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #9906
* scripts/zfs-test.sh: example for -tChristian Schwarz2020-01-231-0/+3
| | | | | | | | Add an example for running a single test case. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Christian Schwarz <[email protected]> Closes #9878
* Add FreeBSD core handling in zloop.shRyan Moeller2020-01-022-15/+31
| | | | | | | And use the correct path to libtool and ztest. Reviewed-By: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #9790
* Update zfs.sh work on FreeBSDRyan Moeller2019-12-192-16/+54
| | | | | | | | Extend the zfs.sh script to load and unload zfs kmods on FreeBSD. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Kjeld Schouten <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #9746
* Update ZTS to work on FreeBSDMatthew Macy2019-12-181-45/+121
| | | | | | | | | | | | Update the common ZTS scripts and individual test cases as needed in order to allow them to be run on FreeBSD. The high level goal is to provide compatibility wrappers whenever possible to minimize changes to individual test cases. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: John Kennedy <[email protected]> Signed-off-by: Matt Macy <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #9692
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-29 17:47:17 +0000