summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* Disable -Wl,-z,defs for ASAN buildsJoao Carlos Mendes Luis2020-07-145-5/+25
| | | | | | | | | | | Commit af65916 added -Wl,-z,defs for the shared libraries. This apparently does not work in some cases with --enable-asan, so only add it for non-ASAN builds. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: João Carlos Mendes Luis <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10557 Closes #10560
* Fixing gang ABD child removal race conditionBrian Atkinson2020-07-144-4/+20
| | | | | | | | | | | | | | | | | | | | On linux the list debug code has been setting off a failure when checking that the node->next->prev value is pointing back at the node. At times this check evaluates to 0xdead. When removing a child from a gang ABD we must acquire the child's abd_mtx to make sure that the same ABD is not being added to another gang ABD while it is being removed from a gang ABD. This fixes a race condition when checking if an ABDs link is already active and part of another gang ABD before adding it to a gang. Added additional debug code for the gang ABD in abd_verify() to make sure each child ABD has active links. Also check to make sure another gang ABD is not added to a gang ABD. Reviewed-by: Serapheim Dimitropoulos <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Matt Ahrens <[email protected]> Signed-off-by: Brian Atkinson <[email protected]> Closes #10511
* Remove dependency on sharetab file and refactor sharing logicGeorge Wilson2020-07-1330-1651/+1545
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | == Motivation and Context The current implementation of 'sharenfs' and 'sharesmb' relies on the use of the sharetab file. The use of this file is os-specific and not required by linux or freebsd. Currently the code must maintain updates to this file which adds complexity and presents a significant performance impact when sharing many datasets. In addition, concurrently running 'zfs sharenfs' command results in missing entries in the sharetab file leading to unexpected failures. == Description This change removes the sharetab logic from the linux and freebsd implementation of 'sharenfs' and 'sharesmb'. It still preserves an os-specific library which contains the logic required for sharing NFS or SMB. The following entry points exist in the vastly simplified libshare library: - sa_enable_share -- shares a dataset but may not commit the change - sa_disable_share -- unshares a dataset but may not commit the change - sa_is_shared -- determine if a dataset is shared - sa_commit_share -- notify NFS/SMB subsystem to commit the shares - sa_validate_shareopts -- determine if sharing options are valid The sa_commit_share entry point is provided as a performance enhancement and is not required. The sa_enable_share/sa_disable_share may commit the share as part of the implementation. Libshare provides a framework for both NFS and SMB but some operating systems may not fully support these protocols or all features of the protocol. NFS Operation: For linux, libshare updates /etc/exports.d/zfs.exports to add and remove shares and then commits the changes by invoking 'exportfs -r'. This file, is automatically read by the kernel NFS implementation which makes for better integration with the NFS systemd service. For FreeBSD, libshare updates /etc/zfs/exports to add and remove shares and then commits the changes by sending a SIGHUP to mountd. SMB Operation: For linux, libshare adds and removes files in /var/lib/samba/usershares by calling the 'net' command directly. There is no need to commit the changes. FreeBSD does not support SMB. == Performance Results To test sharing performance we created a pool with an increasing number of datasets and invoked various zfs actions that would enable and disable sharing. The performance testing was limited to NFS sharing. The following tests were performed on an 8 vCPU system with 128GB and a pool comprised of 4 50GB SSDs: Scale testing: - Share all filesystems in parallel -- zfs sharenfs=on <dataset> & - Unshare all filesystems in parallel -- zfs sharenfs=off <dataset> & Functional testing: - share each filesystem serially -- zfs share -a - unshare each filesystem serially -- zfs unshare -a - reset sharenfs property and unshare -- zfs inherit -r sharenfs <pool> For 'zfs sharenfs=on' scale testing we saw an average reduction in time of 89.43% and for 'zfs sharenfs=off' we saw an average reduction in time of 83.36%. Functional testing also shows a huge improvement: - zfs share -- 97.97% reduction in time - zfs unshare -- 96.47% reduction in time - zfs inhert -r sharenfs -- 99.01% reduction in time Reviewed-by: Matt Ahrens <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Bryant G. Ly <[email protected]> Signed-off-by: George Wilson <[email protected]> External-Issue: DLPX-68690 Closes #1603 Closes #7692 Closes #7943 Closes #10300
* filesystem_limit/snapshot_limit is incorrectly enforced against rootMatthew Ahrens2020-07-1117-48/+104
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The filesystem_limit and snapshot_limit properties limit the number of filesystems or snapshots that can be created below this dataset. According to the manpage, "The limit is not enforced if the user is allowed to change the limit." Two types of users are allowed to change the limit: 1. Those that have been delegated the `filesystem_limit` or `snapshot_limit` permission, e.g. with `zfs allow USER filesystem_limit DATASET`. This works properly. 2. A user with elevated system privileges (e.g. root). This does not work - the root user will incorrectly get an error when trying to create a snapshot/filesystem, if it exceeds the `_limit` property. The problem is that `priv_policy_ns()` does not work if the `cred_t` is not that of the current process. This happens when `dsl_enforce_ds_ss_limits()` is called in syncing context (as part of a sync task's check func) to determine the permissions of the corresponding user process. This commit fixes the issue by passing the `task_struct` (typedef'ed as a `proc_t`) to syncing context, and then using `has_capability()` to determine if that process is privileged. Note that we still need to pass the `cred_t` to syncing context so that we can check if the user was delegated this permission with `zfs allow`. This problem only impacts Linux. Wrappers are added to FreeBSD but it continues to use `priv_check_cred()`, which works on arbitrary `cred_t`. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Matthew Ahrens <[email protected]> Closes #8226 Closes #10545
* libzfs: Add error message for why creating mountpoint failedRyan Moeller2020-07-112-2/+5
| | | | | | | | | | | | | When zfs_mount_at() fails to stat the mountpoint and can't create the directory, we return an error with a message "failed to create mountpoint" but there is no indication why it failed. Add the error string from the syscall to the error aux message. Update do_mount for Linux to return the errno instead of -1. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10550
* FreeBSD: Use a hash table for taskqid lookupsMatthew Macy2020-07-112-52/+128
| | | | | | | | | Previously a tqent could be recycled prematurely, update the code to use a hash table for lookups to resolve this. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Matt Macy <[email protected]> Closes #10529
* Unconditionally enable debugging for libzpoolSerapheim Dimitropoulos2020-07-106-4/+10
| | | | | | | | | | | | | | | | | | | | | | | | We already enable -DDEBUG unconditionally (meaning regardless of this is a debug build or a performance build) for zdb and ztest as they are mostly used for development and debugging. This patch enables -DDEBUG for libzpool extending the debugging checks for zdb, ztest, and a couple of other test utilities. In addition to passing -DDEBUG we also enable -DZFS_DEBUG so all assertion checks work s expected. We do so not only in libzpool but in every utility that links to it, even if the utility doesn't directly use any functionality wrapped in ZFS_DEBUG macro definitions. The reason is that these utilities may still include headers that contain structs that have more fields when ZFS_DEBUG is defined. This can be a problem as enabling that flag for libzpool but not for zdb can lead into random problems (e.g. segmentation faults) as zdb may be have an incorrect view of a struct passed to it by libzpool. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Matthew Ahrens <[email protected]> Signed-off-by: Serapheim Dimitropoulos <[email protected]> Closes #10549
* Fix up FIND_SYSTEM_LIBRARY to work with cross-compilingArvind Sankar2020-07-108-65/+79
| | | | | | | | | | | | | | | | | | | | Make FIND_SYSTEM_LIBRARY respect a configured sysroot, otherwise it might find headers from the build machine and assume the library is available on the host/target. Tighten up error checking: if pkg-config or the user specified _CFLAGS or _LIBS but we can't find the header/library, issue a fatal error. Fix the -L flag to /usr/local/lib instead of just /usr/local. Clean out the _CFLAGS and _LIBS if we located something that we later find doesn't work. Rename FIND_SYSTEM_LIBRARY into the ZFS_AC_ scope. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10538
* Use abs_top_builddir when referencing librariesArvind Sankar2020-07-1024-64/+64
| | | | | | | | | | | | | | | | | | | | | libtool stores absolute paths in the dependency_libs component of the .la files. If the Makefile for a dependent library refers to the libraries by relative path, some libraries end up duplicated on the link command line. As an example, libzfs specifies libzfs_core, libnvpair and libuutil as dependencies to be linked in. The .la file for libzfs_core also specifies libnvpair, but using an absolute path, with the result that libnvpair is present twice in the linker command line for producing libzfs. While the only thing this causes is to slightly slow down the linking, we can avoid it by using absolute paths everywhere, including for convenience libraries just for consistency. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10538
* Add -z defs to LDFLAGSArvind Sankar2020-07-105-6/+8
| | | | | | | | | This will make sure the installed libraries are linked with everything they require. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10538
* Add config.rpath for AM_GNU_GETTEXTArvind Sankar2020-07-1010-12/+700
| | | | | | | | | | | | | | | | | | | | | | | | Commit e8864b1b28c2 ("config: libintl/libiconv for gettext() detection") added an empty config.rpath with a comment that the real one doesn't work with libtool. However, an empty config.rpath doesn't really work: eg. on FreeBSD, where libintl is in /usr/local/lib, configure thinks that gettext doesn't exist and NLS should be disabled, which currently isn't supported in the source, and hence requires manual workaround to directly link -lintl without relying on configure. config.rpath is essential to let it be detected either in --prefix or using --with-libintl-prefix. I also don't see the mentioned issue with libtool flags applied to compilation, it seems to work fine to pass LTLIBINTL to libtool. It's unnecessary to include LTLIBICONV as the configure test will automatically append that to LTLIBINTL if it is necessary to link with libiconv. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10538
* Clean up lib dependenciesArvind Sankar2020-07-1045-167/+170
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | libzutil is currently statically linked into libzfs, libzfs_core and libzpool. Avoid the unnecessary duplication by removing it from libzfs and libzpool, and adding libzfs_core to libzpool. Remove a few unnecessary dependencies: - libuutil from libzfs_core - libtirpc from libspl - keep only libcrypto in libzfs, as we don't use any functions from libssl - librt is only used for clock_gettime, however on modern systems that's in libc rather than librt. Add a configure check to see if we actually need librt - libdl from raidz_test Add a few missing dependencies: - zlib to libefi and libzfs - libuuid to zpool, and libuuid and libudev to zed - libnvpair uses assertions, so add assert.c to provide aok and libspl_assertf Sort the LDADD for programs so that libraries that satisfy dependencies come at the end rather than the beginning of the linker command line. Revamp the configure tests for libaries to use FIND_SYSTEM_LIBRARY instead. This can take advantage of pkg-config, and it also avoids polluting LIBS. List all the required dependencies in the pkgconfig files, and move the one for libzfs_core into the latter's directory. Install pkgconfig files in $(libdir)/pkgconfig on linux and $(prefix)/libdata/pkgconfig on FreeBSD, instead of /usr/share/pkgconfig, as the more correct location for library .pc files. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10538
* Move libspl_assertf into .c fileArvind Sankar2020-07-106-33/+64
| | | | | | | | | | | | Variadic functions cannot be inlined. libspl_assertf ends up being duplicated in every file that uses it. Fix this by moving the function into a new assert.c. Also move the definition of aok into the new file instead of zone.c. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10538
* Fix a persistent L2ARC bug in l2arc_write_done()George Amanakis2020-07-101-5/+27
| | | | | | | | | | | | | | In case l2arc_write_done() handles a zio that was not successful check that the list of log block pointers is not empty when restoring them in the device header. Otherwise zero them out. In any case perform the actual write updating the device header after the zio of l2arc_write_buffers() completes as l2arc_write_done() may have touched the memory holding the log block pointers in the device header. Reviewed-by: Serapheim Dimitropoulos <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: George Amanakis <[email protected]> Closes #10540 Closes #10543
* Add Makefile command to run checkbashisms on all /bin/sh scriptsGabriel A. Devenyi2020-07-091-1/+17
| | | | | | | | Based on the shellcheck make target, add a target which checks for violations of POSIX standards for shell scripts Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Gabriel A. Devenyi <[email protected]> Closes #10513
* ZTS: Make bc conditional use compatible with new BSD bcRyan Moeller2020-07-091-2/+3
| | | | | | | | | | | | | | | | FreeBSD recently replaced the GNU bc and dc in the base system with BSD licensed versions. They are supposed to be compatible with all the features present in the GNU versions, but it turns out they are picky about `if` statements having a corresponding `else`. ZTS uses `echo "if ($x > $y) 1" | bc` in a few places, which causes tests to fail unexpectedly with the new bc. Change the two expressions in ZTS to `if ($x > $y) 1 else 0` for compatibility with the new BSD bc. Reviewed-by: John Kennedy <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10551
* libzfs: Make zfs_cmd_t initialization consistent, use zfs_ioctlRyan Moeller2020-07-093-6/+8
| | | | | | | | | | | | | | | The clang version 8.0.1 shipped in FreeBSD 12.1-RELEASE also oddly throws a warning that is treated as an error on the initialization of the zc struct in zpool_nextboot. The zpool_nextboot code from FreeBSD was not updated to use zfs_ioctl. Switch ioctl to zfs_ioctl in and use {"\0"} to initialize the struct. Do a consistency pass for zfs_cmd_t initialization. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: George Wilson <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10539
* Add zpool_nextboot, move zfs_jail to libzfs.hRyan Moeller2020-07-063-6/+37
| | | | | | | | | | | | FreeBSD has a zfsbootcfg command that wants zpool_nextboot in libzfs. Add the function to FreeBSD's libzfs_compat.c, and while here move the prototype for zfs_jail out of param.h in FreeBSD's SPL and into libzfs.h under an ifdef for FreeBSD, where the prototype for zpool_nextboot joins it. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10524
* Fix a deadlock in the FreeBSD getpages VOPMark Johnston2020-07-061-1/+12
| | | | | | | | | | | | | | | | | | FreeBSD has a per-page "busy" lock which is held when handling a page fault on a mapped file. This lock is also acquired when copying data from the DMU to the page cache in zfs_write(). File range locks are also acquired in both of these paths, in the opposite order with respect to the busy lock. In the getpages VOP, the range lock is only used to determine the extent of optional read-ahead and read-behind operations. To resolve the lock order reversal, modify the getpages VOP to avoid blocking on the range lock. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Mark Johnston <[email protected]> Closes #10519
* Add a "try" operation for range locksMark Johnston2020-07-062-18/+49
| | | | | | | | | | | zfs_rangelock_tryenter() bails immediately instead of waiting for the lock to become available. This will be used to resolve a deadlock in the FreeBSD page-in code. No functional change intended. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Mark Johnston <[email protected]> Closes #10519
* Fix atomic_clear_long_excl wrong returnwinglq2020-07-061-1/+1
| | | | | | | | | When clearing a bit, we should check whether that bit is 0. Note atomic_clear_long_excl is not used. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Liu Qing <[email protected]> Closes #10526
* Update zfs_freebsd_need_inactive to fix mmapped writesRyan Moeller2020-07-031-2/+5
| | | | | | | | | | | | | `zfs_freebsd_need_inactive` appears to been based on an unfinished version of https://reviews.freebsd.org/D22130 which had a bug where files written via mmap wouldn't actually persist. Update the function to match the final version committed to FreeBSD. Authored-by: Mateusz Guzik <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10527 Closes #10528
* Add device rebuild featureBrian Behlendorf2020-07-0365-362/+3281
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The device_rebuild feature enables sequential reconstruction when resilvering. Mirror vdevs can be rebuilt in LBA order which may more quickly restore redundancy depending on the pools average block size, overall fragmentation and the performance characteristics of the devices. However, block checksums cannot be verified as part of the rebuild thus a scrub is automatically started after the sequential resilver completes. The new '-s' option has been added to the `zpool attach` and `zpool replace` command to request sequential reconstruction instead of healing reconstruction when resilvering. zpool attach -s <pool> <existing vdev> <new vdev> zpool replace -s <pool> <old vdev> <new vdev> The `zpool status` output has been updated to report the progress of sequential resilvering in the same way as healing resilvering. The one notable difference is that multiple sequential resilvers may be in progress as long as they're operating on different top-level vdevs. The `zpool wait -t resilver` command was extended to wait on sequential resilvers. From this perspective they are no different than healing resilvers. Sequential resilvers cannot be supported for RAIDZ, but are compatible with the dRAID feature being developed. As part of this change the resilver_restart_* tests were moved in to the functional/replacement directory. Additionally, the replacement tests were renamed and extended to verify both resilvering and rebuilding. Original-patch-by: Isaac Huang <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: John Poduska <[email protected]> Co-authored-by: Mark Maybee <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #10349
* freebsd: changes necessary to coexist with dtrace in treeMatthew Macy2020-07-0113-14/+33
| | | | | | | | Fix header conflicts when building zfs with openzfs as a vendor import. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Matt Macy <[email protected]> Closes #10497
* configure fixesHarald van Dijk2020-07-012-20/+20
| | | | | | | | | | | | | | | | | | a+=b is not supported by all shells. It is equivalent to a=${a}b, so just rewrite it as that. This also fixes commit 9ea6c3d3, which intended to only make the definitions of _dracutdir, _udevdir, and _udevruledir conditional, but actually ensured that _initconfdir no longer got defined if _dracutdir was defined, and defined _udevdir to the value that should have been used for _udevruledir. This also fixes the fact that the checks introduced by commit 9ea6c3d3 could never work: ZFS_AC_PACKAGE was called before the configuration options were processed. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Harald van Dijk <[email protected]> Closes #10518
* pam: fix test usage in configure scriptHarald van Dijk2020-06-291-2/+2
| | | | | | | | | | The standard test command does not support the == operator. Certain shells, including bash, do support it, but in those shells it does exactly the same thing as the standard = operator. Use that instead. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Harald van Dijk <[email protected]> Closes #10509
* Clean up OS-specific ARC and kmem codeMatthew Ahrens2020-06-2911-177/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | OS-specific code (e.g. under `module/os/linux`) does not need to share its code structure with any other operating systems. In particular, the ARC and kmem code need not be similar to the code in illumos, because we won't be syncing this OS-specific code between operating systems. For example, if/when illumos support is added to the common repo, we would add a file `module/os/illumos/zfs/arc_os.c` for the illumos versions of this code. Therefore, we can simplify the code in the OS-specific ARC and kmem routines. These changes do not impact system behavior, they are purely code cleanup. The changes are: Arenas are not used on Linux or FreeBSD (they are always `NULL`), so `heap_arena`, `zio_arena`, and `zio_alloc_arena` can be removed, along with code that uses them. In `arc_available_memory()`: * `desfree` is unused, remove it * rename `freemem` to avoid conflict with pre-existing `#define` * remove checks related to arenas * use units of bytes, rather than converting from bytes to pages and then back to bytes `SPL_KMEM_CACHE_REAP` is unused, remove it. `skc_reap` is unused, remove it. The `count` argument to `spl_kmem_cache_reap_now()` is unused, remove it. `vmem_size()` and associated type and macros are unused, remove them. In `arc_memory_throttle()`, use a less confusing variable name to store the result of `arc_free_memory()`. Reviewed-by: George Wilson <[email protected]> Reviewed-by: Pavel Zakharov <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Matthew Ahrens <[email protected]> Closes #10499
* Mark phony targets as phony.Arvind Sankar2020-06-271-1/+20
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10506
* paxcheck needs to run against the builddir not the srcdirArvind Sankar2020-06-271-1/+1
| | | | | | | | Otherwise it does nothing on an out-of-tree build. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10506
* Move cppcheck suppressions out of .githubArvind Sankar2020-06-272-1/+2
| | | | | | | | Move this file out of .github and add it to distribution. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10506
* Remove config/suppressed-warnings.txtArvind Sankar2020-06-271-7/+0
| | | | | | | | | This doesn't appear to be used by the buildbot any more, let's remove it. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10506
* Avoid installing kernel headers on FreeBSDArvind Sankar2020-06-2714-20/+23
| | | | | | | | | The kernel headers are installed for DKMS on linux, so don't install them unless we're building on linux. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10506
* Style fixesBrian Behlendorf2020-06-272-1/+2
| | | | | | | * Fix cstyle issue in shrinker.h which exceeded 80 columns. * Silence shellcheck warning in zpool.d/smart script. Signed-off-by: Brian Behlendorf <[email protected]>
* Make zstreamdump output the size of the payload for BEGIN recordsAllan Jude2020-06-271-0/+2
| | | | | | | | | This is helpful for determining the size of the nvlist of snapshots and properties Reviewed-by: Matt Ahrens <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Allan Jude <[email protected]> Closes #10505
* Revise SPL wrapper for shrinker callbacksMatthew Ahrens2020-06-273-142/+83
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The SPL provides a wrapper for the kernel's shrinker callbacks, which enables the ZFS code to interface with multiple versions of the shrinker API's from different kernel versions. Specifically, Linux kernels 3.0 - 3.11 has a single "combined" callback, and Linux kernels 3.12 and later have two "split" callbacks. The SPL provides a wrapper function so that the ZFS code only needs to implement one version of the callbacks. Currently the SPL's wrappers are designed such that the ZFS code implements the older, "combined" callback. There are a few downsides to this approach: * The general design within ZFS is for the latest Linux kernel to be considered the "first class" API. * The newer, "split" callback API is easier to understand, because each callback has one purpose. * The current wrappers do not completely abstract out the differing API's, so ZFS code needs `#ifdef` code to handle the differing return values required for different kernel versions. This commit addresses these drawbacks by having the ZFS code provide the latest, "split" callbacks, and the SPL provides a wrapping function for the older, "combined" API. Reviewed-by: Pavel Zakharov <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Matthew Ahrens <[email protected]> Closes #10502
* Use percpu_counter for obj_alloc counter of Linux-backed cachesSerapheim Dimitropoulos2020-06-267-8/+100
| | | | | | | | | | | | | | | | | | | | | | | A previous commit enabled the tracking of object allocations in Linux-backed caches from the SPL layer for debuggability. The commit is: 9a170fc6fe54f1e852b6c39630fe5ef2bbd97c16 Unfortunately, it also introduced minor performance regressions that were highlighted by the ZFS perf test-suite. Within Delphix we found that the regression would be from -1%, all the way up to -8% for some workloads. This commit brings performance back up to par by creating a separate counter for those caches and making it a percpu in order to avoid lock-contention. The initial performance testing was done by myself, and the final round was conducted by @tonynguien who was also the one that discovered the regression and highlighted the culprit. Reviewed-by: Matt Ahrens <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Serapheim Dimitropoulos <[email protected]> Closes #10397
* arcstat: add 'avail', fix 'free'Matthew Ahrens2020-06-262-13/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The meaning of the `free` field is currently `zfs_arc_sys_free`, which is the target amount of memory to leave free for the system, and is constant after booting. This commit changes the meaning of `free` to arc_free_memory(), the amount of memory that the ARC considers to be free. It also adds a new arcstat field `avail`, which tracks `arc_available_memory()`. Since `avail` can be negative, it also updates the arcstat script to pretty-print negative values. example output: $ arcstat -f time,miss,arcsz,c,grow,need,free,avail 1 time miss arcsz c grow need free avail 15:03:02 39K 114G 114G 0 0 2.4G 407M 15:03:03 42K 114G 114G 0 0 2.1G 120M 15:03:04 40K 114G 114G 0 0 1.8G -177M 15:03:05 24K 113G 112G 0 0 1.7G -269M 15:03:06 29K 111G 110G 0 0 1.6G -385M 15:03:07 27K 110G 108G 0 0 1.4G -535M 15:03:08 13K 108G 108G 0 0 2.2G 239M 15:03:09 33K 107G 107G 0 0 1.3G -639M 15:03:10 16K 105G 102G 0 0 2.6G 704M 15:03:11 7.2K 102G 102G 0 0 5.1G 3.1G 15:03:12 42K 103G 102G 0 0 4.8G 2.8G Reviewed-by: George Melikov <[email protected]> Reviewed-by: Pavel Zakharov <[email protected]> Reviewed-by: Tony Nguyen <[email protected]> Signed-off-by: Matthew Ahrens <[email protected]> Closes #10494
* Add block histogram to zdbRobert Novak2020-06-264-2/+479
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The block histogram tracks the changes to psize, lsize and asize both in the count of the number of blocks (by blocksize) and the total length of all of the blocks for that blocksize. It also keeps a running total of the cumulative size of all of the blocks up to each size to help determine the size of caching SSDs to be added to zfs hardware deployments. The block history counts and lengths are summarized in bins which are powers of two. Even rows with counts of zero are printed. This change is accessed by specifying one of two options: zdb -bbb pool zdb -Pbbb pool The first version prints the table in fixed size columns. The second prints in "parseable" output that can be placed into a CSV file. Fixed Column, nicenum output sample: block psize lsize asize size Count Length Cum. Count Length Cum. Count Length Cum. 512: 3.50K 1.75M 1.75M 3.43K 1.71M 1.71M 3.41K 1.71M 1.71M 1K: 3.65K 3.67M 5.43M 3.43K 3.44M 5.15M 3.50K 3.51M 5.22M 2K: 3.45K 6.92M 12.3M 3.41K 6.83M 12.0M 3.59K 7.26M 12.5M 4K: 3.44K 13.8M 26.1M 3.43K 13.7M 25.7M 3.49K 14.1M 26.6M 8K: 3.42K 27.3M 53.5M 3.41K 27.3M 53.0M 3.44K 27.6M 54.2M 16K: 3.43K 54.9M 108M 3.50K 56.1M 109M 3.42K 54.7M 109M 32K: 3.44K 110M 219M 3.41K 109M 218M 3.43K 110M 219M 64K: 3.41K 218M 437M 3.41K 218M 437M 3.44K 221M 439M 128K: 3.41K 437M 874M 3.70K 474M 911M 3.41K 437M 876M 256K: 3.41K 874M 1.71G 3.41K 874M 1.74G 3.41K 874M 1.71G 512K: 3.41K 1.71G 3.41G 3.41K 1.71G 3.45G 3.41K 1.71G 3.42G 1M: 3.41K 3.41G 6.82G 3.41K 3.41G 6.86G 3.41K 3.41G 6.83G 2M: 0 0 6.82G 0 0 6.86G 0 0 6.83G 4M: 0 0 6.82G 0 0 6.86G 0 0 6.83G 8M: 0 0 6.82G 0 0 6.86G 0 0 6.83G 16M: 0 0 6.82G 0 0 6.86G 0 0 6.83G Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Robert E. Novak <[email protected]> Closes: #9158 Closes #10315
* Fixes for make distArvind Sankar2020-06-2655-261/+86
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Reduce the usage of EXTRA_DIST. If files are conditionally included in _SOURCES, _HEADERS etc, automake is smart enough to dist all files that could possibly be included, but this does not apply to EXTRA_DIST, resulting in make dist depending on the configuration. Add some files that were missing altogether in various Makefile's. The changes to disted files in this commit (excluding deleted files): +./cmd/zed/agents/README.md +./etc/init.d/README.md +./lib/libspl/os/freebsd/getexecname.c +./lib/libspl/os/freebsd/gethostid.c +./lib/libspl/os/freebsd/getmntany.c +./lib/libspl/os/freebsd/mnttab.c -./lib/libzfs/libzfs_core.pc -./lib/libzfs/libzfs.pc +./lib/libzfs/os/freebsd/libzfs_compat.c +./lib/libzfs/os/freebsd/libzfs_fsshare.c +./lib/libzfs/os/freebsd/libzfs_ioctl_compat.c +./lib/libzfs/os/freebsd/libzfs_zmount.c +./lib/libzutil/os/freebsd/zutil_compat.c +./lib/libzutil/os/freebsd/zutil_device_path_os.c +./lib/libzutil/os/freebsd/zutil_import_os.c +./module/lua/README.zfs +./module/os/linux/spl/README.md +./tests/README.md +./tests/zfs-tests/tests/functional/cli_root/zfs_clone/zfs_clone_rm_nested.ksh +./tests/zfs-tests/tests/functional/cli_root/zfs_send/zfs_send_encrypted_unloaded.ksh +./tests/zfs-tests/tests/functional/inheritance/README.config +./tests/zfs-tests/tests/functional/inheritance/README.state +./tests/zfs-tests/tests/functional/rsend/rsend_016_neg.ksh +./tests/zfs-tests/tests/perf/fio/sequential_readwrite.fio Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10501
* Include FreeBSD sources in module distArvind Sankar2020-06-261-8/+7
| | | | | | | | Add os/freebsd and Makefile.bsd into distdir target. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10501
* ARC shrinking blocks reads/writesMatthew Ahrens2020-06-263-2/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ZFS registers a memory hook, `__arc_shrinker_func`, which is supposed to allow the ARC to shrink when the kernel experiences memory pressure. The ARC shrinker changes `arc_c` via a call to `arc_reduce_target_size()`. Before commit 3ec34e55271d433e3c, the ARC shrinker would also evict data from the ARC to bring `arc_size` down to the new `arc_c`. However, that commit (seemingly inadvertently) made it so that the ARC shrinker no longer evicts any data or waits for eviction to complete. Repeated calls to the ARC shrinker can reduce `arc_c` drastically, often all the way to `arc_c_min`. Since it doesn't wait for the actual eviction of data from the ARC, this creates a situation where `arc_size` is more than `arc_c` for the several seconds/minutes it takes for `arc_adjust_zthr` to evict data from the ARC. During this time, arc_get_data_impl() will block, so ZFS can't process read/write requests (e.g. from iSCSI, NFS, or read/write syscalls). To ensure that `arc_c` doesn't shrink faster than the adjust thread can keep up, this commit makes the ARC shrinker wait for the eviction to complete, resulting in similar behavior to what we had before commit 3ec34e55271d433e3c. Note: commit 3ec34e55271d433e3c is `OpenZFS 9284 - arc_reclaim_thread has 2 jobs` and was integrated in December 2018, and is part of ZoL 0.8.x but not 0.7.x. Additionally, when the ARC size is reduced drastically, the `arc_adjust_zthr` can be on-CPU for many seconds without blocking. Any threads that are bound to the same CPU that arc_adjust_zthr is running on will not able to run for a long time. To ensure that CPU-bound threads can make progress, this commit changes `arc_evict_state_impl()` make a voluntary preemption call, `cond_resched()`. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: George Wilson <[email protected]> Reviewed-by: Prakash Surya <[email protected]> Reviewed-by: Pavel Zakharov <[email protected]> Reviewed-by: Tony Nguyen <[email protected]> Signed-off-by: Matthew Ahrens <[email protected]> External-issue: DLPX-70703 Closes #10496
* pam: implement a zfs_key pam modulefelixdoerre2020-06-2419-2/+1058
| | | | | | | | | | | | | | | | | Implements a pam module for automatically loading zfs encryption keys for home datasets. The pam module: - loads a zfs key and mounts the dataset when a session opens. - unmounts the dataset and unloads the key when the session closes. - when the user is logged on and changes the password, the module changes the encryption key. Reviewed-by: Richard Laager <[email protected]> Reviewed-by: @jengelh <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Felix Dörre <[email protected]> Closes #9886 Closes #9903
* Drop unnecessary srcdir pathsArvind Sankar2020-06-2438-463/+463
| | | | | | | | | | There's no need to specify the srcdir explicitly in _HEADERS and EXTRA_DIST. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10493
* Fix check for sed --in-placeArvind Sankar2020-06-245-15/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The test added in commit 4313a5b4c51e ("Detect if sed supports --in-place") doesn't work at least on my system (autoconfig-2.69). The issue is that SED has already been found and cached before this function is evaluated, with the result that the test is completely skipped. ... checking for a sed that does not truncate output... /usr/bin/sed ... checking for sed --in-place... (cached) /usr/bin/sed The first test is executed by libtool.m4. This looks to have been around in libtool for at least 15 years or so, not sure why this was not encountered at the time of the original commit. Fix this by caching the value of the ac_inplace flag rather than the path to SED. Also use $SED and add AC_REQUIRE to ensure that we use the sed that was located by the standard configure test. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10493
* Add missing third-party license files to distArvind Sankar2020-06-241-0/+4
| | | | | | | | | | | These license files were added in commit 31b160f0a6c6 ("ICP: Improve AES-GCM performance") but not added to the distributed files. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10493
* Fix tags targets in module/Makefile.in + cleanupArvind Sankar2020-06-242-30/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | These targets look to have been copied from an automake-generated Makefile.in, and can't work since none of the auto-generated automake variables are defined here. Moreover, ctags has been overridden in the top-level Makefile, so the target is pointless anyway, and gtags is not a recursive target. Fix cscopelist by moving it to the top-level Makefile as well, in line with ctags and etags. Also, add -a to ctags command as well, otherwise it won't work if more than one xargs invocation takes place. Add assembler files to ctags/etags, prune all dotted-dirs, and restrict the find to files only. Cleanup: add .PHONY to module/Makefile.in, and fix one recipe with a missing continuation character. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10493
* Fix libspl/asm-generic/atomic for VPATH buildArvind Sankar2020-06-244-29/+8
| | | | | | | | | | | | | | | | | | Currently, asm-generic/atomic.c is compiled into a .S file, with a comment saying this is to simplify the upper-level Makefile. However, this doesn't work properly with a VPATH build, which would require better logic to deal with generated sources correctly. It also doesn't seem more complex to just specify the .c/.S source file, depending on the cpu, instead of only the source directory in lib/libspl/Makefile.am, which eliminates the need to do the intermediate compilation. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10493
* Move zfs_gitrev.h to build directoryArvind Sankar2020-06-247-53/+23
| | | | | | | | | | | | | | | | | | | | | | Currently an out-of-tree build does not work with read-only source directory because zfs_gitrev.h can't be created. Move this file to the build directory, which is more appropriate for a generated file, and drop the dist-hook for zfs_gitrev.h. There is no need to distribute this file since it will be regenerated as part of the compilation in any case. scripts/make_gitrev.sh tries to avoid updating zfs_gitrev.h if there has been no change, however this doesn't cover the case when the source directory is not in git: in that case zfs_gitrev.h gets overwritten even though it's always "unknown". Simplify the logic to always write out a new version of zfs_gitrev.h, compare against the old and overwrite only if different. This is now simple enough to just include in the Makefile, so drop the script. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10493
* Support out-of-tree kmod build on FreeBSDArvind Sankar2020-06-241-7/+30
| | | | | | | | | | | | | | If srcdir != builddir, pass down MAKEOBJDIR to the FreeBSD make to support out-of-tree builds. Also allow passing all the gmake options that FreeBSD make understands to support useful flags like -k, -n, -q etc, and detect the number of CPUs if -j was specified without an argument. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10493
* Add trim_finish notify script for ZEDKevin P. Fleming2020-06-242-1/+39
| | | | | | | | | | | Allow users to configure notifications when TRIM operations are completed on pools. Unlike resilver_finish and scrub_finish, the trim_finish event is generated for each vdev in the pool which was trimmed, so the script will generate a notification for each one. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Kevin P. Fleming <[email protected]> Closes #10491
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-15 11:35:30 +0000