summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* Fixed LDADD library links in Makefiles for cross compilation buildsPetros Koutoupis2020-05-099-1/+12
| | | | | | | | | | | When building on native dev system, there are no issues but when cross-compiling for target system, some linker errors are observed. The only way to avoid these errors is by adjusting the Makefile.am of those various components to add the library dependencies. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: George Melikov <[email protected]> Signed-off-by: Petros Koutoupis <[email protected]> Closes #10304
* ZTS: refreserv_005_pos.kshBrian Behlendorf2020-05-081-3/+3
| | | | | | | | | | When recursively destroying the dataset it's possible for the dataset volume to be open by an unrelated process, like blkid. Use the destroy_dataset() which will retry when this occurs. Reviewed-by: John Kennedy <[email protected]> Reviewed-by: George Melikov <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #10305
* Unlock encrypted root partition over SSHAndrey Prokopenko2020-05-078-3/+92
| | | | | | | | | | | | | This commit add a new feature for Debian-based distributions to unlock encrypted root partition over SSH. This feature is very handy on headless NAS or VPS cloud servers. To use this feature, you will need to install the dropbear-initramfs package. Reviewed-By: Brian Behlendorf <[email protected]> Reviewed-By: Tom Caputi <[email protected]> Signed-off-by: Andrey Prokopenko <[email protected]> Signed-off-by: Richard Laager <[email protected]> Closes #10027
* Rework README.initramfs.markdownRichard Laager2020-05-071-94/+74
| | | | | | | | | | | | | This file is listed as being in Markdown format, but it didn't really use much Markdown. I have added a fair amount of formatting. I have reordered and reworded things to improve the flow of the text. Reviewed-By: Andrey Prokopenko <[email protected]> Reviewed-By: Brian Behlendorf <[email protected]> Reviewed-By: Tom Caputi <[email protected]> Signed-off-by: Richard Laager <[email protected]> Closes #10027
* Cleanup contrib/initramfs automakeRichard Laager2020-05-076-27/+16
| | | | | | | | | | | | | | | | | | | | | The initramfs hook scripts depend on Makefile. This way, if the substitution code is changed, they should update. This brings it in line with etc/init.d (which was modified to match the example in the automake docs). The initramfs hook script cleaning now matches etc/init.d. There was a mix of SUBDIRS recursion and custom install rules for files in subdirectories. This was duplicated for the "hooks" and "scripts" subdirectories. Now everything uses SUBDIRS. I fixed the substitution of DEFAULT_INITCONF_DIR for hooks/zfs. Reviewed-By: Andrey Prokopenko <[email protected]> Reviewed-By: Brian Behlendorf <[email protected]> Reviewed-By: Tom Caputi <[email protected]> Signed-off-by: Richard Laager <[email protected]> Closes #10027
* Improvements on persistent L2ARCGeorge Amanakis2020-05-075-126/+278
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Functional changes: We implement refcounts of log blocks and their aligned size on the cache device along with two corresponding arcstats. The refcounts are reflected in the header of the device and provide valuable information as to whether log blocks are accounted for correctly. These are dynamically adjusted as log blocks are committed/evicted. zdb also uses this information in the device header and compares it to the corresponding values as reported by dump_l2arc_log_blocks() which emulates l2arc_rebuild(). If the refcounts saved in the device header report higher values, zdb exits with an error. For this feature to work correctly there should be no active writes on the device. This is also employed in the tests of persistent L2ARC. We extend the structure of the cache device header by adding the two new variables mirroring the refcounts after the existing variables to preserve backward compatibility in terms of persistent L2ARC. 1) a new arcstat "l2_log_blk_asize" and refcount "l2ad_lb_asize" which reflect the total aligned size of log blocks on the device. This is also reflected in the header of the cache device as "dh_lb_asize". 2) a new arcstat "l2arc_log_blk_count" and refcount "l2ad_lb_count" which reflect the total number of L2ARC log blocks present on cache devices. It is also reflected in the header of the cache device as "dh_lb_count". In l2arc_rebuild_vdev() if the amount of committed log entries in a log block is 0 and the device header is valid we update the device header. This will facilitate trimming of the whole device in this case when TRIM for L2ARC is implemented. Improve loop protection in l2arc_rebuild() by using the starting offset of the payload of each log block instead of the starting offset of the log block. If the zio in l2arc_write_buffers() fails, restore the lbps array in the header of the device to its previous state in l2arc_write_done(). If l2arc_rebuild() ends the rebuild process without restoring any L2ARC log blocks in ARC and without any other error, this means that the lbps array in the header is pointing to non-existent or invalid log blocks. Reset the device header in this case. In l2arc_rebuild() change the zfs_dbgmsg messages to spa_history_log_internal() making them user visible with zpool history command. Non-functional changes: Make the first test in persistent L2ARC use `zdb -lll` to increase coverage in `zdb.c`. Rename psize with asize when referring to log blocks, since L2ARC_SET_PSIZE stores the vdev aligned size for log blocks. Also rename dh_log_blk_entries to dh_log_entries to make it clear that it is a mirror of l2ad_log_entries. Added comments for both changes. Fix inaccurate comments for example in l2arc_log_blk_restore(). Add asserts at the end in l2arc_evict() and l2arc_write_buffers(). Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: George Amanakis <[email protected]> Closes #10228
* Add support for boot environment data to be stored in the labelPaul Dagnelie2020-05-0713-22/+335
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Modern bootloaders leverage data stored in the root filesystem to enable some of their powerful features. GRUB specifically has a grubenv file which can store large amounts of configuration data that can be read and written at boot time and during normal operation. This allows sysadmins to configure useful features like automated failover after failed boot attempts. Unfortunately, due to the Copy-on-Write nature of ZFS, the standard behavior of these tools cannot handle writing to ZFS files safely at boot time. We need an alternative way to store data that allows the bootloader to make changes to the data. This work is very similar to work that was done on Illumos to enable similar functionality in the FreeBSD bootloader. This patch is different in that the data being stored is a raw grubenv file; this file can store arbitrary variables and values, and the scripting provided by grub is powerful enough that special structures are not required to implement advanced behavior. We repurpose the second padding area in each label to store the grubenv file, protected by an embedded checksum. We add two ioctls to get and set this data, and libzfs_core and libzfs functions to access them more easily. There are no direct command line interfaces to these functions; these will be added directly to the bootloader utilities. Reviewed-by: Pavel Zakharov <[email protected]> Reviewed-by: Matthew Ahrens <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Paul Dagnelie <[email protected]> Closes #10009
* Fix column width calculation issue with certain terminal widthsPhilip Pokorny2020-05-061-4/+30
| | | | | | | | | | | | If the reported terminal width is 0 or less than 42, the signed variable width was set to a negative number that was then assigned to the unsigned column width becoming a huge number. Add comments and change logic to better explain what's happening. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Philip Pokorny <[email protected]> Closes #10247
* Enable splitting mirrors with indirect vdevsGeorge Amanakis2020-05-066-7/+89
| | | | | | | | | | | When a top-level vdev is removed from a pool it is converted to an indirect vdev. Until now splitting such mirrored pools was not possible with zpool split. This patch enables handling of indirect vdevs and splitting of those pools with zpool split. Reviewed-by: Matthew Ahrens <[email protected]> Reviewed by: Brian Behlendorf <[email protected]> Signed-off-by: George Amanakis <[email protected]> Closes #10283
* taskq: Don't leak system_delay_taskq on FreeBSDRyan Moeller2020-05-051-0/+1
| | | | | | | | | Adds a missing taskq_destroy() call. Reported by: Jorgen Lundman <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Alexander Motin <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10292
* config/kernel-inode-times: initialize timespecalaviss2020-05-041-0/+1
| | | | | | | | | | | | | Usage of this variable uninitialized triggers -Werror,-Wuninitialized when compiled under clang for linux kernel 5.6, leading the build system to believe that the function is not declared. This commit initializes the variable to suppress the warning and fix the build for kernel 5.6 with clang. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Hiếu Lê <[email protected]> Closes #10279 Closes #10281
* Avoid the GEOM topology lock recursion when autoexpanding a poolRyan Moeller2020-05-041-3/+10
| | | | | | | | | | | | | | | | | | The steps to reproduce the problem: mdconfig -a -t swap -s 3g -u 0 gpart create -s GPT md0 gpart add -t freebsd-zfs -s 1g md0 zpool create -o autoexpand=on foo md0p1 gpart resize -i 1 -s 2g md0 Authored by: pjd <[email protected]> FreeBSD-commit: freebsd/freebsd@bccd2db598ede073d6d06781a5fd3b119c08aa81 Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Alexander Motin <[email protected]> Ported-by: Ryan Moeller <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10270
* Update FreeBSD SPL atomicsRyan Moeller2020-05-042-106/+85
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Sync up with the following changes from FreeBSD: ZFS: add emulation of atomic_swap_64 and atomic_load_64 Some 32-bit platforms do not provide 64-bit atomic operations that ZFS requires, either in userland or at all. We emulate those operations for those platforms using a mutex. That is not entirely correct and it's very efficient. Besides, the loads are plain loads, so torn values are possible. Nevertheless, the emulation seems to work for some definition of work. This change adds atomic_swap_64, which is already used in ZFS code, and atomic_load_64 that can be used to prevent torn reads. Authored by: avg <[email protected]> FreeBSD-commit: freebsd/freebsd@3458e5d1e6354123ec2b0953d29f98126aa442e cleanup of illumos compatibility atomics atomic_cas_32 is implemented using atomic_fcmpset_32 on all platforms. Ditto for atomic_cas_64 and atomic_fcmpset_64 on platforms that have it. The only exception is sparc64 that provides MD atomic_cas_32 and atomic_cas_64. This is slightly inefficient as fcmpset reports whether the operation updated the target and that information is not needed for cas. Nevertheless, there is less code to maintain and to add for new platforms. Also, the operations are done inline now as opposed to function calls before. atomic_add_64_nv is implemented using atomic_fetchadd_64 on platforms that provide it. casptr, cas32, atomic_or_8, atomic_or_8_nv are completely removed as they have no users. atomic_mtx that is used to emulate 64-bit atomics on platforms that lack them is defined only on those platforms. As a result, platform specific opensolaris_atomic.S files have lost most of their code. The only exception is i386 where the compat+contrib code provides 64-bit atomics for userland use. That code assumes availability of cmpxchg8b instruction. FreeBSD does not have that assumption for i386 userland and does not provide 64-bit atomics. Hopefully, this can and will be fixed. Authored by: avg <[email protected]> FreeBSD-commit: freebsd/freebsd@e9642c209b4413f6afb41d3b2607c51d80a1a34 emulate illumos membar_producer with atomic_thread_fence_rel membar_producer is supposed to be a store-store barrier. Also, in the code that FreeBSD has ported from illumos membar_producer is used only with regular stores to regular memory (with respect to caching). We do not have an MI primitive for the store-store barrier, so atomic_thread_fence_rel is the closest we have as it provides (load | store) -> store barrier. Previously, membar_producer was an empty function call on all 32-bit arm-s, 32-bit powerpc, riscv and all mips variants. I think that it was inadequate. On other platforms, such as amd64, arm64, i386, powerpc64, sparc64, membar_producer was implemented using stronger primitives than required for a store-store barrier with respect to regular memory access. For example, it used sfence on amd64 and lock-ed nop in i386 (despite TSO). On powerpc64 we now use recommended lwsync instead of eieio. On sparc64 FreeBSD uses TSO mode. On arm64/aarch64 we now use dmb sy instead of dmb ish. Not sure if this is an improvement, actually. After this change we can drop opensolaris_atomic.S for aarch64, amd64, powerpc64 and sparc64 as all required atomic operations have either direct or light-weight mapping to FreeBSD native atomic operations. Discussed with: kib Authored by: avg <[email protected]> FreeBSD-commit: freebsd/freebsd@50cdda62fced8d21e45858e01dc375a10f1749e fix up r353340, don't assume that fcmpset has strong semantics fcmpset can have two kinds of semantics, weak and strong. For practical purposes, strong semantics means that if fcmpset fails then the reported current value is always different from the expected value. Weak semantics means that the reported current value may be the same as the expected value even though fcmpset failed. That's a so called "sporadic" failure. I originally implemented atomic_cas expecting strong semantics, but many platforms actually have weak one. Reported by: pkubaj (not confirmed if same issue) Discussed with: kib, mjg Authored by: avg <[email protected]> FreeBSD-commit: freebsd/freebsd@238787c74e737e271f17330fbad900acc35651c [PowerPC] [MIPS] Implement 32-bit kernel emulation of atomic64 operations This is a lock-based emulation of 64-bit atomics for kernel use, split off from an earlier patch by jhibbits. This is needed to unblock future improvements that reduce the need for locking on 64-bit platforms by using atomic updates. The implementation allows for future integration with userland atomic64, but as that implies going through sysarch for every use, the current status quo of userland doing its own locking may be for the best. Submitted by: jhibbits (original patch), kevans (mips bits) Reviewed by: jhibbits, jeff, kevans Authored by: bdragon <[email protected]> Differential Revision: https://reviews.freebsd.org/D22976 FreeBSD-commit: freebsd/freebsd@db39dab3a896b3d98e588736e9a2b4ddaeb31f1 Remove sparc64 kernel support Remove all sparc64 specific files Remove all sparc64 ifdefs Removee indireeect sparc64 ifdefs Authored by: imp <[email protected]> FreeBSD-commit: freebsd/freebsd@48b94864c51253da92e4444f0074eec36ef391f Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Alexander Motin <[email protected]> Ported-by: Ryan Moeller <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10250
* ZTS: Count CKSUM for all vdevs in verify_poolRyan Moeller2020-04-301-1/+6
| | | | | | | | | | | The verify_pool function should detect checksum errors on any vdev, but it was only checking at the root of the pool. Accumulate the errors for all vdevs to obtain the correct count. Reviewed-by: John Kennedy <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10271
* zdb: Fix ignored zfs_arc_max tuningRyan Moeller2020-04-301-1/+2
| | | | | | | | | | | | | | Running zdb -l $disk shows a warning that zfs_arc_max is being ignored. zdb sets zfs_arc_max below zfs_arc_min, which causes the value to be ignored by arc_tuning_update(). Set zfs_arc_min to the bare minimum in zdb, which is below zfs_arc_max. Reviewed by: Brian Behlendorf <[email protected]> Reviewed-by: Allan Jude <[email protected]> Reviewed-by: Matthew Ahrens <[email protected]> Reviewed-by: George Melikov <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10269
* OpenZFS 6765 - zfs_zaccess_delete() comments do not accuratelyPaul B. Henson2020-04-301-22/+21
| | | | | | | | | | | | | | | | | | reflect delete permissions for ACLs Authored by: Kevin Crowe <[email protected]> Reviewed by: Gordon Ross <[email protected]> Reviewed by: Yuri Pankov <[email protected]> Reviewed by: Brian Behlendorf <[email protected]> Approved by: Richard Lowe <[email protected]> Ported-by: Paul B. Henson <[email protected]> Porting Notes: * Only comments are updated OpenZFS-issue: https://www.illumos.org/issues/6765 OpenZFS-commit: https://github.com/openzfs/openzfs/commit/da412744bc Closes #10266
* OpenZFS 6762 - POSIX write should imply DELETE_CHILD on directoriesPaul B. Henson2020-04-301-80/+147
| | | | | | | | | | | | | | | - and some additional considerations Authored by: Kevin Crowe <[email protected]> Reviewed by: Gordon Ross <[email protected]> Reviewed by: Yuri Pankov <[email protected]> Reviewed by: Brian Behlendorf <[email protected]> Approved by: Richard Lowe <[email protected]> Ported-by: Paul B. Henson <[email protected]> OpenZFS-issue: https://www.illumos.org/issues/6762 OpenZFS-commit: https://github.com/openzfs/openzfs/commit/1eb4e906ec Closes #10266
* OpenZFS 8984 - fix for 6764 breaks ACL inheritancePaul B. Henson2020-04-301-11/+29
| | | | | | | | | | | | | | Authored by: Dominik Hassler <[email protected]> Reviewed by: Sam Zaydel <[email protected]> Reviewed by: Paul B. Henson <[email protected]> Reviewed by: Prakash Surya <[email protected]> Reviewed by: Brian Behlendorf <[email protected]> Approved by: Matthew Ahrens <[email protected]> Ported-by: Paul B. Henson <[email protected]> OpenZFS-issue: https://www.illumos.org/issues/8984 OpenZFS-commit: https://github.com/openzfs/openzfs/commit/e9bacc6d1a Closes #10266
* OpenZFS 6764 - zfs issues with inheritance flags during chmod(2)Paul B. Henson2020-04-301-85/+79
| | | | | | | | | | | | | | | with aclmode=passthrough Authored by: Albert Lee <[email protected]> Reviewed by: Gordon Ross <[email protected]> Reviewed by: Yuri Pankov <[email protected]> Reviewed by: Brian Behlendorf <[email protected]> Approved by: Richard Lowe <[email protected]> Ported-by: Paul B. Henson <[email protected]> OpenZFS-issue: https://www.illumos.org/issues/6764 OpenZFS-commit: https://github.com/openzfs/openzfs/commit/de0f1ddb59 Closes #10266
* OpenZFS 3254 - add support in zfs for aclmode=restrictedPaul B. Henson2020-04-303-8/+9
| | | | | | | | | | | | | Authored-by: Paul B. Henson <[email protected]> Reviewed by: Albert Lee <[email protected]> Reviewed by: Gordon Ross <[email protected]> Reviewed by: Brian Behlendorf <[email protected]> Approved by: Richard Lowe <[email protected]> Ported-by: Paul B. Henson <[email protected]> OpenZFS-issue: https://www.illumos.org/issues/3254 OpenZFS-commit: https://github.com/openzfs/openzfs/commit/71dbfc287c Closes #10266
* OpenZFS 742 - Resurrect the ZFS "aclmode" property OpenZFS 664 - Umask ↵Paul B. Henson2020-04-306-85/+130
| | | | | | | | | | | | | | | | | | | | | | | masking "deny" ACL entries OpenZFS 279 - Bug in the new ACL (post-PSARC/2010/029) semantics Porting notes: * Updated zfs_acl_chmod to take 'boolean_t isdir' as first parameter rather than 'zfsvfs_t *zfsvfs' * zfs man pages changes mixed between zfs and new zfsprops man pages Reviewed by: Aram Hvrneanu <[email protected]> Reviewed by: Gordon Ross <[email protected]> Reviewed by: Robert Gordon <[email protected]> Reviewed by: [email protected] Reviewed by: Brian Behlendorf <[email protected]> Approved by: Garrett D'Amore <[email protected]> Ported-by: Paul B. Henson <[email protected]> OpenZFS-issue: https://www.illumos.org/issues/742 OpenZFS-issue: https://www.illumos.org/issues/664 OpenZFS-issue: https://www.illumos.org/issues/279 OpenZFS-commit: https://github.com/openzfs/openzfs/commit/a3c49ce110 Closes #10266
* Fix regression caused by c14ca14Adam D. Moss2020-04-291-1/+1
| | | | | | | | | The 'zfs load-key' command was broken for 'keyformat=passphrase'. Use the correct output vars when stdin is an interactive terminal. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: adam moss <[email protected]> Closes #10264 Closes #10265
* Add longjmp support for Thumb-2Brian Behlendorf2020-04-291-1/+18
| | | | | | | | | | | When a Thumb-2 kernel is being used, then longjmp must be implemented using the Thumb-2 instruction set in module/lua/setjmp/setjmp_arm.S. Original-patch-by: @jsrlabs Reviewed-by: @awehrfritz Signed-off-by: Brian Behlendorf <[email protected]> Closes #7408 Closes #9957 Closes #9967
* Support custom URI schemes for the keylocation propertyJason King2020-04-283-194/+360
| | | | | | | | | | | | | | | | | | | | | Every platform has their own preferred methods for implementing URI schemes beyond the currently supported file scheme (e.g. 'https' on FreeBSD would likely use libfetch, while Linux distros and illumos would probably use libcurl, etc). It would be helpful if libzfs can be extended to support additional schemes in a simple manner. A table of (scheme, handler_function) pairs is added to libzfs_crypto.c, and the existing functions in libzfs_crypto.c so that when the key format is ZFS_KEYFORMAT_URI, the scheme from the URI string is extracted, and a matching handler it located in the aforementioned table (returning an error if no matching handler is found). The handler function is then invoked to retrieve the key material (in the format specified by the keyformat property) and the key is loaded or the handler can return an error to abort the key loading process. Reviewed by: Sean Eric Fagan <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Jason King <[email protected]> Closes #10218
* Add more sanity testing for zdb input argsSara Hartse2020-04-284-12/+127
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: John Kennedy <[email protected]> Signed-off-by: sara hartse <[email protected]> Closes #10243
* Add missing zfs_refcount_destroy() in key_mapping_rele()George Amanakis2020-04-281-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Otherwise when running with reference_tracking_enable=TRUE mounting and unmounting an encrypted dataset panics with: Call Trace: dump_stack+0x66/0x90 slab_err+0xcd/0xf2 ? __kmalloc+0x174/0x260 ? __kmem_cache_shutdown+0x158/0x240 __kmem_cache_shutdown.cold+0x1d/0x115 shutdown_cache+0x11/0x140 kmem_cache_destroy+0x210/0x230 spl_kmem_cache_destroy+0x122/0x3e0 [spl] zfs_refcount_fini+0x11/0x20 [zfs] spa_fini+0x4b/0x120 [zfs] zfs_kmod_fini+0x6b/0xa0 [zfs] _fini+0xa/0x68c [zfs] __x64_sys_delete_module+0x19c/0x2b0 do_syscall_64+0x5b/0x1a0 entry_SYSCALL_64_after_hwframe+0x44/0xa9 Reviewed-By: Brian Behlendorf <[email protected]> Reviewed-By: Tom Caputi <[email protected]> Signed-off-by: George Amanakis <[email protected]> Closes #10246
* Fix zlib leak on FreeBSDRyan Moeller2020-04-281-1/+1
| | | | | | | | | | | | | zlib_inflateEnd was accidentally a wrapper for inflateInit instead of inflateEnd, and hilarity ensues. Fix the typo so we free memory instead of allocating more. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: George Melikov <[email protected]> Reviewed-by: Matthew Ahrens <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10225 Closes #10252
* zfs_create: round up volume size to multiple of bsalex2020-04-244-3/+45
| | | | | | | | | | | Round up the volume size requested in `zfs create -V size` to the next higher multiple of the volblocksize. Updates the man page and adds a test to verify the new behavior. Reviewed-by: Brian Behlendorf <[email protected]> Reported-by: puffi <[email protected]> Signed-off-by: Alex John <[email protected]> Closes #8541 Closes #10196
* Fix missing ivset guid with resumed raw base recvTom Caputi2020-04-241-1/+1
| | | | | | | | | | | | | | | | | | | | | | This patch corrects a bug introduced in 61152d1069. When resuming a raw base receive, the dmu_recv code always sets drc->drc_fromsnapobj to the object ID of the previous snapshot. For incrementals, this is correct, but for base sends, this should be left at 0. The presence of this ID eventually allows a check to run which determines whether or not the incoming stream and the previous snapshot have matching IVset guids. This check fails becuase it is not meant to run when there is no previous snapshot. When it does fail, the user receives an error stating that the incoming stream has the problem outlined in errata 4. This patch corrects this issue by simply ensuring drc->drc_fromsnapobj is left as 0 for base receives. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: George Melikov <[email protected]> Signed-off-by: Tom Caputi <[email protected]> Closes #10234 Closes #10239
* Fix unitialized variable in `zstream redup` commandBrian Behlendorf2020-04-231-1/+1
| | | | | | | | | | | | | Fix uninitialized variable in `zstream redup` command. The compiler may determine the 'stream_offset' variable can be uninitialized because not all rdt_lookup() exit paths set it. This should never happen in practice as documented by the assert, but initialize it regardless to resolve the warning. Reviewed-by: Matthew Ahrens <[email protected]> Reviewed-by: George Melikov <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #10241 Closes #10244
* change libspl list member names to match kernelMatthew Ahrens2020-04-232-41/+40
| | | | | | | | | | This aids in debugging, so that we can use the same infrastructure to walk zfs's list_t in the kernel module and in the userland libraries (e.g. when debugging ztest). Reviewed-by: Serapheim Dimitropoulos <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Matthew Ahrens <[email protected]> Closes #10236
* Remove deduplicated send/receive codeMatthew Ahrens2020-04-2326-1081/+219
| | | | | | | | | | | | | | | | | | | | | | | | | Deduplicated send streams (i.e. `zfs send -D` and `zfs receive` of such streams) are deprecated. Deduplicated send streams can be received by first converting them to non-deduplicated with the `zstream redup` command. This commit removes the code for sending and receiving deduplicated send streams. `zfs send -D` will now print a warning, ignore the `-D` flag, and generate a regular (non-deduplicated) send stream. `zfs receive` of a deduplicated send stream will print an error message and fail. The resulting code simplification (especially in the kernel's support for receiving dedup streams) should help enable future performance enhancements. Several new tests are added which leverage `zstream redup`. Reviewed-by: Paul Dagnelie <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Matthew Ahrens <[email protected]> Issue #7887 Issue #10117 Issue #10156 Closes #10212
* Fix more leaks detected by ASANJoao Carlos Mendes Luis2020-04-221-2/+7
| | | | | | | This commit fixes a bunch of missing free() calls in a10d50f99 Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: João Carlos Mendes Luís <[email protected]> Closes #10219
* Use a struct to organize metaslab-group-allocator fieldsMatthew Ahrens2020-04-223-57/+62
| | | | | | | | | | | | | | | | | | | | Each metaslab group (of which there is one per top-level vdev) has several (4, by default) "metaslab group allocators". Each "allocator" has its own metaslab that it prefers to allocate from (the "primary" allocator), and each can perform allocations concurrently with the other allocators. In addition to the primary metaslab, there are several other fields that need to be tracked separately for each allocator. These are currently stored as several arrays in the metaslab_group_t, each array indexed by allocator number. This change organizes all the metaslab-group-allocator-specific fields into a new struct, metaslab_group_allocator_t. The metaslab_group_t now needs only one array indexed by the allocator number - which contains the metaslab_group_allocator_t's. Reviewed-by: Paul Dagnelie <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Matthew Ahrens <[email protected]> Closes #10213
* Don't attempt trimming "hole" vdevsNiklas Haas2020-04-211-1/+2
| | | | | | | | | | | | | On zpools containing hole vdevs (e.g. removed log devices), the `zpool trim` (and presumably `zpool initialize`) commands will attempt calling their respective functions on "hole", which fails, as this is not a real vdev. Avoid this by removing HOLE vdevs in zpool_collect_leaves. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: George Melikov <[email protected]> Signed-off-by: Niklas Haas <[email protected]> Closes #10227
* Fix zfs send progress reportingMatthew Ahrens2020-04-203-1/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | The progress of a send is supposed to be reported by `zfs send -v`, but it is not. This works by creating a new user thread (with pthread_create()) which does ZFS_IOC_SEND_PROGRESS ioctls to check how much progress has been made. This IOCTL finds the specified send (since there may be multiple concurrent sends in the system). The IOCTL also checks that the specified send was started by the current process. On Linux, different threads of the same process are represented as different `struct task_struct`s (and, confusingly, have different PID's). To check if if two threads are in the same process, we need to check if they have the same `struct task_struct:group_leader`. We used to to this correctly, but it was inadvertently changed by 30af21b02569 (Redacted Send) to simply check if the current `struct task_struct` is the one that started the send. This commit changes the code back to checking if the send was started by a `struct task_struct` with the same `group_leader` as the calling thread. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Chris Wedgwood <[email protected]> Reviewed-by: Paul Dagnelie <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Matthew Ahrens <[email protected]> Closes #10215 Closes #10216
* Use new FreeBSD API to largely eliminate object lockingMatthew Macy2020-04-173-39/+56
| | | | | | | | Propagate changes in HEAD that mostly eliminate object locking. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Alexander Motin <[email protected]> Signed-off-by: Matt Macy <[email protected]> Closes #10205
* Persistent L2ARC minor fixesGeorge Amanakis2020-04-173-12/+9
| | | | | | | | | | | Minor fixes on persistent L2ARC improving code readability and fixing a typo in zdb.c when byte-swapping a log block. It also improves the pesist_l2arc_007_pos.ksh test by giving it more time to retrieve log blocks on the cache device. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Adam D. Moss <[email protected]> Signed-off-by: George Amanakis <[email protected]> Closes #10210
* Update FreeBSD tunablesRyan Moeller2020-04-156-43/+71
| | | | | | | | Remove some obsolete legacy compat, rename some misnamed, and add some missing tunables for FreeBSD. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10203
* Don't delete freebsd.run in distcleanRyan Moeller2020-04-151-0/+1
| | | | | | | | | | | Add a comment so the file is not empty. The comment can be removed when FreeBSD-specific tests are added. Reviewed-by: George Melikov <[email protected]> Reviewed-by: Sean Eric Fagan <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10206
* Fix SC2086 note in zpool.d/smartRyan Moeller2020-04-141-1/+1
| | | | | | | | | | ./cmd/zpool/zpool.d/smart:78:32: note: Double quote to prevent globbing and word splitting. [SC2086] Reported by latest shellcheck on FreeBSD. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #10194
* sys/mnttab.h: include sys/stat.h for stat64alaviss2020-04-141-1/+1
| | | | | | | | | | | | Musl libc defined `stat64` as a macro, which causes the build to fail upon compiling os/linux/getmntany.c due to conflicts between the forward declaration and the implementation. This commit fixes that by including <sys/stat.h> in "sys/mnttab.h" directly. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Hiếu Lê <[email protected]> Closes #10195
* Add FreeBSD support to OpenZFSMatthew Macy2020-04-14209-81/+46201
| | | | | | | | | | | | | | | | | | Add the FreeBSD platform code to the OpenZFS repository. As of this commit the source can be compiled and tested on FreeBSD 11 and 12. Subsequent commits are now required to compile on FreeBSD and Linux. Additionally, they must pass the ZFS Test Suite on FreeBSD which is being run by the CI. As of this commit 1230 tests pass on FreeBSD and there are no unexpected failures. Reviewed-by: Sean Eric Fagan <[email protected]> Reviewed-by: Jorgen Lundman <[email protected]> Reviewed-by: Richard Laager <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Co-authored-by: Ryan Moeller <[email protected]> Signed-off-by: Matt Macy <[email protected]> Signed-off-by: Ryan Moeller <[email protected]> Closes #898 Closes #8987
* Fix allocation errors, detected using ASANJoao Carlos Mendes Luis2020-04-131-2/+3
| | | | | | | | | | | | The test for VDEV_TYPE_INDIRECT is done after a memory allocation, and could return from function without freeing it. Since we don't need that allocation yet, just postpone it. Add a missing free() when buffer is no longer needed. Reviewed-by: Matthew Ahrens <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: João Carlos Mendes Luís <[email protected]> Closes #10193
* Disable user space reference trackingBrian Behlendorf2020-04-131-5/+6
| | | | | | | | | | | | | | The memory and cpu cost of reference count tracking with the current implementation is significant. For this reason it has always been disabled by default for the kmods. Apply this same default to user space so ztest doesn't always incur this performance penalty. Our intention is to re-enable this by default for ztest once the code has been optimized. Since we expect to at some point provide a FUSE implementation we wouldn't want this enabled by default for libzpool. Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #10189
* ZTS: Fix and change testcase cache_010_negalex2020-04-135-30/+29
| | | | | | | | | | | | | | | | Commit 379ca9c removed the requirement on aux devices to be block devices only but the test case cache_010_neg was not updated, making it fail consistently. This change changes the test to check that cache devices _can_ be anything that presents a block interface. The testcase is renamed to cache_010_pos and the exceptions for known failure removed from the test runner. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Reported-by: Richard Elling <[email protected]> Signed-off-by: Alex John <[email protected]> Closes #10172
* zvol_write() can use dmu_tx_hold_write_by_dnode()Matthew Ahrens2020-04-101-1/+1
| | | | | | | | | | | | | We can improve the performance of writes to zvols by using dmu_tx_hold_write_by_dnode() instead of dmu_tx_hold_write(). This reduces lock contention on the first block of the dnode object, and also reduces the amount of CPU needed. The benefit will be highest with multi-threaded async writes (i.e. writes that don't call zil_commit()). Reviewed-by: Jorgen Lundman <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Tony Nguyen <[email protected]> Signed-off-by: Matthew Ahrens <[email protected]> Closes #10184
* Minor `zstream redup` command fixesBrian Behlendorf2020-04-102-1/+1
| | | | | | | | | | | | | | | | * Fix uninitialized variable in `zstream redup` command. The 'rdt.ddt_count' variable is uninitialized because it was allocated from the stack and not globally. Initialize it. This was reported by gcc when compiling with debugging enabled. zstream_redup.c:157:16: error: 'rdt.ddt_count' may be used uninitialized in this function [-Werror=maybe-uninitialized] * Remove the cmd/zstreamdump/.gitignore file. It's no longer needed now that the zstreamdump command is a script. Reviewed-by: Matthew Ahrens <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #10192
* Add `zstream redup` command to convert deduplicated send streamsMatthew Ahrens2020-04-1016-36/+728
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Deduplicated send and receive is deprecated. To ease migration to the new dedup-send-less world, the commit adds a `zstream redup` utility to convert deduplicated send streams to normal streams, so that they can continue to be received indefinitely. The new `zstream` command also replaces the functionality of `zstreamdump`, by way of the `zstream dump` subcommand. The `zstreamdump` command is replaced by a shell script which invokes `zstream dump`. The way that `zstream redup` works under the hood is that as we read the send stream, we build up a hash table which maps from `<GUID, object, offset> -> <file_offset>`. Whenever we see a WRITE record, we add a new entry to the hash table, which indicates where in the stream file to find the WRITE record for this block. (The key is `drr_toguid, drr_object, drr_offset`.) For entries other than WRITE_BYREF, we pass them through unchanged (except for the running checksum, which is recalculated). For WRITE_BYREF records, we change them to WRITE records. We find the referenced WRITE record by looking in the hash table (for the record with key `drr_refguid, drr_refobject, drr_refoffset`), and then reading the record header and payload from the specified offset in the stream file. This is why the stream can not be a pipe. The found WRITE record replaces the WRITE_BYREF record, with its `drr_toguid`, `drr_object`, and `drr_offset` fields changed to be the same as the WRITE_BYREF's (i.e. we are writing the same logical block, but with the data supplied by the previous WRITE record). This algorithm requires memory proportional to the number of WRITE records (same as `zfs send -D`), but the size per WRITE record is relatively low (40 bytes, vs. 72 for `zfs send -D`). A 1TB send stream with 8KB blocks (`recordsize=8k`) would use around 5GB of RAM to "redup". Reviewed-by: Jorgen Lundman <[email protected]> Reviewed-by: Paul Dagnelie <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Matthew Ahrens <[email protected]> Closes #10124 Closes #10156
* Persistent L2ARCGeorge Amanakis2020-04-1030-88/+3020
| | | | | | | | | | | | | | | | | | | | | This commit makes the L2ARC persistent across reboots. We implement a light-weight persistent L2ARC metadata structure that allows L2ARC contents to be recovered after a reboot. This significantly eases the impact a reboot has on read performance on systems with large caches. Reviewed-by: Matthew Ahrens <[email protected]> Reviewed-by: George Wilson <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Co-authored-by: Saso Kiselkov <[email protected]> Co-authored-by: Jorgen Lundman <[email protected]> Co-authored-by: George Amanakis <[email protected]> Ported-by: Yuxuan Shui <[email protected]> Signed-off-by: George Amanakis <[email protected]> Closes #925 Closes #1823 Closes #2672 Closes #3744 Closes #9582
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-20 10:19:08 +0000