2010-03-11 Brian Behlendorf * : Tag zfs-0.4.8 - Use 'git log --no-merges' for full change log. * : Build system improvements: - Remove Module.markers and Module.symver{s} in clean target. - Improved kernel source detection when none specified. - Fix RPM definitions for the unknown distro/installation. - Check for spl in ../spl if not found in install path. - Include all headers regardless of depth in packages. - Allow recursive configure/make. * lib/libzpool/kernel.c: Fixed several zdb bugs when opening a pool - zdb wasn't getting the correct device size when the vdev is a block device. In Solaris, fstat64() returns the device size but in Linux an ioctl() is needed. - make sure that we don't try to open a block device in write mode from userspace. This shouldn't happen, because zdb opens devices in read-only mode, and ztest only uses files. * lib/libspl/include/umem.h: Add umem_alloc_aligned() and honor cache_align field for umem cache. Under linux we open block devices with O_DIRECT which means we must provide aligned memory buffers. * lib/libzpool/kernel.c: Fix some incorrect error handling. In vn_open(), if fstat64() returned an error, the real errno was being obscured by calling close(). * scripts/*: Fix scripts to work when invoked from other directories. * module/zfs/arc.c: Fix struct ht_lock padding in arc.c. * lib/libefi/include/sys/uuid.h: Fix duplicate uuid_t typedef. * module/zfs/txg.c: Use CPU percentages for number of commit cb threads. This doesn't change number of threads in the kernel, but it reduces number of threads in ztest (important due to 32-bit address limitations). * cmd/ztest/ztest.c: Clean up emulation of kernel threads in userspace. Updated to use pthread thread specific data rather than keeping a global list. This also fixes at least one easily reproducible crash in ztest * META, config/kernel.m4: Add configure check for kernel build options which are incompatible with the license. If your building against a kernel deemed incompatible configure will fail and suggest how you should rebuild your kernel. * config/kernel-fmode-t.m4: Linux 2.6.28 compat, add a check for the fmode_t type. This typedef first appears in 2.6.28 kernels as part of some block device operation reworking. * module/zfs/dmu_send.c: No inline to keep dmu_recv_stream() stack frame less than 1024 bytes. Recent builds against 2.6.31 flagged dmu_recv_stream() as stack heavy. Further analysis of this function should be performed to further reduce its stack usage. * scripts/common.sh: Split the udev rule from a specific configuration by providing a generic 60-zpool.rules file which uses a small helper util 'zpool_id' to parse a configuration file by default located in /etc/zfs/zdev.conf. The helper script maps a by-path udev name to a more friendly name of for large configurations. Additionally, when running zpool-create.sh in-tree it will no longer use udev because we would have to copy certain helper scripts in to the installed system. To avoid this the config file in simply parsed and symlinks are created in your working tree. The script will use udev if it as run as part of an installed zfs-test package. * module/zfs/zvol.c: Use check_disk_change() instead of revalidate_disk(). For 2.6.27 kernels are earlier revalidate_disk() was not available. However, check_disk_change() has been available for far longer and will properly inform the kernel of the volume change for both older and newer kernels. * module/zfs/dmu.c: Fixed incorrect ASSERT3S() added by ZVOL. * module/zfs/vdev_raidz.c, module/zfs/zvol.c: Minor fixes for 32-bit. * scripts/zfs-update.sh, man/man8/*: Added man pages based on the latest documentation and modified zfs-update.sh script to update them. * .gitignore: Updated .gitignore rules to exclude build products. 2009-11-24 Brian Behlendorf * : Tag zfs-0.4.7 - Use 'git log --no-merges' for full change log. * module/zcommon/include/sys/fs/zfs.h, module/zfs/include/sys/blkdev.h, module/zfs/include/sys/dmu.h, module/zfs/dmu.c, module/zfs/include/sys/zvol.h, module/zfs/zvol.c, module/zfs/zfs_ioctl.c: Added the ZVOL block device, with the addition of the ZVOL real ZFS based block devices are available and can be compared head to head with Linux's MD and LVM block drivers. The Linux ZVOL has not yet had any performance work done but from a user perspective it should be functionally complete and behave like any other Linux block device. The ZVOL has so far been tested using zconfig.sh on the following x86_64 based platforms: FC11, CHAOS4, RHEL5, RHEL6, and SLES11. However, more testing is required to ensure everything is working as designed. * scripts/udev-rules/99-zpool.rules.promise, scripts/zpool-config/promise-raid0-1x16.sh, scripts/zpool-config/promise-raid10-8x2.sh, scripts/zpool-config/promise-raidz-2x8.sh, scripts/zpool-config/promise-raidz2-2x8.sh: Additional test configurations for a small 16 drive JBOD. * module/zfs/arc.c: Linux 2.6.31 compat, mutexes can now exceed 64 bytes depending on the the kernel build options. To account for this increase the pad size to 256 bytes. * module/zfs/vdev_disk.c: Linux 2.6.31 compat, to get the hard sector size use bdev_logical_block_size() this function replaces bdev_hardsect_size(). * module/zfs/zfs_rlock.c: Prevent gcc uninit compiler warning in zfs_range_unlock_reader(). * Makefile.am: Ensure *.order and *.markers build products are removed by distclean rule. 2009-11-02 Brian Behlendorf * : Tag zfs-0.4.6 - Use 'git log --no-merges' for full change log. * Rebased to ZFS b121 from OpenSolaris. * module/zfs/vdev_disk.c: Finally a feature complete implementation: - Handle dynamic bio merge_bdev limitations when constructing the bio set associated with a dio. This previously prevented us from layering cleanly on the md and dm virtual devices. - Removed hard coded 512 byte sector size. - Correctly determine the device size when using a partition. - Hold and extra dio reference when submitting bio's using bio_submit() to prevent a completion race. * lib/libefi/*: Added fully function libefi library from Solaris. This allows us to properly create and access GPT style partition tables which are used when a whole device is added to a zpool. * cmd/zpool/zpool_vdev.c: Fully integrated zpool with Linux package libblkid. This allows zpool to identify existing devices of various types to prevent devices from accidentally being used. When given a whole device with a GPT partition table all partitions will be checked for existing filesystems. At the moment MBR style partition tables cannot be check and the force option must be used. * cmd/zpool/zpool_vdev.c: Solaris devid support has been removed in favor of Linux's udev. This means that a zpool device will always be opened using the path provided at configuration time. This may initially seem limiting but it has certain advantages: - When creating a zpool where the physical location of the device is NOT import simply create the pool using the /dev/disk/by-id paths. This will ensure that regardless of physical location the device will be properly located. - When creating a zpool where the physical location of the device is important use the /dev/disk/by-path paths. This will ensure that devices are never accidentally detected and used in an incorrect location which would compromise the redundancy of the system. - Ever better you can create use your own udev rules file to setup any mapping and naming convention you desire. One example of a custom rule is to map physical device locations using grid with numbers are letter for coordinates. Each letter might represent a specific bus/channel and each number a specific device. For large configurations this provides an easy way to identify devices. * module/zpios/zpios.c: Update to use kobject_set_name() for increased portability. * modules/*/*: Update module init/exit access points to use spl_module_{init,exit}() macro API. This ensures the cwd is immediately set to '/' and may be leveraged latter for any additional module setup/cleanup which is required. * cmd/ztest/ztest.c: Check ftrucate() return code to prevent warnings when --fortify-source options is used in rpm builds. * config/Rules.am: Set DEBUG/NDEBUG globally when building user space components. * scripts/zconfig.sh: Initial hook for running additional sanity tests are part of 'make check'. Currently, there are only two tests which do some basic configuration checking but they should be extended as much as possible to prevent regressions. Tests should also all be written so they run entirely in-tree. * scripts/zpios-sanity.sh: Initial hook for validating real IO using all block devices and all raid configurations. Supported device types include scsi, ide, md, dm, ram, loop, and file. Supported raid types include raid0, raid10, raidz, and raidz2. * scripts/zpool-config/*: Update dragon and x4550 configs to use custom udev rules file with <1-N> naming convention. Add configs for md, dm, and ram block devices to verify functionality. * zfs-test.spec.in: Added zfs-test package which extends the existing in-tree test infrastructure such that it can be run as part of an installed package. This simplifies the testing of tagged releases. * zfs-modules.spec.in: Various spec file tweaks for the supported distros: RHEL5, RHEL6, SLES10, SLES11, Chaos4, Fedora 11. 2009-08-04 Brian Behlendorf * : Tag zfs-0.4.5 - Use 'git log --no-merges' for full change log. * FC11 and SLES11 support: This includes all compatibility changes to support 2.6.29 based kernels and the required build system improvements. * lib/libspl/asm-generic/atomic.c: Generic user space atomic support. * module/zpios/zpios.c: Register a basic compat ioctl handler for 32-bit user vs 64-bit kernel compatibility. This is the default build environment for all 64-bit SLES systems. Additionally replace the use of 'struct timespec' which uses longs internally and is therefore different sizes on 32-bit vs 64-bit objects with 'struct zpios_timespec_t'.a * config/kernel.m4: Check arch/default path when detecting kernel objects under SLES. We still preferentially use arch/arch if available but if that fails it is acceptable to use default. * config/kernel.m4: Remove LINUXINCLUDE from autoconf wrapper. This breaks the 2.6.28+ kernels build system, all kernel build systems at least post 2.6.16 will set this properly so we should not. * lib/libspl/include/assert.h: Add ASSERTV macro to simplify removing variables (the V in ASSERTV) which are only used in ASSERT(). Also revert all previously modified ASSERT()s to their original definitions and use the ASSERTV macro to handle unused variables. * module/zpios/zpios.c: Use spl device interfaces for portability. * scripts/common.sh: The losetup -f option is unavailable for SLES10 use the unused_loop_device() common function. Additionally LOSETUP is now used to portably reference the losetup binary. * module/zfs/vdev_disk.c: Unused destroy_dirty_buffers arg removed. * module/zfs/vdev_disk.c: BIO_RW_SYNC renamed to BIO_RW_SYNCIO. * module/zfs/vdev_disk.c: open/close_bdev_excl() renamed to open/close_bdev_exclusive(). * module/zfs/vdev_disk.c: Empty write barriers are supported as of linux 2.6.24 and are now used to implement DKIOCFLUSHWRITECACHE. * module/zfs/vdev_disk.c: The bi_end_io API changes make partial IO's impossible handle this case cleanly. * module/zfs/vdev_disk.c: BIO_RW_FAILFAST replaced with BIO_RW_FAILFAST_{DEV|_TRANSPORT|_DRIVER}, use the legacy BIO_RW_FAILFAST flag if it exists for now. * module/zfs/dmu_objset.c, module/zfs/spa_history.c: Excessively large stack frames (>2048) were detected in dmu_objset_snapshot() and spa_history_log() and were reduced by allocating from the heap. * Rebased to ZFS b117 from OpenSolaris: * module/zfs/dmu.c: Update zerocopy patch to be consistent with new flags arg passed to dmu_read(). * module/zpios/zpios.c: Update zpios to be aware of the new flags argument available in dmu_read(). Additionally add a zpios command line flag to set the DMU_READ_NO_PREFETCH flag for a test. * module/zfs/include/sys/zfs_context.h: Wrap new sysevent includes until sysevent or something like it is implemented. * module/zfs/zfs_ioctl.c: Wrap all the ACL interfaces with HAVE_ZPL. They are integrated with the vfs layer which is not yet supported. * module/zcommon/zfs_prop.c: Export new quota related symbols. * lib/libzfs/libzfs_util.c: Increase buffer size for nvlist which is needed for large configurations. * Bug fixes: * lib/libspl/include/sys/zfs_debug.h: Removed duplicate file. * lib/libspl/include/sys/isa_defs.h: Fixed missing macro definitions for little endian and big endian. * cmd/zpios/zpios_main.c: Pretty-up the zpios 'make check' output. * module/zfs/vdev_disk.c: Honor spa_mode() when opening block devs. * module/zfs/vdev_disk.c: Initial error handling added to vdev_disk_io_done() to revalidate the media on EIO. * lib/libzfs/libzfs_util.c: Fix an accidentally introduced formatting issue in zfs_nicenum() output. 2009-07-02 Brian Behlendorf * : Tag zfs-0.4.4 - Use 'git log --no-merges' for full change log. * : Generic distro friendly build system / packaging improvements for RPM based distros including CHAOS, RHEL, Fedora, and SLES. These changes bring the zfs-0.4.4 tree in to compliance with the spl-0.4.4 packaging changes. The bottom line is 2 source rpms and 4 binary rpms will now be generated when creating packages there will be: zfs-.src.rpm - Fully rebuildable source rpm for libzfs and utils. zfs-modules-.src.rpm - Fully rebuildable source rpm for kernel modules. zfs-..rpm - Binary rpm for libzfs and utils. The utils in this package are compatible with all zfs-module rpms of the same version. zfs-devel-..rpm - Binary rpm containing headers for building against libzfs libraries. zfs-modules--.arch.rpm - Binary rpm containing the kernel modules for a specific kernel build. The package name contains the kernel version and you should have one of these packages installed to match every kernel on your system. zfs-modules-devel--.arch.rpm - Binary rpm containing development header and module symbols needed for building additional kernel modules which are dependent on the zfs module stack. * : SLES9/10 distro support added: Several significant updates to the build system were required to support this including: - Autoconf macros updated to be aware of the standard install locations for SLES kernel sources and build objects. Additionally changes were made to support multiple names for Module{s}.symvers. - By default on SLES all user space builds are 32-bit even on 64-bit arches. This means we need to be careful to pass -m64 in the autoconf checks which probe the kernel. Additionally, ioctl_compat handles still need to be added so 32-bit user binarys can perform ioctls with the 64-bit kernel. * : Powerpc64 support added: - 64-bit user space atomic support for power64 was obtained from an old version of OpenSolaris which offered minimal powerpc support. The atomic support is not 100% fully implemented but it's a good first step towards cleanly supporting the architecture. - Added powerpc ISA type. - Explicitly use signed char for portability. On x86/x86_64 systems the default char type is signed, on ppc/ppc64 systems the default char type is unsigned. - Core target arch support for conditional compilation of SUBDIRs. Required by libspl for its arch specific atomic implementations. * COPYRIGHT: Readded accidentally dropped COPYRIGHT, it just references the OPENSOLARIS.LICENSE which was still in the project. * module/zfs/dmu_tx.c: Add EXPORT_SYMBOL(dmu_tx_callback_register). * config/user-zlib.m4: Add basic zlib autoconf check for user space. 2009-03-20 Brian Behlendorf * : Tag zfs-0.4.3 - Use 'git log --no-merges' for full change log. * : Rebased to ZFS b108 from OpenSolaris * configure.ac, *Makefile.am: Build system update. This includes resolving various build issues and adding support for the remaining common build targets. Available targets now include: - make all # Build everything - make install # Install everything - make clean # Clean up build products - make distclean # Clean up everything - make dist # Create package tarball - make srpm # Create package source RPM - make rpm # Create package binary RPMs - make tags # Create ctags and etags for everything Extra care was taken to ensure that the source RPMs are fully rebuildable against Fedora/RHEL/Chaos kernels. To build binary RPMs from the source RPM for your system simply run: rpmbuild --rebuild zfs-x.y.z-1.src.rpm This will produce two binary RPMs with correct 'requires' dependencies for your kernel. One will contain all zfs modules and support utilities, the other is a devel package for compiling additional kernel modules which are dependent on the zfs. zfs-x.y.z-1_.x86_64.rpm zfs-devel-x.y.2-1_.x86_64.rpm * config/kernel.m4, config/Rules.am: Removed the build options -Wno-unused -Wno-missing -Wno-unused -Wno-parentheses and -Wno-unknown-pragmas. All upstream warnings which were being suppressed by these options are now handled in the matching gcc-* topic branch. * zpios: Various zpios and test infrastructure improvements. * feature-commit-cb: New feature commit callbacks. * : Various bug fixes which are more clearly detailed in the git commit logs. For a detailed summary of changes post version 0.4.2, check out the 'top' topic branch and view the commit logs. git checkout top git log --no-merges -53 2009-02-05 Brian Behlendorf * : Tag zfs-0.4.2 * lib/libzfs/libzfs_util.c: Remove mknod() this is now handled by by the SPL with an upcall even without GPL-only symbols. * module/zfs/vdev_diskc: Updated linux vdev_disk interfaces to issue multiple bios if needed due to the maximum request size being smaller than the request size passed down from the spa. * scripts/zpool-config/lo-*: Add loopback based test configs. 2009-01-21 Brian Behlendorf * : Tag zfs-0.4.1 * : Rebased to ZFS b105 from OpenSolaris * : Restructured project and build system to more closely mirror the code organization used in OpenSolaris. This allows us minimize the changes to upstream source needed, and allows concurrent user+kernel space builds of all components. 2008-11-19 Brian Behlendorf * : Tag zfs-0.4.0 * : ZFS project migrated from Subversion which leveraged a quilt based patch stack to Git and a TopGit managed patch stack. The new method treats all patches as Git branches which can be more easily shared for distributed development. Consult the top level GIT file for detailed information on how to properly develop for this package using Git+TopGit. 2008-11-12 Brian Behlendorf * : Tag zfs-0.3.4 * zfs-07-create-dev-zfs.patch: Ricardo M. Correia - Make libzfs create /dev/zfs if it doesn't exist. * zfs-05-check-zvol-size.patch: Ricardo M. Correia - Properly check zvol size under Linux. * zfs-04-no-openat-fdopendir.patch: Ricardo M. Correia - Do not use openat() and fdopendir() since they are not available on older systems. * zfs-03-fix-bio-sync.patch: Ricardo M. Correia - Fix memory corruption in RHEL4 due to synchronous IO becoming asynchronous. 2008-11-06 Brian Behlendorf * zfs-02-zpios-fix-stuck-thread-memleak.patch: Ricardo M. Correia - Fix stuck threads and memory leaks when errors occur while writing. * zfs-01-zpios-arg-corruption.patch: Ricardo M. Correia - Fix zpios cmd line argument corruption problem. * zfs-00-minor-fixes.patch: Ricardo M. Correia - Minor build system improvements - Minor script improvements - Create a full copy and not a link tree with quilt - ZPIOS_MAJOR changed from 231 to 232 - BIO_RW_BARRIER flag removed from IO request 2008-06-30 Brian Behlendorf * : Tag zfs-0.3.3 * : Minor script updates and tweaks to be compatible with the latest version of the SPL. 2008-06-13 Brian Behlendorf * vdev_disk.diff: Replace vdev_disk implementation which was based on the kmalloc'ed logical address space with a version which works with vmalloc'ed memory in the virtual address space. This was done to support the new SPL slab implementation which is based on virtual addresses to avoid the need for contigeously allocated memory. 2008-06-05 Brian Behlendorf * arc-vm-integration.diff: Reduce maximum default arc memory usage to 1/4 of total system memory. Because all the bulk data is still allocated on the slab memory fragmentation is a serious concern. To address this in the short term we simply need to leave lots of free memory. * fix-stack.diff: First step towards reducing stack usage so we can run the full ZFS stack using a stock kernel. 2008-06-04 Brian Behlendorf * : Tag zfs-0.3.2 * : Extensive improvements to the build system to detect kernel API changes so we can flexibly build with a wider range of kernel versions. The code has now been testing with the 2.6.18-32chaos and 2.6.25.3-18.fc9 kernels, however we should also be compatible with other kernels in the range of 2.6.18-2.6.25. The only remaining issue preventing us from running with a stock kernel is ZFS stack usage. 2008-05-21 Brian Behlendorf * : Tag zfs-0.3.1 * : License headers including URCL added for release. 2008-05-21 Brian Behlendorf * : Tag zfs-0.3.0 * configure.ac: Improved autotools support and configurable debug. 2008-05-15 Brian Behlendorf * : Updating original ZFS sources to build 89 which includes the new write throttling changes plus support for using ZFS as your root device. Neither of which will work exactly right without some more work but this gets us much closers to the latest source. 2008-02-28 Brian Behlendorf * : First attempt based on SPL module and zfs-lustre sources