From 93e28d661e1d704a9cada86ef2bc4763a6ef3be7 Mon Sep 17 00:00:00 2001 From: Serapheim Dimitropoulos Date: Tue, 16 Jul 2019 10:11:49 -0700 Subject: Log Spacemap Project = Motivation At Delphix we've seen a lot of customer systems where fragmentation is over 75% and random writes take a performance hit because a lot of time is spend on I/Os that update on-disk space accounting metadata. Specifically, we seen cases where 20% to 40% of sync time is spend after sync pass 1 and ~30% of the I/Os on the system is spent updating spacemaps. The problem is that these pools have existed long enough that we've touched almost every metaslab at least once, and random writes scatter frees across all metaslabs every TXG, thus appending to their spacemaps and resulting in many I/Os. To give an example, assuming that every VDEV has 200 metaslabs and our writes fit within a single spacemap block (generally 4K) we have 200 I/Os. Then if we assume 2 levels of indirection, we need 400 additional I/Os and since we are talking about metadata for which we keep 2 extra copies for redundancy we need to triple that number, leading to a total of 1800 I/Os per VDEV every TXG. We could try and decrease the number of metaslabs so we have less I/Os per TXG but then each metaslab would cover a wider range on disk and thus would take more time to be loaded in memory from disk. In addition, after it's loaded, it's range tree would consume more memory. Another idea would be to just increase the spacemap block size which would allow us to fit more entries within an I/O block resulting in fewer I/Os per metaslab and a speedup in loading time. The problem is still that we don't deal with the number of I/Os going up as the number of metaslabs is increasing and the fact is that we generally write a lot to a few metaslabs and a little to the rest of them. Thus, just increasing the block size would actually waste bandwidth because we won't be utilizing our bigger block size. = About this patch This patch introduces the Log Spacemap project which provides the solution to the above problem while taking into account all the aforementioned tradeoffs. The details on how it achieves that can be found in the references sections below and in the code (see Big Theory Statement in spa_log_spacemap.c). Even though the change is fairly constraint within the metaslab and lower-level SPA codepaths, there is a side-change that is user-facing. The change is that VDEV IDs from VDEV holes will no longer be reused. To give some background and reasoning for this, when a log device is removed and its VDEV structure was replaced with a hole (or was compacted; if at the end of the vdev array), its vdev_id could be reused by devices added after that. Now with the pool-wide space maps recording the vdev ID, this behavior can cause problems (e.g. is this entry referring to a segment in the new vdev or the removed log?). Thus, to simplify things the ID reuse behavior is gone and now vdev IDs for top-level vdevs are truly unique within a pool. = Testing The illumos implementation of this feature has been used internally for a year and has been in production for ~6 months. For this patch specifically there don't seem to be any regressions introduced to ZTS and I have been running zloop for a week without any related problems. = Performance Analysis (Linux Specific) All performance results and analysis for illumos can be found in the links of the references. Redoing the same experiments in Linux gave similar results. Below are the specifics of the Linux run. After the pool reached stable state the percentage of the time spent in pass 1 per TXG was 64% on average for the stock bits while the log spacemap bits stayed at 95% during the experiment (graph: sdimitro.github.io/img/linux-lsm/PercOfSyncInPassOne.png). Sync times per TXG were 37.6 seconds on average for the stock bits and 22.7 seconds for the log spacemap bits (related graph: sdimitro.github.io/img/linux-lsm/SyncTimePerTXG.png). As a result the log spacemap bits were able to push more TXGs, which is also the reason why all graphs quantified per TXG have more entries for the log spacemap bits. Another interesting aspect in terms of txg syncs is that the stock bits had 22% of their TXGs reach sync pass 7, 55% reach sync pass 8, and 20% reach 9. The log space map bits reached sync pass 4 in 79% of their TXGs, sync pass 7 in 19%, and sync pass 8 at 1%. This emphasizes the fact that not only we spend less time on metadata but we also iterate less times to convergence in spa_sync() dirtying objects. [related graphs: stock- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGStock.png lsm- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGLSM.png] Finally, the improvement in IOPs that the userland gains from the change is approximately 40%. There is a consistent win in IOPS as you can see from the graphs below but the absolute amount of improvement that the log spacemap gives varies within each minute interval. sdimitro.github.io/img/linux-lsm/StockVsLog3Days.png sdimitro.github.io/img/linux-lsm/StockVsLog10Hours.png = Porting to Other Platforms For people that want to port this commit to other platforms below is a list of ZoL commits that this patch depends on: Make zdb results for checkpoint tests consistent db587941c5ff6dea01932bb78f70db63cf7f38ba Update vdev_is_spacemap_addressable() for new spacemap encoding 419ba5914552c6185afbe1dd17b3ed4b0d526547 Simplify spa_sync by breaking it up to smaller functions 8dc2197b7b1e4d7ebc1420ea30e51c6541f1d834 Factor metaslab_load_wait() in metaslab_load() b194fab0fb6caad18711abccaff3c69ad8b3f6d3 Rename range_tree_verify to range_tree_verify_not_present df72b8bebe0ebac0b20e0750984bad182cb6564a Change target size of metaslabs from 256GB to 16GB c853f382db731e15a87512f4ef1101d14d778a55 zdb -L should skip leak detection altogether 21e7cf5da89f55ce98ec1115726b150e19eefe89 vs_alloc can underflow in L2ARC vdevs 7558997d2f808368867ca7e5234e5793446e8f3f Simplify log vdev removal code 6c926f426a26ffb6d7d8e563e33fc176164175cb Get rid of space_map_update() for ms_synced_length 425d3237ee88abc53d8522a7139c926d278b4b7f Introduce auxiliary metaslab histograms 928e8ad47d3478a3d5d01f0dd6ae74a9371af65e Error path in metaslab_load_impl() forgets to drop ms_sync_lock 8eef997679ba54547f7d361553d21b3291f41ae7 = References Background, Motivation, and Internals of the Feature - OpenZFS 2017 Presentation: youtu.be/jj2IxRkl5bQ - Slides: slideshare.net/SerapheimNikolaosDim/zfs-log-spacemaps-project Flushing Algorithm Internals & Performance Results (Illumos Specific) - Blogpost: sdimitro.github.io/post/zfs-lsm-flushing/ - OpenZFS 2018 Presentation: youtu.be/x6D2dHRjkxw - Slides: slideshare.net/SerapheimNikolaosDim/zfs-log-spacemap-flushing-algorithm Upstream Delphix Issues: DLPX-51539, DLPX-59659, DLPX-57783, DLPX-61438, DLPX-41227, DLPX-59320 DLPX-63385 Reviewed-by: Sean Eric Fagan Reviewed-by: Matt Ahrens Reviewed-by: George Wilson Reviewed-by: Brian Behlendorf Signed-off-by: Serapheim Dimitropoulos Closes #8442 --- tests/runfiles/linux.run | 6 ++ tests/zfs-tests/tests/functional/Makefile.am | 3 +- .../tests/functional/arc/dbufstats_001_pos.ksh | 2 +- .../functional/cli_root/zpool_get/zpool_get.cfg | 1 + .../tests/functional/log_spacemap/Makefile.am | 2 + .../log_spacemap/log_spacemap_import_logs.ksh | 81 ++++++++++++++++++++++ .../functional/removal/removal_condense_export.ksh | 4 +- 7 files changed, 95 insertions(+), 4 deletions(-) create mode 100644 tests/zfs-tests/tests/functional/log_spacemap/Makefile.am create mode 100755 tests/zfs-tests/tests/functional/log_spacemap/log_spacemap_import_logs.ksh (limited to 'tests') diff --git a/tests/runfiles/linux.run b/tests/runfiles/linux.run index 43c7a748f..36ab5ef22 100644 --- a/tests/runfiles/linux.run +++ b/tests/runfiles/linux.run @@ -927,3 +927,9 @@ tags = ['functional', 'zvol', 'zvol_swap'] [tests/functional/libzfs] tests = ['many_fds', 'libzfs_input'] tags = ['functional', 'libzfs'] + +[tests/functional/log_spacemap] +tests = ['log_spacemap_import_logs'] +pre = +post = +tags = ['functional', 'log_spacemap'] diff --git a/tests/zfs-tests/tests/functional/Makefile.am b/tests/zfs-tests/tests/functional/Makefile.am index 2d23eb296..783fdfb8a 100644 --- a/tests/zfs-tests/tests/functional/Makefile.am +++ b/tests/zfs-tests/tests/functional/Makefile.am @@ -33,8 +33,8 @@ SUBDIRS = \ largest_pool \ libzfs \ limits \ - pyzfs \ link_count \ + log_spacemap \ migration \ mmap \ mmp \ @@ -50,6 +50,7 @@ SUBDIRS = \ privilege \ procfs \ projectquota \ + pyzfs \ quota \ raidz \ redacted_send \ diff --git a/tests/zfs-tests/tests/functional/arc/dbufstats_001_pos.ksh b/tests/zfs-tests/tests/functional/arc/dbufstats_001_pos.ksh index 7ec9eaf4c..4884f11bb 100755 --- a/tests/zfs-tests/tests/functional/arc/dbufstats_001_pos.ksh +++ b/tests/zfs-tests/tests/functional/arc/dbufstats_001_pos.ksh @@ -58,7 +58,7 @@ function testdbufstat # stat_name dbufstat_filter from_dbufstat=$(grep -w "$name" "$DBUFSTATS_FILE" | awk '{ print $3 }') from_dbufs=$(dbufstat -bxn -i "$DBUFS_FILE" "$filter" | wc -l) - within_tolerance $from_dbufstat $from_dbufs 9 \ + within_tolerance $from_dbufstat $from_dbufs 15 \ || log_fail "Stat $name exceeded tolerance" } diff --git a/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg b/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg index a9ca718cc..717ee9cb2 100644 --- a/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg +++ b/tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg @@ -79,6 +79,7 @@ typeset -a properties=( "feature@redaction_bookmarks" "feature@redacted_datasets" "feature@bookmark_written" + "feature@log_spacemap" ) # Additional properties added for Linux. diff --git a/tests/zfs-tests/tests/functional/log_spacemap/Makefile.am b/tests/zfs-tests/tests/functional/log_spacemap/Makefile.am new file mode 100644 index 000000000..a1e523426 --- /dev/null +++ b/tests/zfs-tests/tests/functional/log_spacemap/Makefile.am @@ -0,0 +1,2 @@ +pkgdatadir = $(datadir)/@PACKAGE@/zfs-tests/tests/functional/log_spacemap +dist_pkgdata_SCRIPTS = log_spacemap_import_logs.ksh diff --git a/tests/zfs-tests/tests/functional/log_spacemap/log_spacemap_import_logs.ksh b/tests/zfs-tests/tests/functional/log_spacemap/log_spacemap_import_logs.ksh new file mode 100755 index 000000000..702322a0c --- /dev/null +++ b/tests/zfs-tests/tests/functional/log_spacemap/log_spacemap_import_logs.ksh @@ -0,0 +1,81 @@ +#! /bin/ksh -p +# +# CDDL HEADER START +# +# This file and its contents are supplied under the terms of the +# Common Development and Distribution License ("CDDL"), version 1.0. +# You may only use this file in accordance with the terms of version +# 1.0 of the CDDL. +# +# A full copy of the text of the CDDL should have accompanied this +# source. A copy of the CDDL is also available via the Internet at +# http://www.illumos.org/license/CDDL. +# +# CDDL HEADER END +# + +# +# Copyright (c) 2019 by Delphix. All rights reserved. +# + +. $STF_SUITE/include/libtest.shlib + +# +# DESCRIPTION: +# Log spacemaps are generally destroyed at export in order to +# not induce performance overheads at import time. As a result, +# the log spacemap codepaths that read the logs in import times +# are not tested outside of ztest and pools with DEBUG bits doing +# many imports/exports while running the test suite. +# +# This test uses an internal tunable and forces ZFS to keep the +# log spacemaps at export, and then re-imports the pool, thus +# providing explicit testing of those codepaths. It also uses +# another tunable to load all the metaslabs when the pool is +# re-imported so more assertions and verifications will be hit. +# +# STRATEGY: +# 1. Create pool. +# 2. Do a couple of writes to generate some data for spacemap logs. +# 3. Set tunable to keep logs after export. +# 4. Export pool and verify that there are logs with zdb. +# 5. Set tunable to load all metaslabs at import. +# 6. Import pool. +# 7. Reset tunables. +# + +verify_runnable "global" + +function cleanup +{ + log_must set_tunable64 zfs_keep_log_spacemaps_at_export 0 + log_must set_tunable64 metaslab_debug_load 0 + if poolexists $LOGSM_POOL; then + log_must zpool destroy -f $LOGSM_POOL + fi +} +log_onexit cleanup + +LOGSM_POOL="logsm_import" +TESTDISK="$(echo $DISKS | cut -d' ' -f1)" + +log_must zpool create -o cachefile=none -f $LOGSM_POOL $TESTDISK +log_must zfs create $LOGSM_POOL/fs + +log_must dd if=/dev/urandom of=/$LOGSM_POOL/fs/00 bs=128k count=10 +log_must sync +log_must dd if=/dev/urandom of=/$LOGSM_POOL/fs/00 bs=128k count=10 +log_must sync + +log_must set_tunable64 zfs_keep_log_spacemaps_at_export 1 +log_must zpool export $LOGSM_POOL + +LOGSM_COUNT=$(zdb -m -e $LOGSM_POOL | grep "Log Spacemap object" | wc -l) +if (( LOGSM_COUNT == 0 )); then + log_fail "Pool does not have any log spacemaps after being exported" +fi + +log_must set_tunable64 metaslab_debug_load 1 +log_must zpool import $LOGSM_POOL + +log_pass "Log spacemaps imported with no errors" diff --git a/tests/zfs-tests/tests/functional/removal/removal_condense_export.ksh b/tests/zfs-tests/tests/functional/removal/removal_condense_export.ksh index 655352f05..b5c2ed0ba 100755 --- a/tests/zfs-tests/tests/functional/removal/removal_condense_export.ksh +++ b/tests/zfs-tests/tests/functional/removal/removal_condense_export.ksh @@ -30,7 +30,7 @@ function reset default_setup_noexit "$DISKS" "true" log_onexit reset -log_must set_tunable64 zfs_condense_indirect_commit_entry_delay_ms 1000 +log_must set_tunable64 zfs_condense_indirect_commit_entry_delay_ms 5000 log_must set_tunable64 zfs_condense_min_mapping_bytes 1 log_must zfs set recordsize=512 $TESTPOOL/$TESTFS @@ -82,7 +82,7 @@ log_mustnot vdevs_in_pool $TESTPOOL $REMOVEDISK log_must stride_dd -i /dev/urandom -o $TESTDIR/file -b 512 -c 20 -s 1024 sync_pool $TESTPOOL -sleep 5 +sleep 4 sync_pool $TESTPOOL log_must zpool export $TESTPOOL zdb -e -p $REMOVEDISKPATH $TESTPOOL | grep 'Condensing indirect vdev' || \ -- cgit v1.2.3