2013-11-16 06:52:54 +00:00
|
|
|
'\" te
|
|
|
|
.\" Copyright (c) 2013 by Turbo Fredriksson <turbo@bayour.com>. All rights reserved.
|
2020-09-04 17:34:28 +00:00
|
|
|
.\" Copyright (c) 2019, 2020 by Delphix. All rights reserved.
|
2019-02-12 20:01:08 +00:00
|
|
|
.\" Copyright (c) 2019 Datto Inc.
|
2013-11-16 06:52:54 +00:00
|
|
|
.\" The contents of this file are subject to the terms of the Common Development
|
|
|
|
.\" and Distribution License (the "License"). You may not use this file except
|
|
|
|
.\" in compliance with the License. You can obtain a copy of the license at
|
|
|
|
.\" usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing.
|
|
|
|
.\"
|
|
|
|
.\" See the License for the specific language governing permissions and
|
|
|
|
.\" limitations under the License. When distributing Covered Code, include this
|
|
|
|
.\" CDDL HEADER in each file and include the License file at
|
|
|
|
.\" usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this
|
|
|
|
.\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your
|
|
|
|
.\" own identifying information:
|
|
|
|
.\" Portions Copyright [yyyy] [name of copyright owner]
|
2020-08-25 04:25:28 +00:00
|
|
|
.TH ZFS-MODULE-PARAMETERS 5 "Aug 24, 2020" OpenZFS
|
2013-11-16 06:52:54 +00:00
|
|
|
.SH NAME
|
|
|
|
zfs\-module\-parameters \- ZFS module parameters
|
|
|
|
.SH DESCRIPTION
|
|
|
|
.sp
|
|
|
|
.LP
|
|
|
|
Description of the different parameters to the ZFS module.
|
|
|
|
|
|
|
|
.SS "Module parameters"
|
|
|
|
.sp
|
|
|
|
.LP
|
|
|
|
|
2018-03-13 17:52:48 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBdbuf_cache_max_bytes\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2020-07-25 03:38:48 +00:00
|
|
|
Maximum size in bytes of the dbuf cache. The target size is determined by the
|
|
|
|
MIN versus \fB1/2^dbuf_cache_shift\fR (1/32) of the target ARC size. The
|
|
|
|
behavior of the dbuf cache and its associated settings can be observed via the
|
|
|
|
\fB/proc/spl/kstat/zfs/dbufstats\fR kstat.
|
2018-03-13 17:52:48 +00:00
|
|
|
.sp
|
2020-07-25 03:38:48 +00:00
|
|
|
Default value: \fBULONG_MAX\fR.
|
2018-03-13 17:52:48 +00:00
|
|
|
.RE
|
|
|
|
|
2018-07-10 17:49:50 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBdbuf_metadata_cache_max_bytes\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2020-07-25 03:38:48 +00:00
|
|
|
Maximum size in bytes of the metadata dbuf cache. The target size is
|
|
|
|
determined by the MIN versus \fB1/2^dbuf_metadata_cache_shift\fR (1/64) of the
|
|
|
|
target ARC size. The behavior of the metadata dbuf cache and its associated
|
|
|
|
settings can be observed via the \fB/proc/spl/kstat/zfs/dbufstats\fR kstat.
|
2018-07-10 17:49:50 +00:00
|
|
|
.sp
|
2020-07-25 03:38:48 +00:00
|
|
|
Default value: \fBULONG_MAX\fR.
|
2018-07-10 17:49:50 +00:00
|
|
|
.RE
|
|
|
|
|
2018-03-13 17:52:48 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBdbuf_cache_hiwater_pct\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The percentage over \fBdbuf_cache_max_bytes\fR when dbufs must be evicted
|
|
|
|
directly.
|
|
|
|
.sp
|
|
|
|
Default value: \fB10\fR%.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBdbuf_cache_lowater_pct\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The percentage below \fBdbuf_cache_max_bytes\fR when the evict thread stops
|
|
|
|
evicting dbufs.
|
|
|
|
.sp
|
|
|
|
Default value: \fB10\fR%.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBdbuf_cache_shift\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Set the size of the dbuf cache, \fBdbuf_cache_max_bytes\fR, to a log2 fraction
|
2020-04-10 17:33:35 +00:00
|
|
|
of the target ARC size.
|
2018-03-13 17:52:48 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB5\fR.
|
|
|
|
.RE
|
|
|
|
|
2018-07-10 17:49:50 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBdbuf_metadata_cache_shift\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Set the size of the dbuf metadata cache, \fBdbuf_metadata_cache_max_bytes\fR,
|
2020-04-10 17:33:35 +00:00
|
|
|
to a log2 fraction of the target ARC size.
|
2018-07-10 17:49:50 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB6\fR.
|
|
|
|
.RE
|
|
|
|
|
2020-01-07 20:41:53 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBdmu_object_alloc_chunk_shift\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
dnode slots allocated in a single operation as a power of 2. The default value
|
|
|
|
minimizes lock contention for the bulk operation performed.
|
|
|
|
.sp
|
|
|
|
Default value: \fB7\fR (128).
|
|
|
|
.RE
|
|
|
|
|
2019-06-12 20:13:09 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBdmu_prefetch_max\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Limit the amount we can prefetch with one call to this amount (in bytes).
|
|
|
|
This helps to limit the amount of memory that can be used by prefetching.
|
|
|
|
.sp
|
|
|
|
Default value: \fB134,217,728\fR (128MB).
|
|
|
|
.RE
|
|
|
|
|
2016-07-08 20:51:50 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBignore_hole_birth\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2019-04-19 01:15:58 +00:00
|
|
|
This is an alias for \fBsend_holes_without_birth_time\fR.
|
2016-07-08 20:51:50 +00:00
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBl2arc_feed_again\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Turbo L2ARC warm-up. When the L2ARC is cold the fill interval will be set as
|
|
|
|
fast as possible.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes (default) and \fB0\fR to disable.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBl2arc_feed_min_ms\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Min feed interval in milliseconds. Requires \fBl2arc_feed_again=1\fR and only
|
|
|
|
applicable in related situations.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB200\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBl2arc_feed_secs\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Seconds between L2ARC writing
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBl2arc_headroom\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
How far through the ARC lists to search for L2ARC cacheable content, expressed
|
2020-04-10 17:33:35 +00:00
|
|
|
as a multiplier of \fBl2arc_write_max\fR.
|
|
|
|
ARC persistence across reboots can be achieved with persistent L2ARC by setting
|
|
|
|
this parameter to \fB0\fR allowing the full length of ARC lists to be searched
|
|
|
|
for cacheable content.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB2\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBl2arc_headroom_boost\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Scales \fBl2arc_headroom\fR by this percentage when L2ARC contents are being
|
2020-06-09 17:15:08 +00:00
|
|
|
successfully compressed before writing. A value of \fB100\fR disables this
|
|
|
|
feature.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB200\fR%.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
2020-09-08 18:44:37 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBl2arc_mfuonly\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Controls whether only MFU metadata and data are cached from ARC into L2ARC.
|
|
|
|
This may be desired to avoid wasting space on L2ARC when reading/writing large
|
|
|
|
amounts of data that are not expected to be accessed more than once. The
|
|
|
|
default is \fB0\fR, meaning both MRU and MFU data and metadata are cached.
|
|
|
|
When turning off (\fB0\fR) this feature some MRU buffers will still be present
|
Add L2ARC arcstats for MFU/MRU buffers and buffer content type
Currently the ARC state (MFU/MRU) of cached L2ARC buffer and their
content type is unknown. Knowing this information may prove beneficial
in adjusting the L2ARC caching policy.
This commit adds L2ARC arcstats that display the aligned size
(in bytes) of L2ARC buffers according to their content type
(data/metadata) and according to their ARC state (MRU/MFU or
prefetch). It also expands the existing evict_l2_eligible arcstat to
differentiate between MFU and MRU buffers.
L2ARC caches buffers from the MRU and MFU lists of ARC. Upon caching a
buffer, its ARC state (MRU/MFU) is stored in the L2 header
(b_arcs_state). The l2_m{f,r}u_asize arcstats reflect the aligned size
(in bytes) of L2ARC buffers according to their ARC state (based on
b_arcs_state). We also account for the case where an L2ARC and ARC
cached MRU or MRU_ghost buffer transitions to MFU. The l2_prefetch_asize
reflects the alinged size (in bytes) of L2ARC buffers that were cached
while they had the prefetch flag set in ARC. This is dynamically updated
as the prefetch flag of L2ARC buffers changes.
When buffers are evicted from ARC, if they are determined to be L2ARC
eligible then their logical size is recorded in
evict_l2_eligible_m{r,f}u arcstats according to their ARC state upon
eviction.
Persistent L2ARC:
When committing an L2ARC buffer to a log block (L2ARC metadata) its
b_arcs_state and prefetch flag is also stored. If the buffer changes
its arcstate or prefetch flag this is reflected in the above arcstats.
However, the L2ARC metadata cannot currently be updated to reflect this
change.
Example: L2ARC caches an MRU buffer. L2ARC metadata and arcstats count
this as an MRU buffer. The buffer transitions to MFU. The arcstats are
updated to reflect this. Upon pool re-import or on/offlining the L2ARC
device the arcstats are cleared and the buffer will now be counted as an
MRU buffer, as the L2ARC metadata were not updated.
Bug fix:
- If l2arc_noprefetch is set, arc_read_done clears the L2CACHE flag of
an ARC buffer. However, prefetches may be issued in a way that
arc_read_done() is bypassed. Instead, move the related code in
l2arc_write_eligible() to account for those cases too.
Also add a test and update manpages for l2arc_mfuonly module parameter,
and update the manpages and code comments for l2arc_noprefetch.
Move persist_l2arc tests to l2arc.
Reviewed-by: Ryan Moeller <freqlabs@FreeBSD.org>
Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Amanakis <gamanakis@gmail.com>
Closes #10743
2020-09-14 17:10:44 +00:00
|
|
|
in ARC and eventually cached on L2ARC. If \fBl2arc_noprefetch\fR is set to 0,
|
|
|
|
some prefetched buffers will be cached to L2ARC, and those might later
|
|
|
|
transition to MRU, in which case the \fBl2arc_mru_asize\fR arcstat will not
|
|
|
|
be 0. Regardless of \fBl2arc_noprefetch\fR, some MFU buffers might be evicted
|
|
|
|
from ARC, accessed later on as prefetches and transition to MRU as prefetches.
|
|
|
|
If accessed again they are counted as MRU and the \fBl2arc_mru_asize\fR arcstat
|
|
|
|
will not be 0. The ARC status of L2ARC buffers when they were first cached in
|
|
|
|
L2ARC can be seen in the \fBl2arc_mru_asize\fR, \fBl2arc_mfu_asize\fR and
|
|
|
|
\fBl2arc_prefetch_asize\fR arcstats when importing the pool or onlining a cache
|
2020-09-21 16:26:24 +00:00
|
|
|
device if persistent L2ARC is enabled. The \fBevicted_l2_eligible_mru\fR
|
|
|
|
arcstat does not take into account if this option is enabled as the information
|
|
|
|
provided by the evicted_l2_eligible_* arcstats can be used to decide if
|
|
|
|
toggling this option is appropriate for the current workload.
|
2020-09-08 18:44:37 +00:00
|
|
|
.sp
|
|
|
|
Use \fB0\fR for no (default) and \fB1\fR for yes.
|
|
|
|
.RE
|
|
|
|
|
2020-08-25 21:33:36 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBl2arc_meta_percent\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Percent of ARC size allowed for L2ARC-only headers.
|
|
|
|
Since L2ARC buffers are not evicted on memory pressure, too large amount of
|
|
|
|
headers on system with irrationaly large L2ARC can render it slow or unusable.
|
|
|
|
This parameter limits L2ARC writes and rebuild to achieve it.
|
|
|
|
.sp
|
|
|
|
Default value: \fB33\fR%.
|
|
|
|
.RE
|
|
|
|
|
2020-06-09 17:15:08 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBl2arc_trim_ahead\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Trims ahead of the current write size (\fBl2arc_write_max\fR) on L2ARC devices
|
|
|
|
by this percentage of write size if we have filled the device. If set to
|
|
|
|
\fB100\fR we TRIM twice the space required to accommodate upcoming writes. A
|
|
|
|
minimum of 64MB will be trimmed. It also enables TRIM of the whole L2ARC device
|
|
|
|
upon creation or addition to an existing pool or if the header of the device is
|
|
|
|
invalid upon importing a pool or onlining a cache device. A value of \fB0\fR
|
|
|
|
disables TRIM on L2ARC altogether and is the default as it can put significant
|
|
|
|
stress on the underlying storage devices. This will vary depending of how well
|
|
|
|
the specific device handles these commands.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR%.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBl2arc_noprefetch\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Do not write buffers to L2ARC if they were prefetched but not used by
|
Add L2ARC arcstats for MFU/MRU buffers and buffer content type
Currently the ARC state (MFU/MRU) of cached L2ARC buffer and their
content type is unknown. Knowing this information may prove beneficial
in adjusting the L2ARC caching policy.
This commit adds L2ARC arcstats that display the aligned size
(in bytes) of L2ARC buffers according to their content type
(data/metadata) and according to their ARC state (MRU/MFU or
prefetch). It also expands the existing evict_l2_eligible arcstat to
differentiate between MFU and MRU buffers.
L2ARC caches buffers from the MRU and MFU lists of ARC. Upon caching a
buffer, its ARC state (MRU/MFU) is stored in the L2 header
(b_arcs_state). The l2_m{f,r}u_asize arcstats reflect the aligned size
(in bytes) of L2ARC buffers according to their ARC state (based on
b_arcs_state). We also account for the case where an L2ARC and ARC
cached MRU or MRU_ghost buffer transitions to MFU. The l2_prefetch_asize
reflects the alinged size (in bytes) of L2ARC buffers that were cached
while they had the prefetch flag set in ARC. This is dynamically updated
as the prefetch flag of L2ARC buffers changes.
When buffers are evicted from ARC, if they are determined to be L2ARC
eligible then their logical size is recorded in
evict_l2_eligible_m{r,f}u arcstats according to their ARC state upon
eviction.
Persistent L2ARC:
When committing an L2ARC buffer to a log block (L2ARC metadata) its
b_arcs_state and prefetch flag is also stored. If the buffer changes
its arcstate or prefetch flag this is reflected in the above arcstats.
However, the L2ARC metadata cannot currently be updated to reflect this
change.
Example: L2ARC caches an MRU buffer. L2ARC metadata and arcstats count
this as an MRU buffer. The buffer transitions to MFU. The arcstats are
updated to reflect this. Upon pool re-import or on/offlining the L2ARC
device the arcstats are cleared and the buffer will now be counted as an
MRU buffer, as the L2ARC metadata were not updated.
Bug fix:
- If l2arc_noprefetch is set, arc_read_done clears the L2CACHE flag of
an ARC buffer. However, prefetches may be issued in a way that
arc_read_done() is bypassed. Instead, move the related code in
l2arc_write_eligible() to account for those cases too.
Also add a test and update manpages for l2arc_mfuonly module parameter,
and update the manpages and code comments for l2arc_noprefetch.
Move persist_l2arc tests to l2arc.
Reviewed-by: Ryan Moeller <freqlabs@FreeBSD.org>
Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Amanakis <gamanakis@gmail.com>
Closes #10743
2020-09-14 17:10:44 +00:00
|
|
|
applications. In case there are prefetched buffers in L2ARC and this option
|
|
|
|
is later set to \fB1\fR, we do not read the prefetched buffers from L2ARC.
|
|
|
|
Setting this option to \fB0\fR is useful for caching sequential reads from the
|
|
|
|
disks to L2ARC and serve those reads from L2ARC later on. This may be beneficial
|
|
|
|
in case the L2ARC device is significantly faster in sequential reads than the
|
|
|
|
disks of the pool.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR to disable (default) and \fB0\fR to enable caching/reading
|
|
|
|
prefetches to/from L2ARC..
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBl2arc_norw\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2020-04-10 17:33:35 +00:00
|
|
|
No reads during writes.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBl2arc_write_boost\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-04-24 17:56:44 +00:00
|
|
|
Cold L2ARC devices will have \fBl2arc_write_max\fR increased by this amount
|
2015-12-30 17:44:46 +00:00
|
|
|
while they remain cold.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB8,388,608\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBl2arc_write_max\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2020-04-10 17:33:35 +00:00
|
|
|
Max write bytes per interval.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB8,388,608\fR.
|
|
|
|
.RE
|
|
|
|
|
2020-04-10 17:33:35 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBl2arc_rebuild_enabled\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Rebuild the L2ARC when importing a pool (persistent L2ARC). This can be
|
|
|
|
disabled if there are problems importing a pool or attaching an L2ARC device
|
|
|
|
(e.g. the L2ARC device is slow in reading stored log metadata, or the metadata
|
|
|
|
has become somehow fragmented/unusable).
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes (default) and \fB0\fR for no.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBl2arc_rebuild_blocks_min_l2size\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Min size (in bytes) of an L2ARC device required in order to write log blocks
|
|
|
|
in it. The log blocks are used upon importing the pool to rebuild
|
|
|
|
the L2ARC (persistent L2ARC). Rationale: for L2ARC devices less than 1GB, the
|
|
|
|
amount of data l2arc_evict() evicts is significant compared to the amount of
|
|
|
|
restored L2ARC data. In this case do not write log blocks in L2ARC in order not
|
|
|
|
to waste space.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1,073,741,824\fR (1GB).
|
|
|
|
.RE
|
|
|
|
|
2015-05-10 15:40:20 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBmetaslab_aliquot\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Metaslab granularity, in bytes. This is roughly similar to what would be
|
|
|
|
referred to as the "stripe size" in traditional RAID arrays. In normal
|
|
|
|
operation, ZFS will try to write this amount of data to a top-level vdev
|
|
|
|
before moving on to the next one.
|
|
|
|
.sp
|
|
|
|
Default value: \fB524,288\fR.
|
|
|
|
.RE
|
|
|
|
|
2014-07-19 20:19:24 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBmetaslab_bias_enabled\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Enable metaslab group biasing based on its vdev's over- or under-utilization
|
|
|
|
relative to the pool.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes (default) and \fB0\fR for no.
|
|
|
|
.RE
|
|
|
|
|
2017-07-24 18:07:39 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBmetaslab_force_ganging\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Make some blocks above a certain size be gang blocks. This option is used
|
|
|
|
by the test suite to facilitate testing.
|
|
|
|
.sp
|
|
|
|
Default value: \fB16,777,217\fR.
|
|
|
|
.RE
|
|
|
|
|
2020-11-14 18:17:16 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_history_output_max\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
When attempting to log the output nvlist of an ioctl in the on-disk history, the
|
|
|
|
output will not be stored if it is larger than size (in bytes). This must be
|
|
|
|
less then DMU_MAX_ACCESS (64MB). This applies primarily to
|
|
|
|
zfs_ioc_channel_program().
|
|
|
|
.sp
|
|
|
|
Default value: \fB1MB\fR.
|
|
|
|
.RE
|
|
|
|
|
Log Spacemap Project
= Motivation
At Delphix we've seen a lot of customer systems where fragmentation
is over 75% and random writes take a performance hit because a lot
of time is spend on I/Os that update on-disk space accounting metadata.
Specifically, we seen cases where 20% to 40% of sync time is spend
after sync pass 1 and ~30% of the I/Os on the system is spent updating
spacemaps.
The problem is that these pools have existed long enough that we've
touched almost every metaslab at least once, and random writes
scatter frees across all metaslabs every TXG, thus appending to
their spacemaps and resulting in many I/Os. To give an example,
assuming that every VDEV has 200 metaslabs and our writes fit within
a single spacemap block (generally 4K) we have 200 I/Os. Then if we
assume 2 levels of indirection, we need 400 additional I/Os and
since we are talking about metadata for which we keep 2 extra copies
for redundancy we need to triple that number, leading to a total of
1800 I/Os per VDEV every TXG.
We could try and decrease the number of metaslabs so we have less
I/Os per TXG but then each metaslab would cover a wider range on
disk and thus would take more time to be loaded in memory from disk.
In addition, after it's loaded, it's range tree would consume more
memory.
Another idea would be to just increase the spacemap block size
which would allow us to fit more entries within an I/O block
resulting in fewer I/Os per metaslab and a speedup in loading time.
The problem is still that we don't deal with the number of I/Os
going up as the number of metaslabs is increasing and the fact
is that we generally write a lot to a few metaslabs and a little
to the rest of them. Thus, just increasing the block size would
actually waste bandwidth because we won't be utilizing our bigger
block size.
= About this patch
This patch introduces the Log Spacemap project which provides the
solution to the above problem while taking into account all the
aforementioned tradeoffs. The details on how it achieves that can
be found in the references sections below and in the code (see
Big Theory Statement in spa_log_spacemap.c).
Even though the change is fairly constraint within the metaslab
and lower-level SPA codepaths, there is a side-change that is
user-facing. The change is that VDEV IDs from VDEV holes will no
longer be reused. To give some background and reasoning for this,
when a log device is removed and its VDEV structure was replaced
with a hole (or was compacted; if at the end of the vdev array),
its vdev_id could be reused by devices added after that. Now
with the pool-wide space maps recording the vdev ID, this behavior
can cause problems (e.g. is this entry referring to a segment in
the new vdev or the removed log?). Thus, to simplify things the
ID reuse behavior is gone and now vdev IDs for top-level vdevs
are truly unique within a pool.
= Testing
The illumos implementation of this feature has been used internally
for a year and has been in production for ~6 months. For this patch
specifically there don't seem to be any regressions introduced to
ZTS and I have been running zloop for a week without any related
problems.
= Performance Analysis (Linux Specific)
All performance results and analysis for illumos can be found in
the links of the references. Redoing the same experiments in Linux
gave similar results. Below are the specifics of the Linux run.
After the pool reached stable state the percentage of the time
spent in pass 1 per TXG was 64% on average for the stock bits
while the log spacemap bits stayed at 95% during the experiment
(graph: sdimitro.github.io/img/linux-lsm/PercOfSyncInPassOne.png).
Sync times per TXG were 37.6 seconds on average for the stock
bits and 22.7 seconds for the log spacemap bits (related graph:
sdimitro.github.io/img/linux-lsm/SyncTimePerTXG.png). As a result
the log spacemap bits were able to push more TXGs, which is also
the reason why all graphs quantified per TXG have more entries for
the log spacemap bits.
Another interesting aspect in terms of txg syncs is that the stock
bits had 22% of their TXGs reach sync pass 7, 55% reach sync pass 8,
and 20% reach 9. The log space map bits reached sync pass 4 in 79%
of their TXGs, sync pass 7 in 19%, and sync pass 8 at 1%. This
emphasizes the fact that not only we spend less time on metadata
but we also iterate less times to convergence in spa_sync() dirtying
objects.
[related graphs:
stock- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGStock.png
lsm- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGLSM.png]
Finally, the improvement in IOPs that the userland gains from the
change is approximately 40%. There is a consistent win in IOPS as
you can see from the graphs below but the absolute amount of
improvement that the log spacemap gives varies within each minute
interval.
sdimitro.github.io/img/linux-lsm/StockVsLog3Days.png
sdimitro.github.io/img/linux-lsm/StockVsLog10Hours.png
= Porting to Other Platforms
For people that want to port this commit to other platforms below
is a list of ZoL commits that this patch depends on:
Make zdb results for checkpoint tests consistent
db587941c5ff6dea01932bb78f70db63cf7f38ba
Update vdev_is_spacemap_addressable() for new spacemap encoding
419ba5914552c6185afbe1dd17b3ed4b0d526547
Simplify spa_sync by breaking it up to smaller functions
8dc2197b7b1e4d7ebc1420ea30e51c6541f1d834
Factor metaslab_load_wait() in metaslab_load()
b194fab0fb6caad18711abccaff3c69ad8b3f6d3
Rename range_tree_verify to range_tree_verify_not_present
df72b8bebe0ebac0b20e0750984bad182cb6564a
Change target size of metaslabs from 256GB to 16GB
c853f382db731e15a87512f4ef1101d14d778a55
zdb -L should skip leak detection altogether
21e7cf5da89f55ce98ec1115726b150e19eefe89
vs_alloc can underflow in L2ARC vdevs
7558997d2f808368867ca7e5234e5793446e8f3f
Simplify log vdev removal code
6c926f426a26ffb6d7d8e563e33fc176164175cb
Get rid of space_map_update() for ms_synced_length
425d3237ee88abc53d8522a7139c926d278b4b7f
Introduce auxiliary metaslab histograms
928e8ad47d3478a3d5d01f0dd6ae74a9371af65e
Error path in metaslab_load_impl() forgets to drop ms_sync_lock
8eef997679ba54547f7d361553d21b3291f41ae7
= References
Background, Motivation, and Internals of the Feature
- OpenZFS 2017 Presentation:
youtu.be/jj2IxRkl5bQ
- Slides:
slideshare.net/SerapheimNikolaosDim/zfs-log-spacemaps-project
Flushing Algorithm Internals & Performance Results
(Illumos Specific)
- Blogpost:
sdimitro.github.io/post/zfs-lsm-flushing/
- OpenZFS 2018 Presentation:
youtu.be/x6D2dHRjkxw
- Slides:
slideshare.net/SerapheimNikolaosDim/zfs-log-spacemap-flushing-algorithm
Upstream Delphix Issues:
DLPX-51539, DLPX-59659, DLPX-57783, DLPX-61438, DLPX-41227, DLPX-59320
DLPX-63385
Reviewed-by: Sean Eric Fagan <sef@ixsystems.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: George Wilson <gwilson@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8442
2019-07-16 17:11:49 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_keep_log_spacemaps_at_export\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Prevent log spacemaps from being destroyed during pool exports and destroys.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
2017-01-12 19:52:56 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_metaslab_segment_weight_enabled\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Enable/disable segment-based metaslab selection.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes (default) and \fB0\fR for no.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_metaslab_switch_threshold\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
When using segment-based metaslab selection, continue allocating
|
2017-04-24 17:34:37 +00:00
|
|
|
from the active metaslab until \fBzfs_metaslab_switch_threshold\fR
|
2017-01-12 19:52:56 +00:00
|
|
|
worth of buckets have been exhausted.
|
|
|
|
.sp
|
|
|
|
Default value: \fB2\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2014-04-01 00:22:55 +00:00
|
|
|
\fBmetaslab_debug_load\fR (int)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2014-04-01 00:22:55 +00:00
|
|
|
Load all metaslabs during pool import.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBmetaslab_debug_unload\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Prevent metaslabs from being unloaded.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
2014-07-19 20:19:24 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBmetaslab_fragmentation_factor_enabled\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Enable use of the fragmentation metric in computing metaslab weights.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes (default) and \fB0\fR for no.
|
|
|
|
.RE
|
|
|
|
|
looping in metaslab_block_picker impacts performance on fragmented pools
On fragmented pools with high-performance storage, the looping in
metaslab_block_picker() can become the performance-limiting bottleneck.
When looking for a larger block (e.g. a 128K block for the ZIL), we may
search through many free segments (up to hundreds of thousands) to find
one that is large enough to satisfy the allocation. This can take a long
time (up to dozens of ms), and is done while holding the ms_lock, which
other threads may spin waiting for.
When this performance problem is encountered, profiling will show
high CPU time in metaslab_block_picker, as well as in mutex_enter from
various callers.
The problem is very evident on a test system with a sync write workload
with 8K writes to a recordsize=8k filesystem, with 4TB of SSD storage,
84% full and 88% fragmented. It has also been observed on production
systems with 90TB of storage, 76% full and 87% fragmented.
The fix is to change metaslab_df_alloc() to search only up to 16MB from
the previous allocation (of this alignment). After that, we will pick a
segment that is of the exact size requested (or larger). This reduces
the number of iterations to a few hundred on fragmented pools (a ~100x
improvement).
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Tony Nguyen <tony.nguyen@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
External-issue: DLPX-62324
Closes #8877
2019-06-13 20:06:15 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBmetaslab_df_max_search\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum distance to search forward from the last offset. Without this limit,
|
|
|
|
fragmented pools can see >100,000 iterations and metaslab_block_picker()
|
|
|
|
becomes the performance limiting factor on high-performance storage.
|
|
|
|
|
|
|
|
With the default setting of 16MB, we typically see less than 500 iterations,
|
|
|
|
even with very fragmented, ashift=9 pools. The maximum number of iterations
|
|
|
|
possible is: \fBmetaslab_df_max_search / (2 * (1<<ashift))\fR.
|
|
|
|
With the default setting of 16MB this is 16*1024 (with ashift=9) or 2048
|
|
|
|
(with ashift=12).
|
|
|
|
.sp
|
|
|
|
Default value: \fB16,777,216\fR (16MB)
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBmetaslab_df_use_largest_segment\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
If we are not searching forward (due to metaslab_df_max_search,
|
|
|
|
metaslab_df_free_pct, or metaslab_df_alloc_threshold), this tunable controls
|
2020-08-25 04:25:28 +00:00
|
|
|
what segment is used. If it is set, we will use the largest free segment.
|
looping in metaslab_block_picker impacts performance on fragmented pools
On fragmented pools with high-performance storage, the looping in
metaslab_block_picker() can become the performance-limiting bottleneck.
When looking for a larger block (e.g. a 128K block for the ZIL), we may
search through many free segments (up to hundreds of thousands) to find
one that is large enough to satisfy the allocation. This can take a long
time (up to dozens of ms), and is done while holding the ms_lock, which
other threads may spin waiting for.
When this performance problem is encountered, profiling will show
high CPU time in metaslab_block_picker, as well as in mutex_enter from
various callers.
The problem is very evident on a test system with a sync write workload
with 8K writes to a recordsize=8k filesystem, with 4TB of SSD storage,
84% full and 88% fragmented. It has also been observed on production
systems with 90TB of storage, 76% full and 87% fragmented.
The fix is to change metaslab_df_alloc() to search only up to 16MB from
the previous allocation (of this alignment). After that, we will pick a
segment that is of the exact size requested (or larger). This reduces
the number of iterations to a few hundred on fragmented pools (a ~100x
improvement).
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Tony Nguyen <tony.nguyen@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
External-issue: DLPX-62324
Closes #8877
2019-06-13 20:06:15 +00:00
|
|
|
If it is not set, we will use a segment of exactly the requested size (or
|
|
|
|
larger).
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
Metaslab max_size should be persisted while unloaded
When we unload metaslabs today in ZFS, the cached max_size value is
discarded. We instead use the histogram to determine whether or not we
think we can satisfy an allocation from the metaslab. This can result in
situations where, if we're doing I/Os of a size not aligned to a
histogram bucket, a metaslab is loaded even though it cannot satisfy the
allocation we think it can. For example, a metaslab with 16 entries in
the 16k-32k bucket may have entirely 16kB entries. If we try to allocate
a 24kB buffer, we will load that metaslab because we think it should be
able to handle the allocation. Doing so is expensive in CPU time, disk
reads, and average IO latency. This is exacerbated if the write being
attempted is a sync write.
This change makes ZFS cache the max_size after the metaslab is
unloaded. If we ever get a free (or a coalesced group of frees) larger
than the max_size, we will update it. Otherwise, we leave it as is. When
attempting to allocate, we use the max_size as a lower bound, and
respect it unless we are in try_hard. However, we do age the max_size
out at some point, since we expect the actual max_size to increase as we
do more frees. A more sophisticated algorithm here might be helpful, but
this works reasonably well.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #9055
2019-08-05 21:34:27 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_metaslab_max_size_cache_sec\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
When we unload a metaslab, we cache the size of the largest free chunk. We use
|
|
|
|
that cached size to determine whether or not to load a metaslab for a given
|
|
|
|
allocation. As more frees accumulate in that metaslab while it's unloaded, the
|
|
|
|
cached max size becomes less and less accurate. After a number of seconds
|
|
|
|
controlled by this tunable, we stop considering the cached max size and start
|
|
|
|
considering only the histogram instead.
|
|
|
|
.sp
|
|
|
|
Default value: \fB3600 seconds\fR (one hour)
|
|
|
|
.RE
|
|
|
|
|
Cap metaslab memory usage
On systems with large amounts of storage and high fragmentation, a huge
amount of space can be used by storing metaslab range trees. Since
metaslabs are only unloaded during a txg sync, and only if they have
been inactive for 8 txgs, it is possible to get into a state where all
of the system's memory is consumed by range trees and metaslabs, and
txgs cannot sync. While ZFS knows how to evict ARC data when needed,
it has no such mechanism for range tree data. This can result in boot
hangs for some system configurations.
First, we add the ability to unload metaslabs outside of syncing
context. Second, we store a multilist of all loaded metaslabs, sorted
by their selection txg, so we can quickly identify the oldest
metaslabs. We use a multilist to reduce lock contention during heavy
write workloads. Finally, we add logic that will unload a metaslab
when we're loading a new metaslab, if we're using more than a certain
fraction of the available memory on range trees.
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: George Wilson <gwilson@delphix.com>
Reviewed-by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #9128
2019-08-16 15:08:21 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_metaslab_mem_limit\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
When we are loading a new metaslab, we check the amount of memory being used
|
|
|
|
to store metaslab range trees. If it is over a threshold, we attempt to unload
|
|
|
|
the least recently used metaslab to prevent the system from clogging all of
|
|
|
|
its memory with range trees. This tunable sets the percentage of total system
|
|
|
|
memory that is the threshold.
|
|
|
|
.sp
|
2019-08-29 17:20:36 +00:00
|
|
|
Default value: \fB25 percent\fR
|
Cap metaslab memory usage
On systems with large amounts of storage and high fragmentation, a huge
amount of space can be used by storing metaslab range trees. Since
metaslabs are only unloaded during a txg sync, and only if they have
been inactive for 8 txgs, it is possible to get into a state where all
of the system's memory is consumed by range trees and metaslabs, and
txgs cannot sync. While ZFS knows how to evict ARC data when needed,
it has no such mechanism for range tree data. This can result in boot
hangs for some system configurations.
First, we add the ability to unload metaslabs outside of syncing
context. Second, we store a multilist of all loaded metaslabs, sorted
by their selection txg, so we can quickly identify the oldest
metaslabs. We use a multilist to reduce lock contention during heavy
write workloads. Finally, we add logic that will unload a metaslab
when we're loading a new metaslab, if we're using more than a certain
fraction of the available memory on range trees.
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: George Wilson <gwilson@delphix.com>
Reviewed-by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #9128
2019-08-16 15:08:21 +00:00
|
|
|
.RE
|
|
|
|
|
2014-09-13 14:13:00 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2019-01-26 00:38:27 +00:00
|
|
|
\fBzfs_vdev_default_ms_count\fR (int)
|
2014-09-13 14:13:00 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-08-11 21:28:17 +00:00
|
|
|
When a vdev is added target this number of metaslabs per top-level vdev.
|
2014-09-13 14:13:00 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB200\fR.
|
|
|
|
.RE
|
|
|
|
|
Log Spacemap Project
= Motivation
At Delphix we've seen a lot of customer systems where fragmentation
is over 75% and random writes take a performance hit because a lot
of time is spend on I/Os that update on-disk space accounting metadata.
Specifically, we seen cases where 20% to 40% of sync time is spend
after sync pass 1 and ~30% of the I/Os on the system is spent updating
spacemaps.
The problem is that these pools have existed long enough that we've
touched almost every metaslab at least once, and random writes
scatter frees across all metaslabs every TXG, thus appending to
their spacemaps and resulting in many I/Os. To give an example,
assuming that every VDEV has 200 metaslabs and our writes fit within
a single spacemap block (generally 4K) we have 200 I/Os. Then if we
assume 2 levels of indirection, we need 400 additional I/Os and
since we are talking about metadata for which we keep 2 extra copies
for redundancy we need to triple that number, leading to a total of
1800 I/Os per VDEV every TXG.
We could try and decrease the number of metaslabs so we have less
I/Os per TXG but then each metaslab would cover a wider range on
disk and thus would take more time to be loaded in memory from disk.
In addition, after it's loaded, it's range tree would consume more
memory.
Another idea would be to just increase the spacemap block size
which would allow us to fit more entries within an I/O block
resulting in fewer I/Os per metaslab and a speedup in loading time.
The problem is still that we don't deal with the number of I/Os
going up as the number of metaslabs is increasing and the fact
is that we generally write a lot to a few metaslabs and a little
to the rest of them. Thus, just increasing the block size would
actually waste bandwidth because we won't be utilizing our bigger
block size.
= About this patch
This patch introduces the Log Spacemap project which provides the
solution to the above problem while taking into account all the
aforementioned tradeoffs. The details on how it achieves that can
be found in the references sections below and in the code (see
Big Theory Statement in spa_log_spacemap.c).
Even though the change is fairly constraint within the metaslab
and lower-level SPA codepaths, there is a side-change that is
user-facing. The change is that VDEV IDs from VDEV holes will no
longer be reused. To give some background and reasoning for this,
when a log device is removed and its VDEV structure was replaced
with a hole (or was compacted; if at the end of the vdev array),
its vdev_id could be reused by devices added after that. Now
with the pool-wide space maps recording the vdev ID, this behavior
can cause problems (e.g. is this entry referring to a segment in
the new vdev or the removed log?). Thus, to simplify things the
ID reuse behavior is gone and now vdev IDs for top-level vdevs
are truly unique within a pool.
= Testing
The illumos implementation of this feature has been used internally
for a year and has been in production for ~6 months. For this patch
specifically there don't seem to be any regressions introduced to
ZTS and I have been running zloop for a week without any related
problems.
= Performance Analysis (Linux Specific)
All performance results and analysis for illumos can be found in
the links of the references. Redoing the same experiments in Linux
gave similar results. Below are the specifics of the Linux run.
After the pool reached stable state the percentage of the time
spent in pass 1 per TXG was 64% on average for the stock bits
while the log spacemap bits stayed at 95% during the experiment
(graph: sdimitro.github.io/img/linux-lsm/PercOfSyncInPassOne.png).
Sync times per TXG were 37.6 seconds on average for the stock
bits and 22.7 seconds for the log spacemap bits (related graph:
sdimitro.github.io/img/linux-lsm/SyncTimePerTXG.png). As a result
the log spacemap bits were able to push more TXGs, which is also
the reason why all graphs quantified per TXG have more entries for
the log spacemap bits.
Another interesting aspect in terms of txg syncs is that the stock
bits had 22% of their TXGs reach sync pass 7, 55% reach sync pass 8,
and 20% reach 9. The log space map bits reached sync pass 4 in 79%
of their TXGs, sync pass 7 in 19%, and sync pass 8 at 1%. This
emphasizes the fact that not only we spend less time on metadata
but we also iterate less times to convergence in spa_sync() dirtying
objects.
[related graphs:
stock- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGStock.png
lsm- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGLSM.png]
Finally, the improvement in IOPs that the userland gains from the
change is approximately 40%. There is a consistent win in IOPS as
you can see from the graphs below but the absolute amount of
improvement that the log spacemap gives varies within each minute
interval.
sdimitro.github.io/img/linux-lsm/StockVsLog3Days.png
sdimitro.github.io/img/linux-lsm/StockVsLog10Hours.png
= Porting to Other Platforms
For people that want to port this commit to other platforms below
is a list of ZoL commits that this patch depends on:
Make zdb results for checkpoint tests consistent
db587941c5ff6dea01932bb78f70db63cf7f38ba
Update vdev_is_spacemap_addressable() for new spacemap encoding
419ba5914552c6185afbe1dd17b3ed4b0d526547
Simplify spa_sync by breaking it up to smaller functions
8dc2197b7b1e4d7ebc1420ea30e51c6541f1d834
Factor metaslab_load_wait() in metaslab_load()
b194fab0fb6caad18711abccaff3c69ad8b3f6d3
Rename range_tree_verify to range_tree_verify_not_present
df72b8bebe0ebac0b20e0750984bad182cb6564a
Change target size of metaslabs from 256GB to 16GB
c853f382db731e15a87512f4ef1101d14d778a55
zdb -L should skip leak detection altogether
21e7cf5da89f55ce98ec1115726b150e19eefe89
vs_alloc can underflow in L2ARC vdevs
7558997d2f808368867ca7e5234e5793446e8f3f
Simplify log vdev removal code
6c926f426a26ffb6d7d8e563e33fc176164175cb
Get rid of space_map_update() for ms_synced_length
425d3237ee88abc53d8522a7139c926d278b4b7f
Introduce auxiliary metaslab histograms
928e8ad47d3478a3d5d01f0dd6ae74a9371af65e
Error path in metaslab_load_impl() forgets to drop ms_sync_lock
8eef997679ba54547f7d361553d21b3291f41ae7
= References
Background, Motivation, and Internals of the Feature
- OpenZFS 2017 Presentation:
youtu.be/jj2IxRkl5bQ
- Slides:
slideshare.net/SerapheimNikolaosDim/zfs-log-spacemaps-project
Flushing Algorithm Internals & Performance Results
(Illumos Specific)
- Blogpost:
sdimitro.github.io/post/zfs-lsm-flushing/
- OpenZFS 2018 Presentation:
youtu.be/x6D2dHRjkxw
- Slides:
slideshare.net/SerapheimNikolaosDim/zfs-log-spacemap-flushing-algorithm
Upstream Delphix Issues:
DLPX-51539, DLPX-59659, DLPX-57783, DLPX-61438, DLPX-41227, DLPX-59320
DLPX-63385
Reviewed-by: Sean Eric Fagan <sef@ixsystems.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: George Wilson <gwilson@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8442
2019-07-16 17:11:49 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_default_ms_shift\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Default limit for metaslab size.
|
|
|
|
.sp
|
|
|
|
Default value: \fB29\fR [meaning (1 << 29) = 512MB].
|
|
|
|
.RE
|
|
|
|
|
2020-08-21 19:53:17 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_max_auto_ashift\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum ashift used when optimizing for logical -> physical sector size on new
|
|
|
|
top-level vdevs.
|
|
|
|
.sp
|
|
|
|
Default value: \fBASHIFT_MAX\fR (16).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_min_auto_ashift\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum ashift used when creating new top-level vdevs.
|
|
|
|
.sp
|
|
|
|
Default value: \fBASHIFT_MIN\fR (9).
|
|
|
|
.RE
|
|
|
|
|
2016-12-16 22:11:29 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2019-01-26 00:38:27 +00:00
|
|
|
\fBzfs_vdev_min_ms_count\fR (int)
|
2016-12-16 22:11:29 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum number of metaslabs to create in a top-level vdev.
|
|
|
|
.sp
|
|
|
|
Default value: \fB16\fR.
|
|
|
|
.RE
|
|
|
|
|
2017-08-11 21:28:17 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2020-01-07 20:41:53 +00:00
|
|
|
\fBvdev_validate_skip\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Skip label validation steps during pool import. Changing is not recommended
|
|
|
|
unless you know what you are doing and are recovering a damaged label.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_ms_count_limit\fR (int)
|
2017-08-11 21:28:17 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Practical upper limit of total metaslabs per top-level vdev.
|
|
|
|
.sp
|
|
|
|
Default value: \fB131,072\fR.
|
|
|
|
.RE
|
|
|
|
|
2014-07-19 20:19:24 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBmetaslab_preload_enabled\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Enable metaslab group preloading.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes (default) and \fB0\fR for no.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBmetaslab_lba_weighting_enabled\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Give more weight to metaslabs with lower LBAs, assuming they have
|
|
|
|
greater bandwidth as is typically the case on a modern constant
|
|
|
|
angular velocity disk drive.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes (default) and \fB0\fR for no.
|
|
|
|
.RE
|
|
|
|
|
2019-08-29 17:20:36 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBmetaslab_unload_delay\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
After a metaslab is used, we keep it loaded for this many txgs, to attempt to
|
|
|
|
reduce unnecessary reloading. Note that both this many txgs and
|
|
|
|
\fBmetaslab_unload_delay_ms\fR milliseconds must pass before unloading will
|
|
|
|
occur.
|
|
|
|
.sp
|
|
|
|
Default value: \fB32\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBmetaslab_unload_delay_ms\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
After a metaslab is used, we keep it loaded for this many milliseconds, to
|
|
|
|
attempt to reduce unnecessary reloading. Note that both this many
|
|
|
|
milliseconds and \fBmetaslab_unload_delay\fR txgs must pass before unloading
|
|
|
|
will occur.
|
|
|
|
.sp
|
|
|
|
Default value: \fB600000\fR (ten minutes).
|
|
|
|
.RE
|
|
|
|
|
2019-04-19 01:15:58 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBsend_holes_without_birth_time\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
When set, the hole_birth optimization will not be used, and all holes will
|
2019-04-21 20:12:30 +00:00
|
|
|
always be sent on zfs send. This is useful if you suspect your datasets are
|
|
|
|
affected by a bug in hole_birth.
|
2019-04-19 01:15:58 +00:00
|
|
|
.sp
|
|
|
|
Use \fB1\fR for on (default) and \fB0\fR for off.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBspa_config_path\fR (charp)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
SPA config file
|
|
|
|
.sp
|
|
|
|
Default value: \fB/etc/zfs/zpool.cache\fR.
|
|
|
|
.RE
|
|
|
|
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBspa_asize_inflation\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Multiplication factor used to estimate actual disk consumption from the
|
|
|
|
size of data being written. The default value is a worst case estimate,
|
|
|
|
but lower values may be valid for a given pool depending on its
|
|
|
|
configuration. Pool administrators who understand the factors involved
|
|
|
|
may wish to specify a more realistic inflation factor, particularly if
|
|
|
|
they operate close to quota or capacity limits.
|
|
|
|
.sp
|
2015-12-30 17:44:46 +00:00
|
|
|
Default value: \fB24\fR.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.RE
|
|
|
|
|
OpenZFS 9075 - Improve ZFS pool import/load process and corrupted pool recovery
Some work has been done lately to improve the debugability of the ZFS pool
load (and import) process. This includes:
7638 Refactor spa_load_impl into several functions
8961 SPA load/import should tell us why it failed
7277 zdb should be able to print zfs_dbgmsg's
To iterate on top of that, there's a few changes that were made to make the
import process more resilient and crash free. One of the first tasks during the
pool load process is to parse a config provided from userland that describes
what devices the pool is composed of. A vdev tree is generated from that config,
and then all the vdevs are opened.
The Meta Object Set (MOS) of the pool is accessed, and several metadata objects
that are necessary to load the pool are read. The exact configuration of the
pool is also stored inside the MOS. Since the configuration provided from
userland is external and might not accurately describe the vdev tree
of the pool at the txg that is being loaded, it cannot be relied upon to safely
operate the pool. For that reason, the configuration in the MOS is read early
on. In the past, the two configurations were compared together and if there was
a mismatch then the load process was aborted and an error was returned.
The latter was a good way to ensure a pool does not get corrupted, however it
made the pool load process needlessly fragile in cases where the vdev
configuration changed or the userland configuration was outdated. Since the MOS
is stored in 3 copies, the configuration provided by userland doesn't have to be
perfect in order to read its contents. Hence, a new approach has been adopted:
The pool is first opened with the untrusted userland configuration just so that
the real configuration can be read from the MOS. The trusted MOS configuration
is then used to generate a new vdev tree and the pool is re-opened.
When the pool is opened with an untrusted configuration, writes are disabled
to avoid accidentally damaging it. During reads, some sanity checks are
performed on block pointers to see if each DVA points to a known vdev;
when the configuration is untrusted, instead of panicking the system if those
checks fail we simply avoid issuing reads to the invalid DVAs.
This new two-step pool load process now allows rewinding pools accross
vdev tree changes such as device replacement, addition, etc. Loading a pool
from an external config file in a clustering environment also becomes much
safer now since the pool will import even if the config is outdated and didn't,
for instance, register a recent device addition.
With this code in place, it became relatively easy to implement a
long-sought-after feature: the ability to import a pool with missing top level
(i.e. non-redundant) devices. Note that since this almost guarantees some loss
of data, this feature is for now restricted to a read-only import.
Porting notes (ZTS):
* Fix 'make dist' target in zpool_import
* The maximum path length allowed by tar is 99 characters. Several
of the new test cases exceeded this limit resulting in them not
being included in the tarball. Shorten the names slightly.
* Set/get tunables using accessor functions.
* Get last synced txg via the "zfs_txg_history" mechanism.
* Clear zinject handlers in cleanup for import_cache_device_replaced
and import_rewind_device_replaced in order that the zpool can be
exported if there is an error.
* Increase FILESIZE to 8G in zfs-test.sh to allow for a larger
ext4 file system to be created on ZFS_DISK2. Also, there's
no need to partition ZFS_DISK2 at all. The partitioning had
already been disabled for multipath devices. Among other things,
the partitioning steals some space from the ext4 file system,
makes it difficult to accurately calculate the paramters to
parted and can make some of the tests fail.
* Increase FS_SIZE and FILE_SIZE in the zpool_import test
configuration now that FILESIZE is larger.
* Write more data in order that device evacuation take lonnger in
a couple tests.
* Use mkdir -p to avoid errors when the directory already exists.
* Remove use of sudo in import_rewind_config_changed.
Authored by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Andrew Stormont <andyjstormont@gmail.com>
Approved by: Hans Rosenfeld <rosenfeld@grumpf.hope-2000.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9075
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/619c0123
Closes #7459
2016-07-22 14:39:36 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBspa_load_print_vdev_tree\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Whether to print the vdev tree in the debugging message buffer during pool import.
|
|
|
|
Use 0 to disable and 1 to enable.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2014-07-15 18:58:41 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBspa_load_verify_data\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Whether to traverse data blocks during an "extreme rewind" (\fB-X\fR)
|
|
|
|
import. Use 0 to disable and 1 to enable.
|
|
|
|
|
|
|
|
An extreme rewind import normally performs a full traversal of all
|
|
|
|
blocks in the pool for verification. If this parameter is set to 0,
|
|
|
|
the traversal skips non-metadata blocks. It can be toggled once the
|
|
|
|
import has started to stop or start the traversal of non-metadata blocks.
|
|
|
|
.sp
|
2015-12-30 17:44:46 +00:00
|
|
|
Default value: \fB1\fR.
|
2014-07-15 18:58:41 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBspa_load_verify_metadata\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Whether to traverse blocks during an "extreme rewind" (\fB-X\fR)
|
|
|
|
pool import. Use 0 to disable and 1 to enable.
|
|
|
|
|
|
|
|
An extreme rewind import normally performs a full traversal of all
|
2016-03-28 22:13:42 +00:00
|
|
|
blocks in the pool for verification. If this parameter is set to 0,
|
2014-07-15 18:58:41 +00:00
|
|
|
the traversal is not performed. It can be toggled once the import has
|
|
|
|
started to stop or start the traversal.
|
|
|
|
.sp
|
2015-12-30 17:44:46 +00:00
|
|
|
Default value: \fB1\fR.
|
2014-07-15 18:58:41 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2019-08-13 14:11:57 +00:00
|
|
|
\fBspa_load_verify_shift\fR (int)
|
2014-07-15 18:58:41 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2019-08-13 14:11:57 +00:00
|
|
|
Sets the maximum number of bytes to consume during pool import to the log2
|
2020-04-10 17:33:35 +00:00
|
|
|
fraction of the target ARC size.
|
2014-07-15 18:58:41 +00:00
|
|
|
.sp
|
2019-08-13 14:11:57 +00:00
|
|
|
Default value: \fB4\fR.
|
2014-07-15 18:58:41 +00:00
|
|
|
.RE
|
|
|
|
|
2015-09-01 16:45:10 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBspa_slop_shift\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Normally, we don't allow the last 3.2% (1/(2^spa_slop_shift)) of space
|
|
|
|
in the pool to be consumed. This ensures that we don't run the pool
|
|
|
|
completely out of space, due to unaccounted changes (e.g. to the MOS).
|
|
|
|
It also limits the worst-case time to allocate space. If we have
|
|
|
|
less than this amount of free space, most ZPL operations (e.g. write,
|
|
|
|
create) will return ENOSPC.
|
|
|
|
.sp
|
2015-12-30 17:44:46 +00:00
|
|
|
Default value: \fB5\fR.
|
2015-09-01 16:45:10 +00:00
|
|
|
.RE
|
|
|
|
|
OpenZFS 9486 - reduce memory used by device removal on fragmented pools
Device removal allocates a new location for each allocated segment on
the disk that's being removed. Each allocation results in one entry in
the mapping table, which maps from old location + length to new
location. When a fragmented disk is removed, this can result in a large
number of mapping entries, and thus a large amount of memory consumed by
the mapping table. In the worst real-world cases, we've seen around 1GB
of RAM per 1TB of storage removed.
We can improve on this situation by allocating larger segments, which
span across both allocated and free regions of the device being removed.
By including free regions in the allocation (and thus mapping), we
reduce the number of mapping entries. For example, if we have a 4K
allocation followed by 1K free and then 4K allocated, we would allocate
4+1+4 = 9KB, and then move the entire region (including allocated and
free parts). In this case we used one mapping where previously we would
have used two, but often the ratio is much higher (up to 20:1 in
real-world use). We then need to mark the regions that were free on the
removing device as free in the new locations, and also obsolete in the
mapping entry.
This method preserves the fragmentation of the removing device, rather
than consolidating its allocated space into a small number of chunks
where possible. But it results in drastic reduction of memory used by
the mapping table - around 20x in the most-fragmented cases.
In the most fragmented real-world cases, this reduces memory used by the
mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less
fragmented cases will typically also see around 50-100MB of RAM per 1TB
of storage.
Porting notes:
* Add the following as module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* Document the following module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9486
OpenZFS-commit: https://github.com/ahrens/illumos/commit/07152e142e44c
External-issue: DLPX-57962
Closes #7536
2018-02-26 23:33:55 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBvdev_removal_max_span\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
During top-level vdev removal, chunks of data are copied from the vdev
|
|
|
|
which may include free space in order to trade bandwidth for IOPS.
|
|
|
|
This parameter determines the maximum span of free space (in bytes)
|
|
|
|
which will be included as "unnecessary" data in a chunk of copied data.
|
|
|
|
|
|
|
|
The default value here was chosen to align with
|
|
|
|
\fBzfs_vdev_read_gap_limit\fR, which is a similar concept when doing
|
|
|
|
regular reads (but there's no reason it has to be the same).
|
|
|
|
.sp
|
|
|
|
Default value: \fB32,768\fR.
|
|
|
|
.RE
|
|
|
|
|
2020-09-18 19:13:47 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBvdev_file_logical_ashift\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Logical ashift for file-based devices.
|
|
|
|
.sp
|
|
|
|
Default value: \fB9\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBvdev_file_physical_ashift\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Physical ashift for file-based devices.
|
|
|
|
.sp
|
|
|
|
Default value: \fB9\fR.
|
|
|
|
.RE
|
|
|
|
|
2019-06-12 20:13:09 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzap_iterate_prefetch\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
If this is set, when we start iterating over a ZAP object, zfs will prefetch
|
|
|
|
the entire object (all leaf blocks). However, this is limited by
|
|
|
|
\fBdmu_prefetch_max\fR.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for on (default) and \fB0\fR for off.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfetch_array_rd_sz\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2014-06-04 12:23:31 +00:00
|
|
|
If prefetching is enabled, disable prefetching for reads larger than this size.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB1,048,576\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2015-12-26 21:10:31 +00:00
|
|
|
\fBzfetch_max_distance\fR (uint)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2020-10-13 16:32:34 +00:00
|
|
|
Max bytes to prefetch per stream.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2020-10-13 16:32:34 +00:00
|
|
|
Default value: \fB8,388,608\fR (8MB).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfetch_max_idistance\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Max bytes to prefetch indirects for per stream.
|
|
|
|
.sp
|
|
|
|
Default vaule: \fB67,108,864\fR (64MB).
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfetch_max_streams\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2014-06-04 12:23:31 +00:00
|
|
|
Max number of streams per zfetch (prefetch streams per file).
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB8\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfetch_min_sec_reap\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2014-06-04 12:23:31 +00:00
|
|
|
Min time before an active prefetch stream can be reclaimed
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB2\fR.
|
|
|
|
.RE
|
|
|
|
|
2020-01-07 20:41:53 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_abd_scatter_enabled\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Enables ARC from using scatter/gather lists and forces all allocations to be
|
|
|
|
linear in kernel memory. Disabling can improve performance in some code paths
|
|
|
|
at the expense of fragmented kernel memory.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_abd_scatter_max_order\fR (iunt)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum number of consecutive memory pages allocated in a single block for
|
|
|
|
scatter/gather lists. Default value is specified by the kernel itself.
|
|
|
|
.sp
|
|
|
|
Default value: \fB10\fR at the time of this writing.
|
|
|
|
.RE
|
|
|
|
|
abd_alloc should use scatter for >1K allocations
abd_alloc() normally does scatter allocations, thus solving the problem
that ABD originally set out to: the bulk of ZFS's allocations are single
pages, which are faster to allocate and free, and don't suffer from
internal fragmentation (and the inability to reclaim memory because some
buffers in the slab are still allocated).
However, the current code does linear allocations for 4KB and smaller
allocations, defeating the purpose of ABD.
Scatter ABD's use at least one page each, so sub-page allocations waste
some space when allocated as scatter (e.g. 2KB scatter allocation wastes
half of each page). Using linear ABD's for small allocations means that
they will be put on slabs which contain many allocations. This can
improve memory efficiency, but it also makes it much harder for ARC
evictions to actually free pages, because all the buffers on one slab
need to be freed in order for the slab (and underlying pages) to be
freed. Typically, 512B and 1KB kmem caches have 16 buffers per slab, so
it's possible for them to actually waste more memory than scatter (one
page per buf = wasting 3/4 or 7/8th; one buf per slab = wasting
15/16th).
Spill blocks are typically 512B and are heavily used on systems running
selinux with the default dnode size and the `xattr=sa` property set.
By default we will use linear allocations for 512B and 1KB, and scatter
allocations for larger (1.5KB and up).
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: DHE <git@dehacked.net>
Reviewed-by: Chunwei Chen <tuxoko@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Don Brady <don.brady@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #8455
2019-03-01 01:52:55 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_abd_scatter_min_size\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
This is the minimum allocation size that will use scatter (page-based)
|
|
|
|
ABD's. Smaller allocations will use linear ABD's.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1536\fR (512B and 1KB allocations will be linear).
|
|
|
|
.RE
|
|
|
|
|
2016-07-13 12:42:40 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_dnode_limit\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
When the number of bytes consumed by dnodes in the ARC exceeds this number of
|
2016-08-11 03:15:37 +00:00
|
|
|
bytes, try to unpin some of it in response to demand for non-metadata. This
|
2017-06-14 20:23:02 +00:00
|
|
|
value acts as a ceiling to the amount of dnode metadata, and defaults to 0 which
|
2016-08-11 03:15:37 +00:00
|
|
|
indicates that a percent which is based on \fBzfs_arc_dnode_limit_percent\fR of
|
|
|
|
the ARC meta buffers that may be used for dnodes.
|
2016-07-13 12:42:40 +00:00
|
|
|
|
|
|
|
See also \fBzfs_arc_meta_prune\fR which serves a similar purpose but is used
|
|
|
|
when the amount of metadata in the ARC exceeds \fBzfs_arc_meta_limit\fR rather
|
|
|
|
than in response to overall demand for non-metadata.
|
|
|
|
|
|
|
|
.sp
|
2016-08-11 03:15:37 +00:00
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_dnode_limit_percent\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Percentage that can be consumed by dnodes of ARC meta buffers.
|
|
|
|
.sp
|
|
|
|
See also \fBzfs_arc_dnode_limit\fR which serves a similar purpose but has a
|
|
|
|
higher priority if set to nonzero value.
|
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB10\fR%.
|
2016-07-13 12:42:40 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_dnode_reduce_percent\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Percentage of ARC dnodes to try to scan in response to demand for non-metadata
|
2016-11-15 01:03:57 +00:00
|
|
|
when the number of bytes consumed by dnodes exceeds \fBzfs_arc_dnode_limit\fR.
|
2016-07-13 12:42:40 +00:00
|
|
|
|
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB10\fR% of the number of dnodes in the ARC.
|
2016-07-13 12:42:40 +00:00
|
|
|
.RE
|
|
|
|
|
2014-08-20 17:09:40 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_average_blocksize\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The ARC's buffer hash table is sized based on the assumption of an average
|
|
|
|
block size of \fBzfs_arc_average_blocksize\fR (default 8K). This works out
|
|
|
|
to roughly 1MB of hash table per 1GB of physical memory with 8-byte pointers.
|
|
|
|
For configurations with a known larger average block size this value can be
|
|
|
|
increased to reduce the memory footprint.
|
|
|
|
|
|
|
|
.sp
|
|
|
|
Default value: \fB8192\fR.
|
|
|
|
.RE
|
|
|
|
|
Revise ARC shrinker algorithm
The ARC shrinker callback `arc_shrinker_count/_scan()` is invoked by the
kernel's shrinker mechanism when the system is running low on free
pages. This happens via 2 code paths:
1. "direct reclaim": The system is attempting to allocate a page, but we
are low on memory. The ARC shrinker callback is invoked from the
page-allocation code path.
2. "indirect reclaim": kswapd notices that there aren't many free pages,
so it invokes the ARC shrinker callback.
In both cases, the kernel's shrinker code requests that the ARC shrinker
callback release some of its cache, and then it measures how many pages
were released. However, it's measurement of released pages does not
include pages that are freed via `__free_pages()`, which is how the ARC
releases memory (via `abd_free_chunks()`). Rather, the kernel shrinker
code is looking for pages to be placed on the lists of reclaimable pages
(which is separate from actually-free pages).
Because the kernel shrinker code doesn't detect that the ARC has
released pages, it may call the ARC shrinker callback many times,
resulting in the ARC "collapsing" down to `arc_c_min`. This has several
negative impacts:
1. ZFS doesn't use RAM to cache data effectively.
2. In the direct reclaim case, a single page allocation may wait a long
time (e.g. more than a minute) while we evict the entire ARC.
3. Even with the improvements made in 67c0f0dedc5 ("ARC shrinking blocks
reads/writes"), occasionally `arc_size` may stay above `arc_c` for the
entire time of the ARC collapse, thus blocking ZFS read/write operations
in `arc_get_data_impl()`.
To address these issues, this commit limits the ways that the ARC
shrinker callback can be used by the kernel shrinker code, and mitigates
the impact of arc_is_overflowing() on ZFS read/write operations.
With this commit:
1. We limit the amount of data that can be reclaimed from the ARC via
the "direct reclaim" shrinker. This limits the amount of time it takes
to allocate a single page.
2. We do not allow the ARC to shrink via kswapd (indirect reclaim).
Instead we rely on `arc_evict_zthr` to monitor free memory and reduce
the ARC target size to keep sufficient free memory in the system. Note
that we can't simply rely on limiting the amount that we reclaim at once
(as for the direct reclaim case), because kswapd's "boosted" logic can
invoke the callback an unlimited number of times (see
`balance_pgdat()`).
3. When `arc_is_overflowing()` and we want to allocate memory,
`arc_get_data_impl()` will wait only for a multiple of the requested
amount of data to be evicted, rather than waiting for the ARC to no
longer be overflowing. This allows ZFS reads/writes to make progress
even while the ARC is overflowing, while also ensuring that the eviction
thread makes progress towards reducing the total amount of memory used
by the ARC.
4. The amount of memory that the ARC always tries to keep free for the
rest of the system, `arc_sys_free` is increased.
5. Now that the shrinker callback is able to provide feedback to the
kernel's shrinker code about our progress, we can safely enable
the kswapd hook. This will allow the arc to receive notifications
when memory pressure is first detected by the kernel. We also
re-enable the appropriate kstats to track these callbacks.
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: George Wilson <george.wilson@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #10600
2020-08-01 04:10:52 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_eviction_pct\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
When \fBarc_is_overflowing()\fR, \fBarc_get_data_impl()\fR waits for this
|
|
|
|
percent of the requested amount of data to be evicted. For example, by
|
|
|
|
default for every 2KB that's evicted, 1KB of it may be "reused" by a new
|
|
|
|
allocation. Since this is above 100%, it ensures that progress is made
|
|
|
|
towards getting \fBarc_size\fR under \fBarc_c\fR. Since this is finite, it
|
|
|
|
ensures that allocations can still happen, even during the potentially long
|
|
|
|
time that \fBarc_size\fR is more than \fBarc_c\fR.
|
|
|
|
.sp
|
|
|
|
Default value: \fB200\fR.
|
|
|
|
.RE
|
|
|
|
|
2015-01-13 03:52:19 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_evict_batch_limit\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-07-01 11:42:35 +00:00
|
|
|
Number ARC headers to evict per sub-list before proceeding to another sub-list.
|
2015-01-13 03:52:19 +00:00
|
|
|
This batch-style operation prevents entire sub-lists from being evicted at once
|
|
|
|
but comes at a cost of additional unlocking and locking.
|
|
|
|
.sp
|
|
|
|
Default value: \fB10\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_grow_retry\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-10-30 20:15:10 +00:00
|
|
|
If set to a non zero value, it will replace the arc_grow_retry value with this value.
|
2017-11-16 01:27:01 +00:00
|
|
|
The arc_grow_retry value (default 5) is the number of seconds the ARC will wait before
|
2017-10-30 20:15:10 +00:00
|
|
|
trying to resume growth after a memory pressure event.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2017-10-30 20:15:10 +00:00
|
|
|
Default value: \fB0\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2015-07-28 18:30:00 +00:00
|
|
|
\fBzfs_arc_lotsfree_percent\fR (int)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-07-28 18:30:00 +00:00
|
|
|
Throttle I/O when free system memory drops below this percentage of total
|
|
|
|
system memory. Setting this value to 0 will disable the throttle.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB10\fR%.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2015-07-28 18:30:00 +00:00
|
|
|
\fBzfs_arc_max\fR (ulong)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2020-03-27 16:14:46 +00:00
|
|
|
Max size of ARC in bytes. If set to 0 then the max size of ARC is determined
|
|
|
|
by the amount of system memory installed. For Linux, 1/2 of system memory will
|
|
|
|
be used as the limit. For FreeBSD, the larger of all system memory - 1GB or
|
|
|
|
5/8 of system memory will be used as the limit. This value must be at least
|
|
|
|
67108864 (64 megabytes).
|
2015-12-30 17:44:46 +00:00
|
|
|
.sp
|
|
|
|
This value can be changed dynamically with some caveats. It cannot be set back
|
|
|
|
to 0 while running and reducing it below the current ARC size will not cause
|
|
|
|
the ARC to shrink without memory pressure to induce shrinking.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2015-07-28 18:30:00 +00:00
|
|
|
Default value: \fB0\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
2017-10-30 20:15:10 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_meta_adjust_restarts\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The number of restart passes to make while scanning the ARC attempting
|
|
|
|
the free buffers in order to stay below the \fBzfs_arc_meta_limit\fR.
|
|
|
|
This value should not need to be tuned but is available to facilitate
|
|
|
|
performance analysis.
|
|
|
|
.sp
|
|
|
|
Default value: \fB4096\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_meta_limit\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-03-17 22:07:47 +00:00
|
|
|
The maximum allowed size in bytes that meta data buffers are allowed to
|
|
|
|
consume in the ARC. When this limit is reached meta data buffers will
|
|
|
|
be reclaimed even if the overall arc_c_max has not been reached. This
|
2016-08-11 03:15:37 +00:00
|
|
|
value defaults to 0 which indicates that a percent which is based on
|
|
|
|
\fBzfs_arc_meta_limit_percent\fR of the ARC may be used for meta data.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2015-12-30 17:44:46 +00:00
|
|
|
This value my be changed dynamically except that it cannot be set back to 0
|
2016-08-11 03:15:37 +00:00
|
|
|
for a specific percent of the ARC; it must be set to an explicit value.
|
2015-12-30 17:44:46 +00:00
|
|
|
.sp
|
2013-11-16 06:52:54 +00:00
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2016-08-11 03:15:37 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_meta_limit_percent\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Percentage of ARC buffers that can be used for meta data.
|
|
|
|
|
|
|
|
See also \fBzfs_arc_meta_limit\fR which serves a similar purpose but has a
|
|
|
|
higher priority if set to nonzero value.
|
|
|
|
|
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB75\fR%.
|
2016-08-11 03:15:37 +00:00
|
|
|
.RE
|
|
|
|
|
2015-01-13 03:52:19 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_meta_min\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The minimum allowed size in bytes that meta data buffers may consume in
|
|
|
|
the ARC. This value defaults to 0 which disables a floor on the amount
|
|
|
|
of the ARC devoted meta data.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_meta_prune\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-03-17 22:07:47 +00:00
|
|
|
The number of dentries and inodes to be scanned looking for entries
|
|
|
|
which can be dropped. This may be required when the ARC reaches the
|
|
|
|
\fBzfs_arc_meta_limit\fR because dentries and inodes can pin buffers
|
|
|
|
in the ARC. Increasing this value will cause to dentry and inode caches
|
|
|
|
to be pruned more aggressively. Setting this value to 0 will disable
|
|
|
|
pruning the inode and dentry caches.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2015-03-17 22:07:47 +00:00
|
|
|
Default value: \fB10,000\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
2015-03-17 22:08:22 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2017-10-30 20:15:10 +00:00
|
|
|
\fBzfs_arc_meta_strategy\fR (int)
|
2015-03-17 22:08:22 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-10-30 20:15:10 +00:00
|
|
|
Define the strategy for ARC meta data buffer eviction (meta reclaim strategy).
|
|
|
|
A value of 0 (META_ONLY) will evict only the ARC meta data buffers.
|
2017-11-16 01:27:01 +00:00
|
|
|
A value of 1 (BALANCED) indicates that additional data buffers may be evicted if
|
2017-10-30 20:15:10 +00:00
|
|
|
that is required to in order to evict the required number of meta data buffers.
|
2015-03-17 22:08:22 +00:00
|
|
|
.sp
|
2017-10-30 20:15:10 +00:00
|
|
|
Default value: \fB1\fR.
|
2015-03-17 22:08:22 +00:00
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_min\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2020-04-10 17:33:35 +00:00
|
|
|
Min size of ARC in bytes. If set to 0 then arc_c_min will default to
|
2017-10-30 20:15:10 +00:00
|
|
|
consuming the larger of 32M or 1/32 of total system memory.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2017-10-30 20:15:10 +00:00
|
|
|
Default value: \fB0\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2017-11-16 01:27:01 +00:00
|
|
|
\fBzfs_arc_min_prefetch_ms\fR (int)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-11-16 01:27:01 +00:00
|
|
|
Minimum time prefetched blocks are locked in the ARC, specified in ms.
|
2018-02-06 00:57:53 +00:00
|
|
|
A value of \fB0\fR will default to 1000 ms.
|
2017-11-16 01:27:01 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_min_prescient_prefetch_ms\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum time "prescient prefetched" blocks are locked in the ARC, specified
|
2019-08-30 16:41:35 +00:00
|
|
|
in ms. These blocks are meant to be prefetched fairly aggressively ahead of
|
2018-02-06 00:57:53 +00:00
|
|
|
the code that may use them. A value of \fB0\fR will default to 6000 ms.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2015-12-30 17:44:46 +00:00
|
|
|
Default value: \fB0\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
OpenZFS 9075 - Improve ZFS pool import/load process and corrupted pool recovery
Some work has been done lately to improve the debugability of the ZFS pool
load (and import) process. This includes:
7638 Refactor spa_load_impl into several functions
8961 SPA load/import should tell us why it failed
7277 zdb should be able to print zfs_dbgmsg's
To iterate on top of that, there's a few changes that were made to make the
import process more resilient and crash free. One of the first tasks during the
pool load process is to parse a config provided from userland that describes
what devices the pool is composed of. A vdev tree is generated from that config,
and then all the vdevs are opened.
The Meta Object Set (MOS) of the pool is accessed, and several metadata objects
that are necessary to load the pool are read. The exact configuration of the
pool is also stored inside the MOS. Since the configuration provided from
userland is external and might not accurately describe the vdev tree
of the pool at the txg that is being loaded, it cannot be relied upon to safely
operate the pool. For that reason, the configuration in the MOS is read early
on. In the past, the two configurations were compared together and if there was
a mismatch then the load process was aborted and an error was returned.
The latter was a good way to ensure a pool does not get corrupted, however it
made the pool load process needlessly fragile in cases where the vdev
configuration changed or the userland configuration was outdated. Since the MOS
is stored in 3 copies, the configuration provided by userland doesn't have to be
perfect in order to read its contents. Hence, a new approach has been adopted:
The pool is first opened with the untrusted userland configuration just so that
the real configuration can be read from the MOS. The trusted MOS configuration
is then used to generate a new vdev tree and the pool is re-opened.
When the pool is opened with an untrusted configuration, writes are disabled
to avoid accidentally damaging it. During reads, some sanity checks are
performed on block pointers to see if each DVA points to a known vdev;
when the configuration is untrusted, instead of panicking the system if those
checks fail we simply avoid issuing reads to the invalid DVAs.
This new two-step pool load process now allows rewinding pools accross
vdev tree changes such as device replacement, addition, etc. Loading a pool
from an external config file in a clustering environment also becomes much
safer now since the pool will import even if the config is outdated and didn't,
for instance, register a recent device addition.
With this code in place, it became relatively easy to implement a
long-sought-after feature: the ability to import a pool with missing top level
(i.e. non-redundant) devices. Note that since this almost guarantees some loss
of data, this feature is for now restricted to a read-only import.
Porting notes (ZTS):
* Fix 'make dist' target in zpool_import
* The maximum path length allowed by tar is 99 characters. Several
of the new test cases exceeded this limit resulting in them not
being included in the tarball. Shorten the names slightly.
* Set/get tunables using accessor functions.
* Get last synced txg via the "zfs_txg_history" mechanism.
* Clear zinject handlers in cleanup for import_cache_device_replaced
and import_rewind_device_replaced in order that the zpool can be
exported if there is an error.
* Increase FILESIZE to 8G in zfs-test.sh to allow for a larger
ext4 file system to be created on ZFS_DISK2. Also, there's
no need to partition ZFS_DISK2 at all. The partitioning had
already been disabled for multipath devices. Among other things,
the partitioning steals some space from the ext4 file system,
makes it difficult to accurately calculate the paramters to
parted and can make some of the tests fail.
* Increase FS_SIZE and FILE_SIZE in the zpool_import test
configuration now that FILESIZE is larger.
* Write more data in order that device evacuation take lonnger in
a couple tests.
* Use mkdir -p to avoid errors when the directory already exists.
* Remove use of sudo in import_rewind_config_changed.
Authored by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Andrew Stormont <andyjstormont@gmail.com>
Approved by: Hans Rosenfeld <rosenfeld@grumpf.hope-2000.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9075
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/619c0123
Closes #7459
2016-07-22 14:39:36 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_max_missing_tvds\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Number of missing top-level vdevs which will be allowed during
|
|
|
|
pool import (only in read-only mode).
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR
|
|
|
|
.RE
|
|
|
|
|
2020-08-18 16:33:55 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_max_nvlist_src_size\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum size in bytes allowed to be passed as zc_nvlist_src_size for ioctls on
|
|
|
|
/dev/zfs. This prevents a user from causing the kernel to allocate an excessive
|
|
|
|
amount of memory. When the limit is exceeded, the ioctl fails with EINVAL and a
|
|
|
|
description of the error is sent to the zfs-dbgmsg log. This parameter should
|
|
|
|
not need to be touched under normal circumstances. On FreeBSD, the default is
|
|
|
|
based on the system limit on user wired memory. On Linux, the default is
|
|
|
|
\fBKMALLOC_MAX_SIZE\fR .
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR (kernel decides)
|
|
|
|
.RE
|
|
|
|
|
2015-01-13 03:52:19 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2017-02-15 23:49:33 +00:00
|
|
|
\fBzfs_multilist_num_sublists\fR (int)
|
2015-01-13 03:52:19 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
To allow more fine-grained locking, each ARC state contains a series
|
|
|
|
of lists for both data and meta data objects. Locking is performed at
|
|
|
|
the level of these "sub-lists". This parameters controls the number of
|
2017-02-15 23:49:33 +00:00
|
|
|
sub-lists per ARC state, and also applies to other uses of the
|
|
|
|
multilist data structure.
|
2015-01-13 03:52:19 +00:00
|
|
|
.sp
|
2017-02-15 23:49:33 +00:00
|
|
|
Default value: \fB4\fR or the number of online CPUs, whichever is greater
|
2015-01-13 03:52:19 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_overflow_shift\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The ARC size is considered to be overflowing if it exceeds the current
|
|
|
|
ARC target size (arc_c) by a threshold determined by this parameter.
|
|
|
|
The threshold is calculated as a fraction of arc_c using the formula
|
|
|
|
"arc_c >> \fBzfs_arc_overflow_shift\fR".
|
|
|
|
|
|
|
|
The default value of 8 causes the ARC to be considered to be overflowing
|
|
|
|
if it exceeds the target size by 1/256th (0.3%) of the target size.
|
|
|
|
|
|
|
|
When the ARC is overflowing, new buffer allocations are stalled until
|
|
|
|
the reclaim thread catches up and the overflow condition no longer exists.
|
|
|
|
.sp
|
|
|
|
Default value: \fB8\fR.
|
|
|
|
.RE
|
|
|
|
|
2015-06-26 22:59:23 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
|
|
|
|
\fBzfs_arc_p_min_shift\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-10-30 20:15:10 +00:00
|
|
|
If set to a non zero value, this will update arc_p_min_shift (default 4)
|
|
|
|
with the new value.
|
2017-11-16 01:27:01 +00:00
|
|
|
arc_p_min_shift is used to shift of arc_c for calculating both min and max
|
2017-10-30 20:15:10 +00:00
|
|
|
max arc_p
|
2015-06-26 22:59:23 +00:00
|
|
|
.sp
|
2017-10-30 20:15:10 +00:00
|
|
|
Default value: \fB0\fR.
|
2015-06-26 22:59:23 +00:00
|
|
|
.RE
|
|
|
|
|
2014-01-03 18:36:26 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_p_dampener_disable\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Disable arc_p adapt dampener
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes (default) and \fB0\fR to disable.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_shrink_shift\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-10-30 20:15:10 +00:00
|
|
|
If set to a non zero value, this will update arc_shrink_shift (default 7)
|
|
|
|
with the new value.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2017-10-30 20:15:10 +00:00
|
|
|
Default value: \fB0\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
2017-03-16 01:34:56 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_pc_percent\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Percent of pagecache to reclaim arc to
|
|
|
|
|
|
|
|
This tunable allows ZFS arc to play more nicely with the kernel's LRU
|
2020-04-10 17:33:35 +00:00
|
|
|
pagecache. It can guarantee that the ARC size won't collapse under scanning
|
2017-03-16 01:34:56 +00:00
|
|
|
pressure on the pagecache, yet still allows arc to be reclaimed down to
|
|
|
|
zfs_arc_min if necessary. This value is specified as percent of pagecache
|
|
|
|
size (as measured by NR_FILE_PAGES) where that percent may exceed 100. This
|
|
|
|
only operates during memory pressure/reclaim.
|
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB0\fR% (disabled).
|
2017-03-16 01:34:56 +00:00
|
|
|
.RE
|
|
|
|
|
Revise ARC shrinker algorithm
The ARC shrinker callback `arc_shrinker_count/_scan()` is invoked by the
kernel's shrinker mechanism when the system is running low on free
pages. This happens via 2 code paths:
1. "direct reclaim": The system is attempting to allocate a page, but we
are low on memory. The ARC shrinker callback is invoked from the
page-allocation code path.
2. "indirect reclaim": kswapd notices that there aren't many free pages,
so it invokes the ARC shrinker callback.
In both cases, the kernel's shrinker code requests that the ARC shrinker
callback release some of its cache, and then it measures how many pages
were released. However, it's measurement of released pages does not
include pages that are freed via `__free_pages()`, which is how the ARC
releases memory (via `abd_free_chunks()`). Rather, the kernel shrinker
code is looking for pages to be placed on the lists of reclaimable pages
(which is separate from actually-free pages).
Because the kernel shrinker code doesn't detect that the ARC has
released pages, it may call the ARC shrinker callback many times,
resulting in the ARC "collapsing" down to `arc_c_min`. This has several
negative impacts:
1. ZFS doesn't use RAM to cache data effectively.
2. In the direct reclaim case, a single page allocation may wait a long
time (e.g. more than a minute) while we evict the entire ARC.
3. Even with the improvements made in 67c0f0dedc5 ("ARC shrinking blocks
reads/writes"), occasionally `arc_size` may stay above `arc_c` for the
entire time of the ARC collapse, thus blocking ZFS read/write operations
in `arc_get_data_impl()`.
To address these issues, this commit limits the ways that the ARC
shrinker callback can be used by the kernel shrinker code, and mitigates
the impact of arc_is_overflowing() on ZFS read/write operations.
With this commit:
1. We limit the amount of data that can be reclaimed from the ARC via
the "direct reclaim" shrinker. This limits the amount of time it takes
to allocate a single page.
2. We do not allow the ARC to shrink via kswapd (indirect reclaim).
Instead we rely on `arc_evict_zthr` to monitor free memory and reduce
the ARC target size to keep sufficient free memory in the system. Note
that we can't simply rely on limiting the amount that we reclaim at once
(as for the direct reclaim case), because kswapd's "boosted" logic can
invoke the callback an unlimited number of times (see
`balance_pgdat()`).
3. When `arc_is_overflowing()` and we want to allocate memory,
`arc_get_data_impl()` will wait only for a multiple of the requested
amount of data to be evicted, rather than waiting for the ARC to no
longer be overflowing. This allows ZFS reads/writes to make progress
even while the ARC is overflowing, while also ensuring that the eviction
thread makes progress towards reducing the total amount of memory used
by the ARC.
4. The amount of memory that the ARC always tries to keep free for the
rest of the system, `arc_sys_free` is increased.
5. Now that the shrinker callback is able to provide feedback to the
kernel's shrinker code about our progress, we can safely enable
the kswapd hook. This will allow the arc to receive notifications
when memory pressure is first detected by the kernel. We also
re-enable the appropriate kstats to track these callbacks.
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: George Wilson <george.wilson@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #10600
2020-08-01 04:10:52 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_shrinker_limit\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
This is a limit on how many pages the ARC shrinker makes available for
|
|
|
|
eviction in response to one page allocation attempt. Note that in
|
|
|
|
practice, the kernel's shrinker can ask us to evict up to about 4x this
|
|
|
|
for one allocation attempt.
|
|
|
|
.sp
|
|
|
|
The default limit of 10,000 (in practice, 160MB per allocation attempt with
|
|
|
|
4K pages) limits the amount of time spent attempting to reclaim ARC memory to
|
|
|
|
less than 100ms per allocation attempt, even with a small average compressed
|
|
|
|
block size of ~8KB.
|
|
|
|
.sp
|
|
|
|
The parameter can be set to 0 (zero) to disable the limit.
|
|
|
|
.sp
|
|
|
|
This parameter only applies on Linux.
|
|
|
|
.sp
|
|
|
|
Default value: \fB10,000\fR.
|
|
|
|
.RE
|
|
|
|
|
2015-07-27 20:17:32 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_arc_sys_free\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The target number of bytes the ARC should leave as free memory on the system.
|
|
|
|
Defaults to the larger of 1/64 of physical memory or 512K. Setting this
|
|
|
|
option to a non-zero value will override the default.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_autoimport_disable\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2014-06-04 12:23:31 +00:00
|
|
|
Disable pool import at module load by ignoring the cache file (typically \fB/etc/zfs/zpool.cache\fR).
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2015-04-24 18:03:26 +00:00
|
|
|
Use \fB1\fR for yes (default) and \fB0\fR for no.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
2018-03-05 01:34:51 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2020-01-07 20:41:53 +00:00
|
|
|
\fBzfs_checksum_events_per_second\fR (uint)
|
2018-03-05 01:34:51 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Rate limit checksum events to this many per second. Note that this should
|
|
|
|
not be set below the zed thresholds (currently 10 checksums over 10 sec)
|
|
|
|
or else zed may not trigger any action.
|
|
|
|
.sp
|
|
|
|
Default value: 20
|
|
|
|
.RE
|
|
|
|
|
OpenZFS 8909 - 8585 can cause a use-after-free kernel panic
Authored by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: John Kennedy <jwk404@gmail.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Brad Lewis <brad.lewis@delphix.com>
Reviewed by: Igor Kozhukhov <igor@dilos.org>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Robert Mustacchi <rm@joyent.com>
Ported-by: Prakash Surya <prakash.surya@delphix.com>
PROBLEM
=======
There's a race condition that exists if `zil_free_lwb` races with either
`zil_commit_waiter_timeout` and/or `zil_lwb_flush_vdevs_done`.
Here's an example panic due to this bug:
> ::status
debugging crash dump vmcore.0 (64-bit) from ip-10-110-205-40
operating system: 5.11 dlpx-5.2.2.0_2017-12-04-17-28-32b6ba51fb (i86pc)
image uuid: 4af0edfb-e58e-6ed8-cafc-d3e9167c7513
panic message:
BAD TRAP: type=e (#pf Page fault) rp=ffffff0010555970 addr=60 occurred in module "zfs" due to a NULL pointer dereference
dump content: kernel pages only
> $c
zio_shrink+0x12()
zil_lwb_write_issue+0x30d(ffffff03dcd15cc0, ffffff03e0730e20)
zil_commit_waiter_timeout+0xa2(ffffff03dcd15cc0, ffffff03d97ffcf8)
zil_commit_waiter+0xf3(ffffff03dcd15cc0, ffffff03d97ffcf8)
zil_commit+0x80(ffffff03dcd15cc0, 9a9)
zfs_write+0xc34(ffffff03dc38b140, ffffff0010555e60, 40, ffffff03e00fb758, 0)
fop_write+0x5b(ffffff03dc38b140, ffffff0010555e60, 40, ffffff03e00fb758, 0)
write+0x250(42, fffffd7ff4832000, 2000)
sys_syscall+0x177()
If there's an outstanding lwb that's in `zil_commit_waiter_timeout`
waiting to timeout, waiting on it's waiter's CV, we must be sure not to
call `zil_free_lwb`. If we end up calling `zil_free_lwb`, then that LWB
may be freed and can result in a use-after-free situation where the
stale lwb pointer stored in the `zil_commit_waiter_t` structure of the
thread waiting on the waiter's CV is used.
A similar situation can occur if an lwb is issued to disk, and thus in
the `LWB_STATE_ISSUED` state, and `zil_free_lwb` is called while the
disk is servicing that lwb. In this situation, the lwb will be freed by
`zil_free_lwb`, which will result in a use-after-free situation when the
lwb's zio completes, and `zil_lwb_flush_vdevs_done` is called.
This race condition is prevented in `zil_close` by calling `zil_commit`
before `zil_free_lwb` is called, which will ensure all outstanding (i.e.
all lwb's in the `LWB_STATE_OPEN` and/or `LWB_STATE_ISSUED` states)
reach the `LWB_STATE_DONE` state before the lwb's are freed
(`zil_commit` will not return untill all the lwb's are
`LWB_STATE_DONE`).
Further, this race condition is prevented in `zil_sync` by only calling
`zil_free_lwb` for lwb's that do not have their `lwb_buf` pointer set.
All lwb's not in the `LWB_STATE_DONE` state will have a non-null value
for this pointer; the pointer is only cleared in
`zil_lwb_flush_vdevs_done`, at which point the lwb's state will be
changed to `LWB_STATE_DONE`.
This race *is* present in `zil_suspend`, leading to this bug.
At first glance, it would appear as though this would not be true
because `zil_suspend` will call `zil_commit`, just like `zil_close`, but
the problem is that `zil_suspend` will set the zilog's `zl_suspend`
field prior to calling `zil_commit`. Further, in `zil_commit`, if
`zl_suspend` is set, `zil_commit` will take a special branch of logic
and use `txg_wait_synced` instead of performing the normal `zil_commit`
logic.
This call to `txg_wait_synced` might be good enough for the data to
reach disk safely before it returns, but it does not ensure that all
outstanding lwb's reach the `LWB_STATE_DONE` state before it returns.
This is because, if there's an lwb "stuck" in
`zil_commit_waiter_timeout`, waiting for it's lwb to timeout, it will
maintain a non-null value for it's `lwb_buf` field and thus `zil_sync`
will not free that lwb. Thus, even though the lwb's data is already on
disk, the lwb will be left lingering, waiting on the CV, and will
eventually timeout and be issued to disk even though the write is
unnecessary.
So, after `zil_commit` is called from `zil_suspend`, we incorrectly
assume that there are not outstanding lwb's, and proceed to free all
lwb's found on the zilog's lwb list. As a result, we free the lwb that
will later be used `zil_commit_waiter_timeout`.
SOLUTION
========
The solution to this, is to ensure all outstanding lwb's complete before
calling `zil_free_lwb` via `zil_destroy` in `zil_suspend`. This patch
accomplishes this goal by forcing the normal `zil_commit` logic when
called from `zil_sync`.
Now, `zil_suspend` will call `zil_commit_impl` which will always use the
normal logic of waiting/issuing lwb's to disk before it returns. As a
result, any lwb's outstanding when `zil_commit_impl` is called will be
guaranteed to reach the `LWB_STATE_DONE` state by the time it returns.
Further, no new lwb's will be created via `zil_commit` since the zilog's
`zl_suspend` flag will be set. This will force all new callers of
`zil_commit` to use `txg_wait_synced` instead of creating and issuing
new lwb's.
Thus, all lwb's left on the zilog's lwb list when `zil_destroy` is
called will be in the `LWB_STATE_DONE` state, and we'll avoid this race
condition.
OpenZFS-issue: https://www.illumos.org/issues/8909
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/ece62b6f8d
Closes #6940
2017-12-07 19:26:32 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_commit_timeout_pct\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
This controls the amount of time that a ZIL block (lwb) will remain "open"
|
|
|
|
when it isn't "full", and it has a thread waiting for it to be committed to
|
|
|
|
stable storage. The timeout is scaled based on a percentage of the last lwb
|
|
|
|
latency to avoid significantly impacting the latency of each individual
|
|
|
|
transaction record (itx).
|
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB5\fR%.
|
OpenZFS 8909 - 8585 can cause a use-after-free kernel panic
Authored by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: John Kennedy <jwk404@gmail.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Brad Lewis <brad.lewis@delphix.com>
Reviewed by: Igor Kozhukhov <igor@dilos.org>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Robert Mustacchi <rm@joyent.com>
Ported-by: Prakash Surya <prakash.surya@delphix.com>
PROBLEM
=======
There's a race condition that exists if `zil_free_lwb` races with either
`zil_commit_waiter_timeout` and/or `zil_lwb_flush_vdevs_done`.
Here's an example panic due to this bug:
> ::status
debugging crash dump vmcore.0 (64-bit) from ip-10-110-205-40
operating system: 5.11 dlpx-5.2.2.0_2017-12-04-17-28-32b6ba51fb (i86pc)
image uuid: 4af0edfb-e58e-6ed8-cafc-d3e9167c7513
panic message:
BAD TRAP: type=e (#pf Page fault) rp=ffffff0010555970 addr=60 occurred in module "zfs" due to a NULL pointer dereference
dump content: kernel pages only
> $c
zio_shrink+0x12()
zil_lwb_write_issue+0x30d(ffffff03dcd15cc0, ffffff03e0730e20)
zil_commit_waiter_timeout+0xa2(ffffff03dcd15cc0, ffffff03d97ffcf8)
zil_commit_waiter+0xf3(ffffff03dcd15cc0, ffffff03d97ffcf8)
zil_commit+0x80(ffffff03dcd15cc0, 9a9)
zfs_write+0xc34(ffffff03dc38b140, ffffff0010555e60, 40, ffffff03e00fb758, 0)
fop_write+0x5b(ffffff03dc38b140, ffffff0010555e60, 40, ffffff03e00fb758, 0)
write+0x250(42, fffffd7ff4832000, 2000)
sys_syscall+0x177()
If there's an outstanding lwb that's in `zil_commit_waiter_timeout`
waiting to timeout, waiting on it's waiter's CV, we must be sure not to
call `zil_free_lwb`. If we end up calling `zil_free_lwb`, then that LWB
may be freed and can result in a use-after-free situation where the
stale lwb pointer stored in the `zil_commit_waiter_t` structure of the
thread waiting on the waiter's CV is used.
A similar situation can occur if an lwb is issued to disk, and thus in
the `LWB_STATE_ISSUED` state, and `zil_free_lwb` is called while the
disk is servicing that lwb. In this situation, the lwb will be freed by
`zil_free_lwb`, which will result in a use-after-free situation when the
lwb's zio completes, and `zil_lwb_flush_vdevs_done` is called.
This race condition is prevented in `zil_close` by calling `zil_commit`
before `zil_free_lwb` is called, which will ensure all outstanding (i.e.
all lwb's in the `LWB_STATE_OPEN` and/or `LWB_STATE_ISSUED` states)
reach the `LWB_STATE_DONE` state before the lwb's are freed
(`zil_commit` will not return untill all the lwb's are
`LWB_STATE_DONE`).
Further, this race condition is prevented in `zil_sync` by only calling
`zil_free_lwb` for lwb's that do not have their `lwb_buf` pointer set.
All lwb's not in the `LWB_STATE_DONE` state will have a non-null value
for this pointer; the pointer is only cleared in
`zil_lwb_flush_vdevs_done`, at which point the lwb's state will be
changed to `LWB_STATE_DONE`.
This race *is* present in `zil_suspend`, leading to this bug.
At first glance, it would appear as though this would not be true
because `zil_suspend` will call `zil_commit`, just like `zil_close`, but
the problem is that `zil_suspend` will set the zilog's `zl_suspend`
field prior to calling `zil_commit`. Further, in `zil_commit`, if
`zl_suspend` is set, `zil_commit` will take a special branch of logic
and use `txg_wait_synced` instead of performing the normal `zil_commit`
logic.
This call to `txg_wait_synced` might be good enough for the data to
reach disk safely before it returns, but it does not ensure that all
outstanding lwb's reach the `LWB_STATE_DONE` state before it returns.
This is because, if there's an lwb "stuck" in
`zil_commit_waiter_timeout`, waiting for it's lwb to timeout, it will
maintain a non-null value for it's `lwb_buf` field and thus `zil_sync`
will not free that lwb. Thus, even though the lwb's data is already on
disk, the lwb will be left lingering, waiting on the CV, and will
eventually timeout and be issued to disk even though the write is
unnecessary.
So, after `zil_commit` is called from `zil_suspend`, we incorrectly
assume that there are not outstanding lwb's, and proceed to free all
lwb's found on the zilog's lwb list. As a result, we free the lwb that
will later be used `zil_commit_waiter_timeout`.
SOLUTION
========
The solution to this, is to ensure all outstanding lwb's complete before
calling `zil_free_lwb` via `zil_destroy` in `zil_suspend`. This patch
accomplishes this goal by forcing the normal `zil_commit` logic when
called from `zil_sync`.
Now, `zil_suspend` will call `zil_commit_impl` which will always use the
normal logic of waiting/issuing lwb's to disk before it returns. As a
result, any lwb's outstanding when `zil_commit_impl` is called will be
guaranteed to reach the `LWB_STATE_DONE` state by the time it returns.
Further, no new lwb's will be created via `zil_commit` since the zilog's
`zl_suspend` flag will be set. This will force all new callers of
`zil_commit` to use `txg_wait_synced` instead of creating and issuing
new lwb's.
Thus, all lwb's left on the zilog's lwb list when `zil_destroy` is
called will be in the `LWB_STATE_DONE` state, and we'll avoid this race
condition.
OpenZFS-issue: https://www.illumos.org/issues/8909
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/ece62b6f8d
Closes #6940
2017-12-07 19:26:32 +00:00
|
|
|
.RE
|
|
|
|
|
2020-01-07 20:41:53 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_condense_indirect_commit_entry_delay_ms\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Vdev indirection layer (used for device removal) sleeps for this many
|
|
|
|
milliseconds during mapping generation. Intended for use with the test suite
|
|
|
|
to throttle vdev removal speed.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR (no throttle).
|
|
|
|
.RE
|
|
|
|
|
OpenZFS 9486 - reduce memory used by device removal on fragmented pools
Device removal allocates a new location for each allocated segment on
the disk that's being removed. Each allocation results in one entry in
the mapping table, which maps from old location + length to new
location. When a fragmented disk is removed, this can result in a large
number of mapping entries, and thus a large amount of memory consumed by
the mapping table. In the worst real-world cases, we've seen around 1GB
of RAM per 1TB of storage removed.
We can improve on this situation by allocating larger segments, which
span across both allocated and free regions of the device being removed.
By including free regions in the allocation (and thus mapping), we
reduce the number of mapping entries. For example, if we have a 4K
allocation followed by 1K free and then 4K allocated, we would allocate
4+1+4 = 9KB, and then move the entire region (including allocated and
free parts). In this case we used one mapping where previously we would
have used two, but often the ratio is much higher (up to 20:1 in
real-world use). We then need to mark the regions that were free on the
removing device as free in the new locations, and also obsolete in the
mapping entry.
This method preserves the fragmentation of the removing device, rather
than consolidating its allocated space into a small number of chunks
where possible. But it results in drastic reduction of memory used by
the mapping table - around 20x in the most-fragmented cases.
In the most fragmented real-world cases, this reduces memory used by the
mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less
fragmented cases will typically also see around 50-100MB of RAM per 1TB
of storage.
Porting notes:
* Add the following as module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* Document the following module parameters:
* zfs_condense_indirect_vdevs_enable
* zfs_condense_max_obsolete_bytes
* zfs_condense_min_mapping_bytes
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9486
OpenZFS-commit: https://github.com/ahrens/illumos/commit/07152e142e44c
External-issue: DLPX-57962
Closes #7536
2018-02-26 23:33:55 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_condense_indirect_vdevs_enable\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Enable condensing indirect vdev mappings. When set to a non-zero value,
|
|
|
|
attempt to condense indirect vdev mappings if the mapping uses more than
|
|
|
|
\fBzfs_condense_min_mapping_bytes\fR bytes of memory and if the obsolete
|
|
|
|
space map object uses more than \fBzfs_condense_max_obsolete_bytes\fR
|
|
|
|
bytes on-disk. The condensing process is an attempt to save memory by
|
|
|
|
removing obsolete mappings.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_condense_max_obsolete_bytes\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Only attempt to condense indirect vdev mappings if the on-disk size
|
|
|
|
of the obsolete space map object is greater than this number of bytes
|
|
|
|
(see \fBfBzfs_condense_indirect_vdevs_enable\fR).
|
|
|
|
.sp
|
|
|
|
Default value: \fB1,073,741,824\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_condense_min_mapping_bytes\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum size vdev mapping to attempt to condense (see
|
|
|
|
\fBzfs_condense_indirect_vdevs_enable\fR).
|
|
|
|
.sp
|
|
|
|
Default value: \fB131,072\fR.
|
|
|
|
.RE
|
|
|
|
|
2015-09-01 20:19:10 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_dbgmsg_enable\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Internally ZFS keeps a small log to facilitate debugging. By default the log
|
|
|
|
is disabled, to enable it set this option to 1. The contents of the log can
|
|
|
|
be accessed by reading the /proc/spl/kstat/zfs/dbgmsg file. Writing 0 to
|
|
|
|
this proc file clears the log.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_dbgmsg_maxsize\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The maximum size in bytes of the internal ZFS debug log.
|
|
|
|
.sp
|
|
|
|
Default value: \fB4M\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_dbuf_state_index\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
This feature is currently unused. It is normally used for controlling what
|
|
|
|
reporting is available under /proc/spl/kstat/zfs.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_deadman_enabled\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-01-31 22:19:08 +00:00
|
|
|
When a pool sync operation takes longer than \fBzfs_deadman_synctime_ms\fR
|
2017-12-18 22:06:07 +00:00
|
|
|
milliseconds, or when an individual I/O takes longer than
|
|
|
|
\fBzfs_deadman_ziotime_ms\fR milliseconds, then the operation is considered to
|
|
|
|
be "hung". If \fBzfs_deadman_enabled\fR is set then the deadman behavior is
|
|
|
|
invoked as described by the \fBzfs_deadman_failmode\fR module option.
|
|
|
|
By default the deadman is enabled and configured to \fBwait\fR which results
|
|
|
|
in "hung" I/Os only being logged. The deadman is automatically disabled
|
|
|
|
when a pool gets suspended.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2017-12-18 22:06:07 +00:00
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_deadman_failmode\fR (charp)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Controls the failure behavior when the deadman detects a "hung" I/O. Valid
|
|
|
|
values are \fBwait\fR, \fBcontinue\fR, and \fBpanic\fR.
|
|
|
|
.sp
|
|
|
|
\fBwait\fR - Wait for a "hung" I/O to complete. For each "hung" I/O a
|
|
|
|
"deadman" event will be posted describing that I/O.
|
|
|
|
.sp
|
|
|
|
\fBcontinue\fR - Attempt to recover from a "hung" I/O by re-dispatching it
|
|
|
|
to the I/O pipeline if possible.
|
|
|
|
.sp
|
|
|
|
\fBpanic\fR - Panic the system. This can be used to facilitate an automatic
|
|
|
|
fail-over to a properly configured fail-over partner.
|
|
|
|
.sp
|
|
|
|
Default value: \fBwait\fR.
|
2017-01-31 22:19:08 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_deadman_checktime_ms\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-12-18 22:06:07 +00:00
|
|
|
Check time in milliseconds. This defines the frequency at which we check
|
|
|
|
for hung I/O and potentially invoke the \fBzfs_deadman_failmode\fR behavior.
|
2017-01-31 22:19:08 +00:00
|
|
|
.sp
|
2017-12-18 22:06:07 +00:00
|
|
|
Default value: \fB60,000\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
\fBzfs_deadman_synctime_ms\fR (ulong)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-01-31 22:19:08 +00:00
|
|
|
Interval in milliseconds after which the deadman is triggered and also
|
2017-12-18 22:06:07 +00:00
|
|
|
the interval after which a pool sync operation is considered to be "hung".
|
|
|
|
Once this limit is exceeded the deadman will be invoked every
|
|
|
|
\fBzfs_deadman_checktime_ms\fR milliseconds until the pool sync completes.
|
|
|
|
.sp
|
|
|
|
Default value: \fB600,000\fR.
|
|
|
|
.RE
|
2017-01-31 22:19:08 +00:00
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2017-12-18 22:06:07 +00:00
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_deadman_ziotime_ms\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Interval in milliseconds after which the deadman is triggered and an
|
2018-11-09 00:47:24 +00:00
|
|
|
individual I/O operation is considered to be "hung". As long as the I/O
|
2017-12-18 22:06:07 +00:00
|
|
|
remains "hung" the deadman will be invoked every \fBzfs_deadman_checktime_ms\fR
|
|
|
|
milliseconds until the I/O completes.
|
|
|
|
.sp
|
|
|
|
Default value: \fB300,000\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_dedup_prefetch\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Enable prefetching dedup-ed blks
|
|
|
|
.sp
|
2014-08-30 02:13:26 +00:00
|
|
|
Use \fB1\fR for yes and \fB0\fR to disable (default).
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_delay_min_dirty_percent\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Start to delay each transaction once there is this amount of dirty data,
|
|
|
|
expressed as a percentage of \fBzfs_dirty_data_max\fR.
|
|
|
|
This value should be >= zfs_vdev_async_write_active_max_dirty_percent.
|
|
|
|
See the section "ZFS TRANSACTION DELAY".
|
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB60\fR%.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_delay_scale\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
This controls how quickly the transaction delay approaches infinity.
|
|
|
|
Larger values cause longer delays for a given amount of dirty data.
|
|
|
|
.sp
|
|
|
|
For the smoothest delay, this value should be about 1 billion divided
|
|
|
|
by the maximum number of operations per second. This will smoothly
|
|
|
|
handle between 10x and 1/10th this number.
|
|
|
|
.sp
|
|
|
|
See the section "ZFS TRANSACTION DELAY".
|
|
|
|
.sp
|
|
|
|
Note: \fBzfs_delay_scale\fR * \fBzfs_dirty_data_max\fR must be < 2^64.
|
|
|
|
.sp
|
|
|
|
Default value: \fB500,000\fR.
|
|
|
|
.RE
|
|
|
|
|
2020-01-07 20:41:53 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_disable_ivset_guid_check\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Disables requirement for IVset guids to be present and match when doing a raw
|
|
|
|
receive of encrypted datasets. Intended for users whose pools were created with
|
2020-10-09 03:10:13 +00:00
|
|
|
OpenZFS pre-release versions and now have compatibility issues.
|
2020-01-07 20:41:53 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_key_max_salt_uses\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum number of uses of a single salt value before generating a new one for
|
|
|
|
encrypted datasets. The default value is also the maximum that will be
|
|
|
|
accepted.
|
|
|
|
.sp
|
|
|
|
Default value: \fB400,000,000\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_object_mutex_size\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Size of the znode hashtable used for holds.
|
|
|
|
|
|
|
|
Due to the need to hold locks on objects that may not exist yet, kernel mutexes
|
|
|
|
are not created per-object and instead a hashtable is used where collisions
|
|
|
|
will result in objects waiting when there is not actually contention on the
|
|
|
|
same object.
|
|
|
|
.sp
|
|
|
|
Default value: \fB64\fR.
|
|
|
|
.RE
|
|
|
|
|
2018-03-05 01:34:51 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2018-11-26 18:23:58 +00:00
|
|
|
\fBzfs_slow_io_events_per_second\fR (int)
|
2018-03-05 01:34:51 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2018-11-09 00:47:24 +00:00
|
|
|
Rate limit delay zevents (which report slow I/Os) to this many per second.
|
2018-03-05 01:34:51 +00:00
|
|
|
.sp
|
|
|
|
Default value: 20
|
|
|
|
.RE
|
|
|
|
|
Log Spacemap Project
= Motivation
At Delphix we've seen a lot of customer systems where fragmentation
is over 75% and random writes take a performance hit because a lot
of time is spend on I/Os that update on-disk space accounting metadata.
Specifically, we seen cases where 20% to 40% of sync time is spend
after sync pass 1 and ~30% of the I/Os on the system is spent updating
spacemaps.
The problem is that these pools have existed long enough that we've
touched almost every metaslab at least once, and random writes
scatter frees across all metaslabs every TXG, thus appending to
their spacemaps and resulting in many I/Os. To give an example,
assuming that every VDEV has 200 metaslabs and our writes fit within
a single spacemap block (generally 4K) we have 200 I/Os. Then if we
assume 2 levels of indirection, we need 400 additional I/Os and
since we are talking about metadata for which we keep 2 extra copies
for redundancy we need to triple that number, leading to a total of
1800 I/Os per VDEV every TXG.
We could try and decrease the number of metaslabs so we have less
I/Os per TXG but then each metaslab would cover a wider range on
disk and thus would take more time to be loaded in memory from disk.
In addition, after it's loaded, it's range tree would consume more
memory.
Another idea would be to just increase the spacemap block size
which would allow us to fit more entries within an I/O block
resulting in fewer I/Os per metaslab and a speedup in loading time.
The problem is still that we don't deal with the number of I/Os
going up as the number of metaslabs is increasing and the fact
is that we generally write a lot to a few metaslabs and a little
to the rest of them. Thus, just increasing the block size would
actually waste bandwidth because we won't be utilizing our bigger
block size.
= About this patch
This patch introduces the Log Spacemap project which provides the
solution to the above problem while taking into account all the
aforementioned tradeoffs. The details on how it achieves that can
be found in the references sections below and in the code (see
Big Theory Statement in spa_log_spacemap.c).
Even though the change is fairly constraint within the metaslab
and lower-level SPA codepaths, there is a side-change that is
user-facing. The change is that VDEV IDs from VDEV holes will no
longer be reused. To give some background and reasoning for this,
when a log device is removed and its VDEV structure was replaced
with a hole (or was compacted; if at the end of the vdev array),
its vdev_id could be reused by devices added after that. Now
with the pool-wide space maps recording the vdev ID, this behavior
can cause problems (e.g. is this entry referring to a segment in
the new vdev or the removed log?). Thus, to simplify things the
ID reuse behavior is gone and now vdev IDs for top-level vdevs
are truly unique within a pool.
= Testing
The illumos implementation of this feature has been used internally
for a year and has been in production for ~6 months. For this patch
specifically there don't seem to be any regressions introduced to
ZTS and I have been running zloop for a week without any related
problems.
= Performance Analysis (Linux Specific)
All performance results and analysis for illumos can be found in
the links of the references. Redoing the same experiments in Linux
gave similar results. Below are the specifics of the Linux run.
After the pool reached stable state the percentage of the time
spent in pass 1 per TXG was 64% on average for the stock bits
while the log spacemap bits stayed at 95% during the experiment
(graph: sdimitro.github.io/img/linux-lsm/PercOfSyncInPassOne.png).
Sync times per TXG were 37.6 seconds on average for the stock
bits and 22.7 seconds for the log spacemap bits (related graph:
sdimitro.github.io/img/linux-lsm/SyncTimePerTXG.png). As a result
the log spacemap bits were able to push more TXGs, which is also
the reason why all graphs quantified per TXG have more entries for
the log spacemap bits.
Another interesting aspect in terms of txg syncs is that the stock
bits had 22% of their TXGs reach sync pass 7, 55% reach sync pass 8,
and 20% reach 9. The log space map bits reached sync pass 4 in 79%
of their TXGs, sync pass 7 in 19%, and sync pass 8 at 1%. This
emphasizes the fact that not only we spend less time on metadata
but we also iterate less times to convergence in spa_sync() dirtying
objects.
[related graphs:
stock- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGStock.png
lsm- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGLSM.png]
Finally, the improvement in IOPs that the userland gains from the
change is approximately 40%. There is a consistent win in IOPS as
you can see from the graphs below but the absolute amount of
improvement that the log spacemap gives varies within each minute
interval.
sdimitro.github.io/img/linux-lsm/StockVsLog3Days.png
sdimitro.github.io/img/linux-lsm/StockVsLog10Hours.png
= Porting to Other Platforms
For people that want to port this commit to other platforms below
is a list of ZoL commits that this patch depends on:
Make zdb results for checkpoint tests consistent
db587941c5ff6dea01932bb78f70db63cf7f38ba
Update vdev_is_spacemap_addressable() for new spacemap encoding
419ba5914552c6185afbe1dd17b3ed4b0d526547
Simplify spa_sync by breaking it up to smaller functions
8dc2197b7b1e4d7ebc1420ea30e51c6541f1d834
Factor metaslab_load_wait() in metaslab_load()
b194fab0fb6caad18711abccaff3c69ad8b3f6d3
Rename range_tree_verify to range_tree_verify_not_present
df72b8bebe0ebac0b20e0750984bad182cb6564a
Change target size of metaslabs from 256GB to 16GB
c853f382db731e15a87512f4ef1101d14d778a55
zdb -L should skip leak detection altogether
21e7cf5da89f55ce98ec1115726b150e19eefe89
vs_alloc can underflow in L2ARC vdevs
7558997d2f808368867ca7e5234e5793446e8f3f
Simplify log vdev removal code
6c926f426a26ffb6d7d8e563e33fc176164175cb
Get rid of space_map_update() for ms_synced_length
425d3237ee88abc53d8522a7139c926d278b4b7f
Introduce auxiliary metaslab histograms
928e8ad47d3478a3d5d01f0dd6ae74a9371af65e
Error path in metaslab_load_impl() forgets to drop ms_sync_lock
8eef997679ba54547f7d361553d21b3291f41ae7
= References
Background, Motivation, and Internals of the Feature
- OpenZFS 2017 Presentation:
youtu.be/jj2IxRkl5bQ
- Slides:
slideshare.net/SerapheimNikolaosDim/zfs-log-spacemaps-project
Flushing Algorithm Internals & Performance Results
(Illumos Specific)
- Blogpost:
sdimitro.github.io/post/zfs-lsm-flushing/
- OpenZFS 2018 Presentation:
youtu.be/x6D2dHRjkxw
- Slides:
slideshare.net/SerapheimNikolaosDim/zfs-log-spacemap-flushing-algorithm
Upstream Delphix Issues:
DLPX-51539, DLPX-59659, DLPX-57783, DLPX-61438, DLPX-41227, DLPX-59320
DLPX-63385
Reviewed-by: Sean Eric Fagan <sef@ixsystems.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: George Wilson <gwilson@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8442
2019-07-16 17:11:49 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_unflushed_max_mem_amt\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Upper-bound limit for unflushed metadata changes to be held by the
|
|
|
|
log spacemap in memory (in bytes).
|
|
|
|
.sp
|
|
|
|
Default value: \fB1,073,741,824\fR (1GB).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_unflushed_max_mem_ppm\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Percentage of the overall system memory that ZFS allows to be used
|
|
|
|
for unflushed metadata changes by the log spacemap.
|
|
|
|
(value is calculated over 1000000 for finer granularity).
|
|
|
|
.sp
|
|
|
|
Default value: \fB1000\fR (which is divided by 1000000, resulting in
|
|
|
|
the limit to be \fB0.1\fR% of memory)
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_unflushed_log_block_max\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Describes the maximum number of log spacemap blocks allowed for each pool.
|
|
|
|
The default value of 262144 means that the space in all the log spacemaps
|
|
|
|
can add up to no more than 262144 blocks (which means 32GB of logical
|
|
|
|
space before compression and ditto blocks, assuming that blocksize is
|
|
|
|
128k).
|
|
|
|
.sp
|
|
|
|
This tunable is important because it involves a trade-off between import
|
|
|
|
time after an unclean export and the frequency of flushing metaslabs.
|
|
|
|
The higher this number is, the more log blocks we allow when the pool is
|
|
|
|
active which means that we flush metaslabs less often and thus decrease
|
|
|
|
the number of I/Os for spacemap updates per TXG.
|
|
|
|
At the same time though, that means that in the event of an unclean export,
|
|
|
|
there will be more log spacemap blocks for us to read, inducing overhead
|
|
|
|
in the import time of the pool.
|
|
|
|
The lower the number, the amount of flushing increases destroying log
|
|
|
|
blocks quicker as they become obsolete faster, which leaves less blocks
|
|
|
|
to be read during import time after a crash.
|
|
|
|
.sp
|
|
|
|
Each log spacemap block existing during pool import leads to approximately
|
|
|
|
one extra logical I/O issued.
|
|
|
|
This is the reason why this tunable is exposed in terms of blocks rather
|
|
|
|
than space used.
|
|
|
|
.sp
|
|
|
|
Default value: \fB262144\fR (256K).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_unflushed_log_block_min\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
If the number of metaslabs is small and our incoming rate is high, we
|
|
|
|
could get into a situation that we are flushing all our metaslabs every
|
|
|
|
TXG.
|
|
|
|
Thus we always allow at least this many log blocks.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1000\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_unflushed_log_block_pct\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Tunable used to determine the number of blocks that can be used for
|
|
|
|
the spacemap log, expressed as a percentage of the total number of
|
|
|
|
metaslabs in the pool.
|
|
|
|
.sp
|
|
|
|
Default value: \fB400\fR (read as \fB400\fR% - meaning that the number
|
|
|
|
of log spacemap blocks are capped at 4 times the number of
|
|
|
|
metaslabs in the pool).
|
|
|
|
.RE
|
|
|
|
|
2019-02-12 18:41:15 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_unlink_suspend_progress\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
When enabled, files will not be asynchronously removed from the list of pending
|
|
|
|
unlinks and the space they consume will be leaked. Once this option has been
|
|
|
|
disabled and the dataset is remounted, the pending unlinks will be processed
|
|
|
|
and the freed space returned to the pool.
|
|
|
|
This option is used by the test suite to facilitate testing.
|
|
|
|
.sp
|
|
|
|
Uses \fB0\fR (default) to allow progress and \fB1\fR to pause progress.
|
|
|
|
.RE
|
|
|
|
|
2015-08-21 01:43:10 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_delete_blocks\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
This is the used to define a large file for the purposes of delete. Files
|
|
|
|
containing more than \fBzfs_delete_blocks\fR will be deleted asynchronously
|
|
|
|
while smaller files are deleted synchronously. Decreasing this value will
|
|
|
|
reduce the time spent in an unlink(2) system call at the expense of a longer
|
|
|
|
delay before the freed space is available.
|
|
|
|
.sp
|
|
|
|
Default value: \fB20,480\fR.
|
|
|
|
.RE
|
|
|
|
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_dirty_data_max\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Determines the dirty space limit in bytes. Once this limit is exceeded, new
|
|
|
|
writes are halted until space frees up. This parameter takes precedence
|
|
|
|
over \fBzfs_dirty_data_max_percent\fR.
|
|
|
|
See the section "ZFS TRANSACTION DELAY".
|
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB10\fR% of physical RAM, capped at \fBzfs_dirty_data_max_max\fR.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_dirty_data_max_max\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum allowable value of \fBzfs_dirty_data_max\fR, expressed in bytes.
|
|
|
|
This limit is only enforced at module load time, and will be ignored if
|
|
|
|
\fBzfs_dirty_data_max\fR is later changed. This parameter takes
|
|
|
|
precedence over \fBzfs_dirty_data_max_max_percent\fR. See the section
|
|
|
|
"ZFS TRANSACTION DELAY".
|
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB25\fR% of physical RAM.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_dirty_data_max_max_percent\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum allowable value of \fBzfs_dirty_data_max\fR, expressed as a
|
|
|
|
percentage of physical RAM. This limit is only enforced at module load
|
|
|
|
time, and will be ignored if \fBzfs_dirty_data_max\fR is later changed.
|
|
|
|
The parameter \fBzfs_dirty_data_max_max\fR takes precedence over this
|
|
|
|
one. See the section "ZFS TRANSACTION DELAY".
|
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB25\fR%.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_dirty_data_max_percent\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Determines the dirty space limit, expressed as a percentage of all
|
|
|
|
memory. Once this limit is exceeded, new writes are halted until space frees
|
|
|
|
up. The parameter \fBzfs_dirty_data_max\fR takes precedence over this
|
|
|
|
one. See the section "ZFS TRANSACTION DELAY".
|
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB10\fR%, subject to \fBzfs_dirty_data_max_max\fR.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2017-12-12 23:46:58 +00:00
|
|
|
\fBzfs_dirty_data_sync_percent\fR (int)
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-12-12 23:46:58 +00:00
|
|
|
Start syncing out a transaction group if there's at least this much dirty data
|
|
|
|
as a percentage of \fBzfs_dirty_data_max\fR. This should be less than
|
|
|
|
\fBzfs_vdev_async_write_active_min_dirty_percent\fR.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.sp
|
2017-12-12 23:46:58 +00:00
|
|
|
Default value: \fB20\fR% of \fBzfs_dirty_data_max\fR.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.RE
|
|
|
|
|
linux: add basic fallocate(mode=0/2) compatibility
Implement semi-compatible functionality for mode=0 (preallocation)
and mode=FALLOC_FL_KEEP_SIZE (preallocation beyond EOF) for ZPL.
Since ZFS does COW and snapshots, preallocating blocks for a file
cannot guarantee that writes to the file will not run out of space.
Even if the first overwrite was guaranteed, it would not handle any
later overwrite of blocks due to COW, so strict compliance is futile.
Instead, make a best-effort check that at least enough free space is
currently available in the pool (with a bit of margin), then create
a sparse file of the requested size and continue on with life.
This does not handle all cases (e.g. several fallocate() calls before
writing into the files when the filesystem is nearly full), which
would require a more complex mechanism to be implemented, probably
based on a modified version of dmu_prealloc(), but is usable as-is.
A new module option zfs_fallocate_reserve_percent is used to control
the reserve margin for any single fallocate call. By default, this
is 110% of the requested preallocation size, so an additional 10% of
available space is reserved for overhead to allow the application a
good chance of finishing the write when the fallocate() succeeds.
If the heuristics of this basic fallocate implementation are not
desirable, the old non-functional behavior of returning EOPNOTSUPP
for calls can be restored by setting zfs_fallocate_reserve_percent=0.
The parameter of zfs_statvfs() is changed to take an inode instead
of a dentry, since no dentry is available in zfs_fallocate_common().
A few tests from @behlendorf cover basic fallocate functionality.
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Arshad Hussain <arshad.super@gmail.com>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
Issue #326
Closes #10408
2020-06-18 18:22:11 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_fallocate_reserve_percent\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Since ZFS is a copy-on-write filesystem with snapshots, blocks cannot be
|
|
|
|
preallocated for a file in order to guarantee that later writes will not
|
|
|
|
run out of space. Instead, fallocate() space preallocation only checks
|
|
|
|
that sufficient space is currently available in the pool or the user's
|
|
|
|
project quota allocation, and then creates a sparse file of the requested
|
|
|
|
size. The requested space is multiplied by \fBzfs_fallocate_reserve_percent\fR
|
|
|
|
to allow additional space for indirect blocks and other internal metadata.
|
|
|
|
Setting this value to 0 disables support for fallocate(2) and returns
|
|
|
|
EOPNOTSUPP for fallocate() space preallocation again.
|
|
|
|
.sp
|
|
|
|
Default value: \fB110\fR%
|
|
|
|
.RE
|
|
|
|
|
2015-12-09 23:34:16 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_fletcher_4_impl\fR (string)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Select a fletcher 4 implementation.
|
|
|
|
.sp
|
2016-06-24 03:32:40 +00:00
|
|
|
Supported selectors are: \fBfastest\fR, \fBscalar\fR, \fBsse2\fR, \fBssse3\fR,
|
2019-10-30 19:26:14 +00:00
|
|
|
\fBavx2\fR, \fBavx512f\fR, \fBavx512bw\fR, and \fBaarch64_neon\fR.
|
2016-07-06 11:42:04 +00:00
|
|
|
All of the selectors except \fBfastest\fR and \fBscalar\fR require instruction
|
|
|
|
set extensions to be available and will only appear if ZFS detects that they are
|
|
|
|
present at runtime. If multiple implementations of fletcher 4 are available,
|
|
|
|
the \fBfastest\fR will be chosen using a micro benchmark. Selecting \fBscalar\fR
|
|
|
|
results in the original, CPU based calculation, being used. Selecting any option
|
|
|
|
other than \fBfastest\fR and \fBscalar\fR results in vector instructions from
|
|
|
|
the respective CPU instruction set being used.
|
2015-12-09 23:34:16 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fBfastest\fR.
|
|
|
|
.RE
|
|
|
|
|
2016-01-23 00:41:02 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_free_bpobj_enabled\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Enable/disable the processing of the free_bpobj object.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
2014-09-07 15:06:08 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
This project allows top-level vdevs to be removed from the storage pool
with "zpool remove", reducing the total amount of storage in the pool.
This operation copies all allocated regions of the device to be removed
onto other devices, recording the mapping from old to new location.
After the removal is complete, read and free operations to the removed
(now "indirect") vdev must be remapped and performed at the new location
on disk. The indirect mapping table is kept in memory whenever the pool
is loaded, so there is minimal performance overhead when doing operations
on the indirect vdev.
The size of the in-memory mapping table will be reduced when its entries
become "obsolete" because they are no longer used by any block pointers
in the pool. An entry becomes obsolete when all the blocks that use
it are freed. An entry can also become obsolete when all the snapshots
that reference it are deleted, and the block pointers that reference it
have been "remapped" in all filesystems/zvols (and clones). Whenever an
indirect block is written, all the block pointers in it will be "remapped"
to their new (concrete) locations if possible. This process can be
accelerated by using the "zfs remap" command to proactively rewrite all
indirect blocks that reference indirect (removed) vdevs.
Note that when a device is removed, we do not verify the checksum of
the data that is copied. This makes the process much faster, but if it
were used on redundant vdevs (i.e. mirror or raidz vdevs), it would be
possible to copy the wrong data, when we have the correct data on e.g.
the other side of the mirror.
At the moment, only mirrors and simple top-level vdevs can be removed
and no removal is allowed if any of the top-level vdevs are raidz.
Porting Notes:
* Avoid zero-sized kmem_alloc() in vdev_compact_children().
The device evacuation code adds a dependency that
vdev_compact_children() be able to properly empty the vdev_child
array by setting it to NULL and zeroing vdev_children. Under Linux,
kmem_alloc() and related functions return a sentinel pointer rather
than NULL for zero-sized allocations.
* Remove comment regarding "mpt" driver where zfs_remove_max_segment
is initialized to SPA_MAXBLOCKSIZE.
Change zfs_condense_indirect_commit_entry_delay_ticks to
zfs_condense_indirect_commit_entry_delay_ms for consistency with
most other tunables in which delays are specified in ms.
* ZTS changes:
Use set_tunable rather than mdb
Use zpool sync as appropriate
Use sync_pool instead of sync
Kill jobs during test_removal_with_operation to allow unmount/export
Don't add non-disk names such as "mirror" or "raidz" to $DISKS
Use $TEST_BASE_DIR instead of /tmp
Increase HZ from 100 to 1000 which is more common on Linux
removal_multiple_indirection.ksh
Reduce iterations in order to not time out on the code
coverage builders.
removal_resume_export:
Functionally, the test case is correct but there exists a race
where the kernel thread hasn't been fully started yet and is
not visible. Wait for up to 1 second for the removal thread
to be started before giving up on it. Also, increase the
amount of data copied in order that the removal not finish
before the export has a chance to fail.
* MMP compatibility, the concept of concrete versus non-concrete devices
has slightly changed the semantics of vdev_writeable(). Update
mmp_random_leaf_impl() accordingly.
* Updated dbuf_remap() to handle the org.zfsonlinux:large_dnode pool
feature which is not supported by OpenZFS.
* Added support for new vdev removal tracepoints.
* Test cases removal_with_zdb and removal_condense_export have been
intentionally disabled. When run manually they pass as intended,
but when running in the automated test environment they produce
unreliable results on the latest Fedora release.
They may work better once the upstream pool import refectoring is
merged into ZoL at which point they will be re-enabled.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Alex Reece <alex@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/7614
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/f539f1eb
Closes #6900
2016-09-22 16:30:13 +00:00
|
|
|
\fBzfs_async_block_max_blocks\fR (ulong)
|
2014-09-07 15:06:08 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum number of blocks freed in a single txg.
|
|
|
|
.sp
|
Remove limit on number of async zio_frees of non-dedup blocks
The module parameter zfs_async_block_max_blocks limits the number of
blocks that can be freed by the background freeing of filesystems and
snapshots (from "zfs destroy"), in one TXG. This is useful when freeing
dedup blocks, becuase each zio_free() of a dedup block can require an
i/o to read the relevant part of the dedup table (DDT), and will also
dirty that block.
zfs_async_block_max_blocks is set to 100,000 by default. For the more
typical case where dedup is not used, this can have a negative
performance impact on the rate of background freeing (from "zfs
destroy"). For example, with recordsize=8k, and TXG's syncing once
every 5 seconds, we can free only 160MB of data per second, which may be
much less than the rate we can write data.
This change increases zfs_async_block_max_blocks to be unlimited by
default. To address the dedup freeing issue, a new tunable is
introduced, zfs_max_async_dedup_frees, which limits the number of
zio_free()'s of dedup blocks done by background destroys, per txg. The
default is 100,000 free's (same as the old zfs_async_block_max_blocks
default).
Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #10000
2020-02-14 16:39:46 +00:00
|
|
|
Default value: \fBULONG_MAX\fR (unlimited).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_max_async_dedup_frees\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum number of dedup blocks freed in a single txg.
|
|
|
|
.sp
|
2014-09-07 15:06:08 +00:00
|
|
|
Default value: \fB100,000\fR.
|
|
|
|
.RE
|
|
|
|
|
2016-08-26 18:43:21 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_override_estimate_recordsize\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Record size calculation override for zfs send estimates.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_async_read_max_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Maximum asynchronous read I/Os active to each device.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB3\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_async_read_min_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum asynchronous read I/Os active to each device.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_async_write_active_max_dirty_percent\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
When the pool has more than
|
|
|
|
\fBzfs_vdev_async_write_active_max_dirty_percent\fR dirty data, use
|
|
|
|
\fBzfs_vdev_async_write_max_active\fR to limit active async writes. If
|
|
|
|
the dirty data is between min and max, the active I/O limit is linearly
|
|
|
|
interpolated. See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB60\fR%.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_async_write_active_min_dirty_percent\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
When the pool has less than
|
|
|
|
\fBzfs_vdev_async_write_active_min_dirty_percent\fR dirty data, use
|
|
|
|
\fBzfs_vdev_async_write_min_active\fR to limit active async writes. If
|
|
|
|
the dirty data is between min and max, the active I/O limit is linearly
|
|
|
|
interpolated. See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB30\fR%.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_async_write_max_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Maximum asynchronous write I/Os active to each device.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB10\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_async_write_min_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum asynchronous write I/Os active to each device.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
2017-03-26 02:36:28 +00:00
|
|
|
Lower values are associated with better latency on rotational media but poorer
|
|
|
|
resilver performance. The default value of 2 was chosen as a compromise. A
|
|
|
|
value of 3 has been shown to improve resilver performance further at a cost of
|
|
|
|
further increasing latency.
|
|
|
|
.sp
|
|
|
|
Default value: \fB2\fR.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.RE
|
|
|
|
|
OpenZFS 9102 - zfs should be able to initialize storage devices
PROBLEM
========
The first access to a block incurs a performance penalty on some platforms
(e.g. AWS's EBS, VMware VMDKs). Therefore we recommend that volumes are
"thick provisioned", where supported by the platform (VMware). This can
create a large delay in getting a new virtual machines up and running (or
adding storage to an existing Engine). If the thick provision step is
omitted, write performance will be suboptimal until all blocks on the LUN
have been written.
SOLUTION
=========
This feature introduces a way to 'initialize' the disks at install or in the
background to make sure we don't incur this first read penalty.
When an entire LUN is added to ZFS, we make all space available immediately,
and allow ZFS to find unallocated space and zero it out. This works with
concurrent writes to arbitrary offsets, ensuring that we don't zero out
something that has been (or is in the middle of being) written. This scheme
can also be applied to existing pools (affecting only free regions on the
vdev). Detailed design:
- new subcommand:zpool initialize [-cs] <pool> [<vdev> ...]
- start, suspend, or cancel initialization
- Creates new open-context thread for each vdev
- Thread iterates through all metaslabs in this vdev
- Each metaslab:
- select a metaslab
- load the metaslab
- mark the metaslab as being zeroed
- walk all free ranges within that metaslab and translate
them to ranges on the leaf vdev
- issue a "zeroing" I/O on the leaf vdev that corresponds to
a free range on the metaslab we're working on
- continue until all free ranges for this metaslab have been
"zeroed"
- reset/unmark the metaslab being zeroed
- if more metaslabs exist, then repeat above tasks.
- if no more metaslabs, then we're done.
- progress for the initialization is stored on-disk in the vdev’s
leaf zap object. The following information is stored:
- the last offset that has been initialized
- the state of the initialization process (i.e. active,
suspended, or canceled)
- the start time for the initialization
- progress is reported via the zpool status command and shows
information for each of the vdevs that are initializing
Porting notes:
- Added zfs_initialize_value module parameter to set the pattern
written by "zpool initialize".
- Added zfs_vdev_{initializing,removal}_{min,max}_active module options.
Authored by: George Wilson <george.wilson@delphix.com>
Reviewed by: John Wren Kennedy <john.kennedy@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: loli10K <ezomori.nozomu@gmail.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Richard Lowe <richlowe@richlowe.net>
Signed-off-by: Tim Chase <tim@chase2k.com>
Ported-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/9102
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/c3963210eb
Closes #8230
2018-12-19 14:54:59 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_initializing_max_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum initializing I/Os active to each device.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_initializing_min_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum initializing I/Os active to each device.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_max_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The maximum number of I/Os active to each device. Ideally, this will be >=
|
Reduce latency effects of non-interactive I/O
Investigating influence of scrub (especially sequential) on random read
latency I've noticed that on some HDDs single 4KB read may take up to 4
seconds! Deeper investigation shown that many HDDs heavily prioritize
sequential reads even when those are submitted with queue depth of 1.
This patch addresses the latency from two sides:
- by using _min_active queue depths for non-interactive requests while
the interactive request(s) are active and few requests after;
- by throttling it further if no interactive requests has completed
while configured amount of non-interactive did.
While there, I've also modified vdev_queue_class_to_issue() to give
more chances to schedule at least _min_active requests to the lowest
priorities. It should reduce starvation if several non-interactive
processes are running same time with some interactive and I think should
make possible setting of zfs_vdev_max_active to as low as 1.
I've benchmarked this change with 4KB random reads from ZVOL with 16KB
block size on newly written non-fragmented pool. On fragmented pool I
also saw improvements, but not so dramatic. Below are log2 histograms
of the random read latency in milliseconds for different devices:
4 2x mirror vdevs of SATA HDD WDC WD20EFRX-68EUZN0 before:
0, 0, 2, 1, 12, 21, 19, 18, 10, 15, 17, 21
after:
0, 0, 0, 24, 101, 195, 419, 250, 47, 4, 0, 0
, that means maximum latency reduction from 2s to 500ms.
4 2x mirror vdevs of SATA HDD WDC WD80EFZX-68UW8N0 before:
0, 0, 2, 31, 38, 28, 18, 12, 17, 20, 24, 10, 3
after:
0, 0, 55, 247, 455, 470, 412, 181, 36, 0, 0, 0, 0
, i.e. from 4s to 250ms.
1 SAS HDD SEAGATE ST14000NM0048 before:
0, 0, 29, 70, 107, 45, 27, 1, 0, 0, 1, 4, 19
after:
1, 29, 681, 1261, 676, 1633, 67, 1, 0, 0, 0, 0, 0
, i.e. from 4s to 125ms.
1 SAS SSD SEAGATE XS3840TE70014 before (microseconds):
0, 0, 0, 0, 0, 0, 0, 0, 70, 18343, 82548, 618
after:
0, 0, 0, 0, 0, 0, 0, 0, 283, 92351, 34844, 90
I've also measured scrub time during the test and on idle pools. On
idle fragmented pool I've measured scrub getting few percent faster
due to use of QD3 instead of QD2 before. On idle non-fragmented pool
I've measured no difference. On busy non-fragmented pool I've measured
scrub time increase about 1.5-1.7x, while IOPS increase reached 5-9x.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored-By: iXsystems, Inc.
Closes #11166
2020-11-24 17:26:42 +00:00
|
|
|
the sum of each queue's max_active. See the section "ZFS I/O SCHEDULER".
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB1,000\fR.
|
|
|
|
.RE
|
|
|
|
|
2020-07-03 18:05:50 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_rebuild_max_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum sequential resilver I/Os active to each device.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB3\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_rebuild_min_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum sequential resilver I/Os active to each device.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
OpenZFS 9102 - zfs should be able to initialize storage devices
PROBLEM
========
The first access to a block incurs a performance penalty on some platforms
(e.g. AWS's EBS, VMware VMDKs). Therefore we recommend that volumes are
"thick provisioned", where supported by the platform (VMware). This can
create a large delay in getting a new virtual machines up and running (or
adding storage to an existing Engine). If the thick provision step is
omitted, write performance will be suboptimal until all blocks on the LUN
have been written.
SOLUTION
=========
This feature introduces a way to 'initialize' the disks at install or in the
background to make sure we don't incur this first read penalty.
When an entire LUN is added to ZFS, we make all space available immediately,
and allow ZFS to find unallocated space and zero it out. This works with
concurrent writes to arbitrary offsets, ensuring that we don't zero out
something that has been (or is in the middle of being) written. This scheme
can also be applied to existing pools (affecting only free regions on the
vdev). Detailed design:
- new subcommand:zpool initialize [-cs] <pool> [<vdev> ...]
- start, suspend, or cancel initialization
- Creates new open-context thread for each vdev
- Thread iterates through all metaslabs in this vdev
- Each metaslab:
- select a metaslab
- load the metaslab
- mark the metaslab as being zeroed
- walk all free ranges within that metaslab and translate
them to ranges on the leaf vdev
- issue a "zeroing" I/O on the leaf vdev that corresponds to
a free range on the metaslab we're working on
- continue until all free ranges for this metaslab have been
"zeroed"
- reset/unmark the metaslab being zeroed
- if more metaslabs exist, then repeat above tasks.
- if no more metaslabs, then we're done.
- progress for the initialization is stored on-disk in the vdev’s
leaf zap object. The following information is stored:
- the last offset that has been initialized
- the state of the initialization process (i.e. active,
suspended, or canceled)
- the start time for the initialization
- progress is reported via the zpool status command and shows
information for each of the vdevs that are initializing
Porting notes:
- Added zfs_initialize_value module parameter to set the pattern
written by "zpool initialize".
- Added zfs_vdev_{initializing,removal}_{min,max}_active module options.
Authored by: George Wilson <george.wilson@delphix.com>
Reviewed by: John Wren Kennedy <john.kennedy@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: loli10K <ezomori.nozomu@gmail.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Richard Lowe <richlowe@richlowe.net>
Signed-off-by: Tim Chase <tim@chase2k.com>
Ported-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/9102
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/c3963210eb
Closes #8230
2018-12-19 14:54:59 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_removal_max_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum removal I/Os active to each device.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB2\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_removal_min_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum removal I/Os active to each device.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_scrub_max_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Maximum scrub I/Os active to each device.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB2\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_scrub_min_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum scrub I/Os active to each device.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_sync_read_max_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Maximum synchronous read I/Os active to each device.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB10\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_sync_read_min_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum synchronous read I/Os active to each device.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB10\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_sync_write_max_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Maximum synchronous write I/Os active to each device.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB10\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_sync_write_min_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum synchronous write I/Os active to each device.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB10\fR.
|
|
|
|
.RE
|
|
|
|
|
2019-03-29 16:13:20 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_trim_max_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum trim/discard I/Os active to each device.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB2\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_trim_min_active\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum trim/discard I/Os active to each device.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
Reduce latency effects of non-interactive I/O
Investigating influence of scrub (especially sequential) on random read
latency I've noticed that on some HDDs single 4KB read may take up to 4
seconds! Deeper investigation shown that many HDDs heavily prioritize
sequential reads even when those are submitted with queue depth of 1.
This patch addresses the latency from two sides:
- by using _min_active queue depths for non-interactive requests while
the interactive request(s) are active and few requests after;
- by throttling it further if no interactive requests has completed
while configured amount of non-interactive did.
While there, I've also modified vdev_queue_class_to_issue() to give
more chances to schedule at least _min_active requests to the lowest
priorities. It should reduce starvation if several non-interactive
processes are running same time with some interactive and I think should
make possible setting of zfs_vdev_max_active to as low as 1.
I've benchmarked this change with 4KB random reads from ZVOL with 16KB
block size on newly written non-fragmented pool. On fragmented pool I
also saw improvements, but not so dramatic. Below are log2 histograms
of the random read latency in milliseconds for different devices:
4 2x mirror vdevs of SATA HDD WDC WD20EFRX-68EUZN0 before:
0, 0, 2, 1, 12, 21, 19, 18, 10, 15, 17, 21
after:
0, 0, 0, 24, 101, 195, 419, 250, 47, 4, 0, 0
, that means maximum latency reduction from 2s to 500ms.
4 2x mirror vdevs of SATA HDD WDC WD80EFZX-68UW8N0 before:
0, 0, 2, 31, 38, 28, 18, 12, 17, 20, 24, 10, 3
after:
0, 0, 55, 247, 455, 470, 412, 181, 36, 0, 0, 0, 0
, i.e. from 4s to 250ms.
1 SAS HDD SEAGATE ST14000NM0048 before:
0, 0, 29, 70, 107, 45, 27, 1, 0, 0, 1, 4, 19
after:
1, 29, 681, 1261, 676, 1633, 67, 1, 0, 0, 0, 0, 0
, i.e. from 4s to 125ms.
1 SAS SSD SEAGATE XS3840TE70014 before (microseconds):
0, 0, 0, 0, 0, 0, 0, 0, 70, 18343, 82548, 618
after:
0, 0, 0, 0, 0, 0, 0, 0, 283, 92351, 34844, 90
I've also measured scrub time during the test and on idle pools. On
idle fragmented pool I've measured scrub getting few percent faster
due to use of QD3 instead of QD2 before. On idle non-fragmented pool
I've measured no difference. On busy non-fragmented pool I've measured
scrub time increase about 1.5-1.7x, while IOPS increase reached 5-9x.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored-By: iXsystems, Inc.
Closes #11166
2020-11-24 17:26:42 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_nia_delay\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
For non-interactive I/O (scrub, resilver, removal, initialize and rebuild),
|
|
|
|
the number of concurrently-active I/O's is limited to *_min_active, unless
|
|
|
|
the vdev is "idle". When there are no interactive I/Os active (sync or
|
|
|
|
async), and zfs_vdev_nia_delay I/Os have completed since the last
|
|
|
|
interactive I/O, then the vdev is considered to be "idle", and the number
|
|
|
|
of concurrently-active non-interactive I/O's is increased to *_max_active.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB5\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_nia_credit\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Some HDDs tend to prioritize sequential I/O so high, that concurrent
|
|
|
|
random I/O latency reaches several seconds. On some HDDs it happens
|
|
|
|
even if sequential I/Os are submitted one at a time, and so setting
|
|
|
|
*_max_active to 1 does not help. To prevent non-interactive I/Os, like
|
|
|
|
scrub, from monopolizing the device no more than zfs_vdev_nia_credit
|
|
|
|
I/Os can be sent while there are outstanding incomplete interactive
|
|
|
|
I/Os. This enforced wait ensures the HDD services the interactive I/O
|
|
|
|
within a reasonable amount of time.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB5\fR.
|
|
|
|
.RE
|
|
|
|
|
2016-10-14 00:59:18 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_queue_depth_pct\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-04-25 04:01:04 +00:00
|
|
|
Maximum number of queued allocations per top-level vdev expressed as
|
|
|
|
a percentage of \fBzfs_vdev_async_write_max_active\fR which allows the
|
|
|
|
system to detect devices that are more capable of handling allocations
|
|
|
|
and to allocate more blocks to those devices. It allows for dynamic
|
|
|
|
allocation distribution when devices are imbalanced as fuller devices
|
|
|
|
will tend to be slower than empty devices.
|
|
|
|
|
|
|
|
See also \fBzio_dva_throttle_enabled\fR.
|
2016-10-14 00:59:18 +00:00
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB1000\fR%.
|
2016-10-14 00:59:18 +00:00
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_expire_snapshot\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Seconds to expire .zfs/snapshot
|
|
|
|
.sp
|
|
|
|
Default value: \fB300\fR.
|
|
|
|
.RE
|
|
|
|
|
2015-08-28 21:54:32 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_admin_snapshot\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Allow the creation, removal, or renaming of entries in the .zfs/snapshot
|
|
|
|
directory to cause the creation, destruction, or renaming of snapshots.
|
|
|
|
When enabled this functionality works both locally and over NFS exports
|
|
|
|
which have the 'no_root_squash' option set. This functionality is disabled
|
|
|
|
by default.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_flags\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2014-12-23 00:54:43 +00:00
|
|
|
Set additional debugging flags. The following flags may be bitwise-or'd
|
|
|
|
together.
|
|
|
|
.sp
|
|
|
|
.TS
|
|
|
|
box;
|
|
|
|
rB lB
|
|
|
|
lB lB
|
|
|
|
r l.
|
|
|
|
Value Symbolic Name
|
|
|
|
Description
|
|
|
|
_
|
|
|
|
1 ZFS_DEBUG_DPRINTF
|
|
|
|
Enable dprintf entries in the debug log.
|
|
|
|
_
|
|
|
|
2 ZFS_DEBUG_DBUF_VERIFY *
|
|
|
|
Enable extra dbuf verifications.
|
|
|
|
_
|
|
|
|
4 ZFS_DEBUG_DNODE_VERIFY *
|
|
|
|
Enable extra dnode verifications.
|
|
|
|
_
|
|
|
|
8 ZFS_DEBUG_SNAPNAMES
|
|
|
|
Enable snapshot name verification.
|
|
|
|
_
|
|
|
|
16 ZFS_DEBUG_MODIFY
|
|
|
|
Check for illegally modified ARC buffers.
|
|
|
|
_
|
|
|
|
64 ZFS_DEBUG_ZIO_FREE
|
|
|
|
Enable verification of block frees.
|
|
|
|
_
|
|
|
|
128 ZFS_DEBUG_HISTOGRAM_VERIFY
|
|
|
|
Enable extra spacemap histogram verifications.
|
2017-07-26 06:09:48 +00:00
|
|
|
_
|
|
|
|
256 ZFS_DEBUG_METASLAB_VERIFY
|
|
|
|
Verify space accounting on disk matches in-core range_trees.
|
|
|
|
_
|
|
|
|
512 ZFS_DEBUG_SET_ERROR
|
|
|
|
Enable SET_ERROR and dprintf entries in the debug log.
|
2019-03-29 16:13:20 +00:00
|
|
|
_
|
|
|
|
1024 ZFS_DEBUG_INDIRECT_REMAP
|
|
|
|
Verify split blocks created by device removal.
|
|
|
|
_
|
|
|
|
2048 ZFS_DEBUG_TRIM
|
|
|
|
Verify TRIM ranges are always within the allocatable range tree.
|
Log Spacemap Project
= Motivation
At Delphix we've seen a lot of customer systems where fragmentation
is over 75% and random writes take a performance hit because a lot
of time is spend on I/Os that update on-disk space accounting metadata.
Specifically, we seen cases where 20% to 40% of sync time is spend
after sync pass 1 and ~30% of the I/Os on the system is spent updating
spacemaps.
The problem is that these pools have existed long enough that we've
touched almost every metaslab at least once, and random writes
scatter frees across all metaslabs every TXG, thus appending to
their spacemaps and resulting in many I/Os. To give an example,
assuming that every VDEV has 200 metaslabs and our writes fit within
a single spacemap block (generally 4K) we have 200 I/Os. Then if we
assume 2 levels of indirection, we need 400 additional I/Os and
since we are talking about metadata for which we keep 2 extra copies
for redundancy we need to triple that number, leading to a total of
1800 I/Os per VDEV every TXG.
We could try and decrease the number of metaslabs so we have less
I/Os per TXG but then each metaslab would cover a wider range on
disk and thus would take more time to be loaded in memory from disk.
In addition, after it's loaded, it's range tree would consume more
memory.
Another idea would be to just increase the spacemap block size
which would allow us to fit more entries within an I/O block
resulting in fewer I/Os per metaslab and a speedup in loading time.
The problem is still that we don't deal with the number of I/Os
going up as the number of metaslabs is increasing and the fact
is that we generally write a lot to a few metaslabs and a little
to the rest of them. Thus, just increasing the block size would
actually waste bandwidth because we won't be utilizing our bigger
block size.
= About this patch
This patch introduces the Log Spacemap project which provides the
solution to the above problem while taking into account all the
aforementioned tradeoffs. The details on how it achieves that can
be found in the references sections below and in the code (see
Big Theory Statement in spa_log_spacemap.c).
Even though the change is fairly constraint within the metaslab
and lower-level SPA codepaths, there is a side-change that is
user-facing. The change is that VDEV IDs from VDEV holes will no
longer be reused. To give some background and reasoning for this,
when a log device is removed and its VDEV structure was replaced
with a hole (or was compacted; if at the end of the vdev array),
its vdev_id could be reused by devices added after that. Now
with the pool-wide space maps recording the vdev ID, this behavior
can cause problems (e.g. is this entry referring to a segment in
the new vdev or the removed log?). Thus, to simplify things the
ID reuse behavior is gone and now vdev IDs for top-level vdevs
are truly unique within a pool.
= Testing
The illumos implementation of this feature has been used internally
for a year and has been in production for ~6 months. For this patch
specifically there don't seem to be any regressions introduced to
ZTS and I have been running zloop for a week without any related
problems.
= Performance Analysis (Linux Specific)
All performance results and analysis for illumos can be found in
the links of the references. Redoing the same experiments in Linux
gave similar results. Below are the specifics of the Linux run.
After the pool reached stable state the percentage of the time
spent in pass 1 per TXG was 64% on average for the stock bits
while the log spacemap bits stayed at 95% during the experiment
(graph: sdimitro.github.io/img/linux-lsm/PercOfSyncInPassOne.png).
Sync times per TXG were 37.6 seconds on average for the stock
bits and 22.7 seconds for the log spacemap bits (related graph:
sdimitro.github.io/img/linux-lsm/SyncTimePerTXG.png). As a result
the log spacemap bits were able to push more TXGs, which is also
the reason why all graphs quantified per TXG have more entries for
the log spacemap bits.
Another interesting aspect in terms of txg syncs is that the stock
bits had 22% of their TXGs reach sync pass 7, 55% reach sync pass 8,
and 20% reach 9. The log space map bits reached sync pass 4 in 79%
of their TXGs, sync pass 7 in 19%, and sync pass 8 at 1%. This
emphasizes the fact that not only we spend less time on metadata
but we also iterate less times to convergence in spa_sync() dirtying
objects.
[related graphs:
stock- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGStock.png
lsm- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGLSM.png]
Finally, the improvement in IOPs that the userland gains from the
change is approximately 40%. There is a consistent win in IOPS as
you can see from the graphs below but the absolute amount of
improvement that the log spacemap gives varies within each minute
interval.
sdimitro.github.io/img/linux-lsm/StockVsLog3Days.png
sdimitro.github.io/img/linux-lsm/StockVsLog10Hours.png
= Porting to Other Platforms
For people that want to port this commit to other platforms below
is a list of ZoL commits that this patch depends on:
Make zdb results for checkpoint tests consistent
db587941c5ff6dea01932bb78f70db63cf7f38ba
Update vdev_is_spacemap_addressable() for new spacemap encoding
419ba5914552c6185afbe1dd17b3ed4b0d526547
Simplify spa_sync by breaking it up to smaller functions
8dc2197b7b1e4d7ebc1420ea30e51c6541f1d834
Factor metaslab_load_wait() in metaslab_load()
b194fab0fb6caad18711abccaff3c69ad8b3f6d3
Rename range_tree_verify to range_tree_verify_not_present
df72b8bebe0ebac0b20e0750984bad182cb6564a
Change target size of metaslabs from 256GB to 16GB
c853f382db731e15a87512f4ef1101d14d778a55
zdb -L should skip leak detection altogether
21e7cf5da89f55ce98ec1115726b150e19eefe89
vs_alloc can underflow in L2ARC vdevs
7558997d2f808368867ca7e5234e5793446e8f3f
Simplify log vdev removal code
6c926f426a26ffb6d7d8e563e33fc176164175cb
Get rid of space_map_update() for ms_synced_length
425d3237ee88abc53d8522a7139c926d278b4b7f
Introduce auxiliary metaslab histograms
928e8ad47d3478a3d5d01f0dd6ae74a9371af65e
Error path in metaslab_load_impl() forgets to drop ms_sync_lock
8eef997679ba54547f7d361553d21b3291f41ae7
= References
Background, Motivation, and Internals of the Feature
- OpenZFS 2017 Presentation:
youtu.be/jj2IxRkl5bQ
- Slides:
slideshare.net/SerapheimNikolaosDim/zfs-log-spacemaps-project
Flushing Algorithm Internals & Performance Results
(Illumos Specific)
- Blogpost:
sdimitro.github.io/post/zfs-lsm-flushing/
- OpenZFS 2018 Presentation:
youtu.be/x6D2dHRjkxw
- Slides:
slideshare.net/SerapheimNikolaosDim/zfs-log-spacemap-flushing-algorithm
Upstream Delphix Issues:
DLPX-51539, DLPX-59659, DLPX-57783, DLPX-61438, DLPX-41227, DLPX-59320
DLPX-63385
Reviewed-by: Sean Eric Fagan <sef@ixsystems.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: George Wilson <gwilson@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8442
2019-07-16 17:11:49 +00:00
|
|
|
_
|
|
|
|
4096 ZFS_DEBUG_LOG_SPACEMAP
|
|
|
|
Verify that the log summary is consistent with the spacemap log
|
|
|
|
and enable zfs_dbgmsgs for metaslab loading and flushing.
|
2014-12-23 00:54:43 +00:00
|
|
|
.TE
|
|
|
|
.sp
|
|
|
|
* Requires debug build.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2014-12-23 00:54:43 +00:00
|
|
|
Default value: \fB0\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
2014-06-05 21:20:08 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_free_leak_on_eio\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
If destroy encounters an EIO while reading metadata (e.g. indirect
|
|
|
|
blocks), space referenced by the missing metadata can not be freed.
|
|
|
|
Normally this causes the background destroy to become "stalled", as
|
|
|
|
it is unable to make forward progress. While in this stalled state,
|
|
|
|
all remaining space to free from the error-encountering filesystem is
|
|
|
|
"temporarily leaked". Set this flag to cause it to ignore the EIO,
|
|
|
|
permanently leak the space from indirect blocks that can not be read,
|
|
|
|
and continue to free everything else that it can.
|
|
|
|
|
|
|
|
The default, "stalling" behavior is useful if the storage partially
|
|
|
|
fails (i.e. some but not all i/os fail), and then later recovers. In
|
|
|
|
this case, we will be able to continue pool operations while it is
|
|
|
|
partially failed, and when it recovers, we can continue to free the
|
|
|
|
space, with no leaks. However, note that this case is actually
|
|
|
|
fairly rare.
|
|
|
|
|
|
|
|
Typically pools either (a) fail completely (but perhaps temporarily,
|
|
|
|
e.g. a top-level vdev going offline), or (b) have localized,
|
|
|
|
permanent errors (e.g. disk returns the wrong data due to bit flip or
|
|
|
|
firmware bug). In case (a), this setting does not matter because the
|
|
|
|
pool will be suspended and the sync thread will not be able to make
|
|
|
|
forward progress regardless. In case (b), because the error is
|
|
|
|
permanent, the best we can do is leak the minimum amount of space,
|
|
|
|
which is what setting this flag will do. Therefore, it is reasonable
|
|
|
|
for this flag to normally be set, but we chose the more conservative
|
|
|
|
approach of not setting it, so that there is no possibility of
|
|
|
|
leaking space in the "partial temporary" failure case.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_free_min_time_ms\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2016-11-15 01:03:57 +00:00
|
|
|
During a \fBzfs destroy\fR operation using \fBfeature@async_destroy\fR a minimum
|
2015-12-30 17:44:46 +00:00
|
|
|
of this much time will be spent working on freeing blocks per txg.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB1,000\fR.
|
|
|
|
.RE
|
|
|
|
|
2020-01-07 20:41:53 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_obsolete_min_time_ms\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2020-06-10 04:24:09 +00:00
|
|
|
Similar to \fBzfs_free_min_time_ms\fR but for cleanup of old indirection records
|
2020-01-07 20:41:53 +00:00
|
|
|
for removed vdevs.
|
|
|
|
.sp
|
|
|
|
Default value: \fB500\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_immediate_write_sz\fR (long)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Largest data block to write to zil. Larger blocks will be treated as if the
|
2016-11-15 01:03:57 +00:00
|
|
|
dataset being written to had the property setting \fBlogbias=throughput\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB32,768\fR.
|
|
|
|
.RE
|
|
|
|
|
OpenZFS 9102 - zfs should be able to initialize storage devices
PROBLEM
========
The first access to a block incurs a performance penalty on some platforms
(e.g. AWS's EBS, VMware VMDKs). Therefore we recommend that volumes are
"thick provisioned", where supported by the platform (VMware). This can
create a large delay in getting a new virtual machines up and running (or
adding storage to an existing Engine). If the thick provision step is
omitted, write performance will be suboptimal until all blocks on the LUN
have been written.
SOLUTION
=========
This feature introduces a way to 'initialize' the disks at install or in the
background to make sure we don't incur this first read penalty.
When an entire LUN is added to ZFS, we make all space available immediately,
and allow ZFS to find unallocated space and zero it out. This works with
concurrent writes to arbitrary offsets, ensuring that we don't zero out
something that has been (or is in the middle of being) written. This scheme
can also be applied to existing pools (affecting only free regions on the
vdev). Detailed design:
- new subcommand:zpool initialize [-cs] <pool> [<vdev> ...]
- start, suspend, or cancel initialization
- Creates new open-context thread for each vdev
- Thread iterates through all metaslabs in this vdev
- Each metaslab:
- select a metaslab
- load the metaslab
- mark the metaslab as being zeroed
- walk all free ranges within that metaslab and translate
them to ranges on the leaf vdev
- issue a "zeroing" I/O on the leaf vdev that corresponds to
a free range on the metaslab we're working on
- continue until all free ranges for this metaslab have been
"zeroed"
- reset/unmark the metaslab being zeroed
- if more metaslabs exist, then repeat above tasks.
- if no more metaslabs, then we're done.
- progress for the initialization is stored on-disk in the vdev’s
leaf zap object. The following information is stored:
- the last offset that has been initialized
- the state of the initialization process (i.e. active,
suspended, or canceled)
- the start time for the initialization
- progress is reported via the zpool status command and shows
information for each of the vdevs that are initializing
Porting notes:
- Added zfs_initialize_value module parameter to set the pattern
written by "zpool initialize".
- Added zfs_vdev_{initializing,removal}_{min,max}_active module options.
Authored by: George Wilson <george.wilson@delphix.com>
Reviewed by: John Wren Kennedy <john.kennedy@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: loli10K <ezomori.nozomu@gmail.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Richard Lowe <richlowe@richlowe.net>
Signed-off-by: Tim Chase <tim@chase2k.com>
Ported-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/9102
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/c3963210eb
Closes #8230
2018-12-19 14:54:59 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_initialize_value\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Pattern written to vdev free space by \fBzpool initialize\fR.
|
|
|
|
.sp
|
|
|
|
Default value: \fB16,045,690,984,833,335,022\fR (0xdeadbeefdeadbeee).
|
|
|
|
.RE
|
|
|
|
|
Add subcommand to wait for background zfs activity to complete
Currently the best way to wait for the completion of a long-running
operation in a pool, like a scrub or device removal, is to poll 'zpool
status' and parse its output, which is neither efficient nor convenient.
This change adds a 'wait' subcommand to the zpool command. When invoked,
'zpool wait' will block until a specified type of background activity
completes. Currently, this subcommand can wait for any of the following:
- Scrubs or resilvers to complete
- Devices to initialized
- Devices to be replaced
- Devices to be removed
- Checkpoints to be discarded
- Background freeing to complete
For example, a scrub that is in progress could be waited for by running
zpool wait -t scrub <pool>
This also adds a -w flag to the attach, checkpoint, initialize, replace,
remove, and scrub subcommands. When used, this flag makes the operations
kicked off by these subcommands synchronous instead of asynchronous.
This functionality is implemented using a new ioctl. The type of
activity to wait for is provided as input to the ioctl, and the ioctl
blocks until all activity of that type has completed. An ioctl was used
over other methods of kernel-userspace communiction primarily for the
sake of portability.
Porting Notes:
This is ported from Delphix OS change DLPX-44432. The following changes
were made while porting:
- Added ZoL-style ioctl input declaration.
- Reorganized error handling in zpool_initialize in libzfs to integrate
better with changes made for TRIM support.
- Fixed check for whether a checkpoint discard is in progress.
Previously it also waited if the pool had a checkpoint, instead of
just if a checkpoint was being discarded.
- Exposed zfs_initialize_chunk_size as a ZoL-style tunable.
- Updated more existing tests to make use of new 'zpool wait'
functionality, tests that don't exist in Delphix OS.
- Used existing ZoL tunable zfs_scan_suspend_progress, together with
zinject, in place of a new tunable zfs_scan_max_blks_per_txg.
- Added support for a non-integral interval argument to zpool wait.
Future work:
ZoL has support for trimming devices, which Delphix OS does not. In the
future, 'zpool wait' could be extended to add the ability to wait for
trim operations to complete.
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: John Gallagher <john.gallagher@delphix.com>
Closes #9162
2019-09-14 01:09:06 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_initialize_chunk_size\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Size of writes used by \fBzpool initialize\fR.
|
|
|
|
This option is used by the test suite to facilitate testing.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1,048,576\fR
|
|
|
|
.RE
|
|
|
|
|
2019-07-26 17:54:14 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_livelist_max_entries\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The threshold size (in block pointers) at which we create a new sub-livelist.
|
|
|
|
Larger sublists are more costly from a memory perspective but the fewer
|
|
|
|
sublists there are, the lower the cost of insertion.
|
|
|
|
.sp
|
|
|
|
Default value: \fB500,000\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_livelist_min_percent_shared\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
If the amount of shared space between a snapshot and its clone drops below
|
|
|
|
this threshold, the clone turns off the livelist and reverts to the old deletion
|
|
|
|
method. This is in place because once a clone has been overwritten enough
|
|
|
|
livelists no long give us a benefit.
|
|
|
|
.sp
|
|
|
|
Default value: \fB75\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_livelist_condense_new_alloc\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Incremented each time an extra ALLOC blkptr is added to a livelist entry while
|
|
|
|
it is being condensed.
|
|
|
|
This option is used by the test suite to track race conditions.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_livelist_condense_sync_cancel\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Incremented each time livelist condensing is canceled while in
|
|
|
|
spa_livelist_condense_sync.
|
|
|
|
This option is used by the test suite to track race conditions.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_livelist_condense_sync_pause\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
When set, the livelist condense process pauses indefinitely before
|
|
|
|
executing the synctask - spa_livelist_condense_sync.
|
|
|
|
This option is used by the test suite to trigger race conditions.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_livelist_condense_zthr_cancel\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Incremented each time livelist condensing is canceled while in
|
|
|
|
spa_livelist_condense_cb.
|
|
|
|
This option is used by the test suite to track race conditions.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_livelist_condense_zthr_pause\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
When set, the livelist condense process pauses indefinitely before
|
|
|
|
executing the open context condensing work in spa_livelist_condense_cb.
|
|
|
|
This option is used by the test suite to trigger race conditions.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2018-06-15 22:10:42 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_lua_max_instrlimit\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The maximum execution time limit that can be set for a ZFS channel program,
|
|
|
|
specified as a number of Lua instructions.
|
|
|
|
.sp
|
|
|
|
Default value: \fB100,000,000\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_lua_max_memlimit\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The maximum memory limit that can be set for a ZFS channel program, specified
|
|
|
|
in bytes.
|
|
|
|
.sp
|
|
|
|
Default value: \fB104,857,600\fR.
|
|
|
|
.RE
|
|
|
|
|
2016-09-12 15:15:20 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_max_dataset_nesting\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The maximum depth of nested datasets. This value can be tuned temporarily to
|
|
|
|
fix existing datasets that exceed the predefined limit.
|
|
|
|
.sp
|
|
|
|
Default value: \fB50\fR.
|
|
|
|
.RE
|
|
|
|
|
Log Spacemap Project
= Motivation
At Delphix we've seen a lot of customer systems where fragmentation
is over 75% and random writes take a performance hit because a lot
of time is spend on I/Os that update on-disk space accounting metadata.
Specifically, we seen cases where 20% to 40% of sync time is spend
after sync pass 1 and ~30% of the I/Os on the system is spent updating
spacemaps.
The problem is that these pools have existed long enough that we've
touched almost every metaslab at least once, and random writes
scatter frees across all metaslabs every TXG, thus appending to
their spacemaps and resulting in many I/Os. To give an example,
assuming that every VDEV has 200 metaslabs and our writes fit within
a single spacemap block (generally 4K) we have 200 I/Os. Then if we
assume 2 levels of indirection, we need 400 additional I/Os and
since we are talking about metadata for which we keep 2 extra copies
for redundancy we need to triple that number, leading to a total of
1800 I/Os per VDEV every TXG.
We could try and decrease the number of metaslabs so we have less
I/Os per TXG but then each metaslab would cover a wider range on
disk and thus would take more time to be loaded in memory from disk.
In addition, after it's loaded, it's range tree would consume more
memory.
Another idea would be to just increase the spacemap block size
which would allow us to fit more entries within an I/O block
resulting in fewer I/Os per metaslab and a speedup in loading time.
The problem is still that we don't deal with the number of I/Os
going up as the number of metaslabs is increasing and the fact
is that we generally write a lot to a few metaslabs and a little
to the rest of them. Thus, just increasing the block size would
actually waste bandwidth because we won't be utilizing our bigger
block size.
= About this patch
This patch introduces the Log Spacemap project which provides the
solution to the above problem while taking into account all the
aforementioned tradeoffs. The details on how it achieves that can
be found in the references sections below and in the code (see
Big Theory Statement in spa_log_spacemap.c).
Even though the change is fairly constraint within the metaslab
and lower-level SPA codepaths, there is a side-change that is
user-facing. The change is that VDEV IDs from VDEV holes will no
longer be reused. To give some background and reasoning for this,
when a log device is removed and its VDEV structure was replaced
with a hole (or was compacted; if at the end of the vdev array),
its vdev_id could be reused by devices added after that. Now
with the pool-wide space maps recording the vdev ID, this behavior
can cause problems (e.g. is this entry referring to a segment in
the new vdev or the removed log?). Thus, to simplify things the
ID reuse behavior is gone and now vdev IDs for top-level vdevs
are truly unique within a pool.
= Testing
The illumos implementation of this feature has been used internally
for a year and has been in production for ~6 months. For this patch
specifically there don't seem to be any regressions introduced to
ZTS and I have been running zloop for a week without any related
problems.
= Performance Analysis (Linux Specific)
All performance results and analysis for illumos can be found in
the links of the references. Redoing the same experiments in Linux
gave similar results. Below are the specifics of the Linux run.
After the pool reached stable state the percentage of the time
spent in pass 1 per TXG was 64% on average for the stock bits
while the log spacemap bits stayed at 95% during the experiment
(graph: sdimitro.github.io/img/linux-lsm/PercOfSyncInPassOne.png).
Sync times per TXG were 37.6 seconds on average for the stock
bits and 22.7 seconds for the log spacemap bits (related graph:
sdimitro.github.io/img/linux-lsm/SyncTimePerTXG.png). As a result
the log spacemap bits were able to push more TXGs, which is also
the reason why all graphs quantified per TXG have more entries for
the log spacemap bits.
Another interesting aspect in terms of txg syncs is that the stock
bits had 22% of their TXGs reach sync pass 7, 55% reach sync pass 8,
and 20% reach 9. The log space map bits reached sync pass 4 in 79%
of their TXGs, sync pass 7 in 19%, and sync pass 8 at 1%. This
emphasizes the fact that not only we spend less time on metadata
but we also iterate less times to convergence in spa_sync() dirtying
objects.
[related graphs:
stock- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGStock.png
lsm- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGLSM.png]
Finally, the improvement in IOPs that the userland gains from the
change is approximately 40%. There is a consistent win in IOPS as
you can see from the graphs below but the absolute amount of
improvement that the log spacemap gives varies within each minute
interval.
sdimitro.github.io/img/linux-lsm/StockVsLog3Days.png
sdimitro.github.io/img/linux-lsm/StockVsLog10Hours.png
= Porting to Other Platforms
For people that want to port this commit to other platforms below
is a list of ZoL commits that this patch depends on:
Make zdb results for checkpoint tests consistent
db587941c5ff6dea01932bb78f70db63cf7f38ba
Update vdev_is_spacemap_addressable() for new spacemap encoding
419ba5914552c6185afbe1dd17b3ed4b0d526547
Simplify spa_sync by breaking it up to smaller functions
8dc2197b7b1e4d7ebc1420ea30e51c6541f1d834
Factor metaslab_load_wait() in metaslab_load()
b194fab0fb6caad18711abccaff3c69ad8b3f6d3
Rename range_tree_verify to range_tree_verify_not_present
df72b8bebe0ebac0b20e0750984bad182cb6564a
Change target size of metaslabs from 256GB to 16GB
c853f382db731e15a87512f4ef1101d14d778a55
zdb -L should skip leak detection altogether
21e7cf5da89f55ce98ec1115726b150e19eefe89
vs_alloc can underflow in L2ARC vdevs
7558997d2f808368867ca7e5234e5793446e8f3f
Simplify log vdev removal code
6c926f426a26ffb6d7d8e563e33fc176164175cb
Get rid of space_map_update() for ms_synced_length
425d3237ee88abc53d8522a7139c926d278b4b7f
Introduce auxiliary metaslab histograms
928e8ad47d3478a3d5d01f0dd6ae74a9371af65e
Error path in metaslab_load_impl() forgets to drop ms_sync_lock
8eef997679ba54547f7d361553d21b3291f41ae7
= References
Background, Motivation, and Internals of the Feature
- OpenZFS 2017 Presentation:
youtu.be/jj2IxRkl5bQ
- Slides:
slideshare.net/SerapheimNikolaosDim/zfs-log-spacemaps-project
Flushing Algorithm Internals & Performance Results
(Illumos Specific)
- Blogpost:
sdimitro.github.io/post/zfs-lsm-flushing/
- OpenZFS 2018 Presentation:
youtu.be/x6D2dHRjkxw
- Slides:
slideshare.net/SerapheimNikolaosDim/zfs-log-spacemap-flushing-algorithm
Upstream Delphix Issues:
DLPX-51539, DLPX-59659, DLPX-57783, DLPX-61438, DLPX-41227, DLPX-59320
DLPX-63385
Reviewed-by: Sean Eric Fagan <sef@ixsystems.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: George Wilson <gwilson@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8442
2019-07-16 17:11:49 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_max_log_walking\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The number of past TXGs that the flushing algorithm of the log spacemap
|
|
|
|
feature uses to estimate incoming log blocks.
|
|
|
|
.sp
|
|
|
|
Default value: \fB5\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_max_logsm_summary_length\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum number of rows allowed in the summary of the spacemap log.
|
|
|
|
.sp
|
|
|
|
Default value: \fB10\fR.
|
|
|
|
.RE
|
|
|
|
|
2014-11-03 20:15:08 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_max_recordsize\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
We currently support block sizes from 512 bytes to 16MB. The benefits of
|
2018-11-09 00:47:24 +00:00
|
|
|
larger blocks, and thus larger I/O, need to be weighed against the cost of
|
2014-11-03 20:15:08 +00:00
|
|
|
COWing a giant block to modify one byte. Additionally, very large blocks
|
|
|
|
can have an impact on i/o latency, and also potentially on the memory
|
|
|
|
allocator. Therefore, we do not allow the recordsize to be set larger than
|
|
|
|
zfs_max_recordsize (default 1MB). Larger blocks can be created by changing
|
|
|
|
this tunable, and pools with larger blocks can always be imported and used,
|
|
|
|
regardless of this setting.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1,048,576\fR.
|
|
|
|
.RE
|
|
|
|
|
Implement Redacted Send/Receive
Redacted send/receive allows users to send subsets of their data to
a target system. One possible use case for this feature is to not
transmit sensitive information to a data warehousing, test/dev, or
analytics environment. Another is to save space by not replicating
unimportant data within a given dataset, for example in backup tools
like zrepl.
Redacted send/receive is a three-stage process. First, a clone (or
clones) is made of the snapshot to be sent to the target. In this
clone (or clones), all unnecessary or unwanted data is removed or
modified. This clone is then snapshotted to create the "redaction
snapshot" (or snapshots). Second, the new zfs redact command is used
to create a redaction bookmark. The redaction bookmark stores the
list of blocks in a snapshot that were modified by the redaction
snapshot(s). Finally, the redaction bookmark is passed as a parameter
to zfs send. When sending to the snapshot that was redacted, the
redaction bookmark is used to filter out blocks that contain sensitive
or unwanted information, and those blocks are not included in the send
stream. When sending from the redaction bookmark, the blocks it
contains are considered as candidate blocks in addition to those
blocks in the destination snapshot that were modified since the
creation_txg of the redaction bookmark. This step is necessary to
allow the target to rehydrate data in the case where some blocks are
accidentally or unnecessarily modified in the redaction snapshot.
The changes to bookmarks to enable fast space estimation involve
adding deadlists to bookmarks. There is also logic to manage the
life cycles of these deadlists.
The new size estimation process operates in cases where previously
an accurate estimate could not be provided. In those cases, a send
is performed where no data blocks are read, reducing the runtime
significantly and providing a byte-accurate size estimate.
Reviewed-by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Prashanth Sreenivasa <pks@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Chris Williamson <chris.williamson@delphix.com>
Reviewed-by: Pavel Zhakarov <pavel.zakharov@delphix.com>
Reviewed-by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #7958
2019-06-19 16:48:13 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_allow_redacted_dataset_mount\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Allow datasets received with redacted send/receive to be mounted. Normally
|
|
|
|
disabled because these datasets may be missing key data.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
Log Spacemap Project
= Motivation
At Delphix we've seen a lot of customer systems where fragmentation
is over 75% and random writes take a performance hit because a lot
of time is spend on I/Os that update on-disk space accounting metadata.
Specifically, we seen cases where 20% to 40% of sync time is spend
after sync pass 1 and ~30% of the I/Os on the system is spent updating
spacemaps.
The problem is that these pools have existed long enough that we've
touched almost every metaslab at least once, and random writes
scatter frees across all metaslabs every TXG, thus appending to
their spacemaps and resulting in many I/Os. To give an example,
assuming that every VDEV has 200 metaslabs and our writes fit within
a single spacemap block (generally 4K) we have 200 I/Os. Then if we
assume 2 levels of indirection, we need 400 additional I/Os and
since we are talking about metadata for which we keep 2 extra copies
for redundancy we need to triple that number, leading to a total of
1800 I/Os per VDEV every TXG.
We could try and decrease the number of metaslabs so we have less
I/Os per TXG but then each metaslab would cover a wider range on
disk and thus would take more time to be loaded in memory from disk.
In addition, after it's loaded, it's range tree would consume more
memory.
Another idea would be to just increase the spacemap block size
which would allow us to fit more entries within an I/O block
resulting in fewer I/Os per metaslab and a speedup in loading time.
The problem is still that we don't deal with the number of I/Os
going up as the number of metaslabs is increasing and the fact
is that we generally write a lot to a few metaslabs and a little
to the rest of them. Thus, just increasing the block size would
actually waste bandwidth because we won't be utilizing our bigger
block size.
= About this patch
This patch introduces the Log Spacemap project which provides the
solution to the above problem while taking into account all the
aforementioned tradeoffs. The details on how it achieves that can
be found in the references sections below and in the code (see
Big Theory Statement in spa_log_spacemap.c).
Even though the change is fairly constraint within the metaslab
and lower-level SPA codepaths, there is a side-change that is
user-facing. The change is that VDEV IDs from VDEV holes will no
longer be reused. To give some background and reasoning for this,
when a log device is removed and its VDEV structure was replaced
with a hole (or was compacted; if at the end of the vdev array),
its vdev_id could be reused by devices added after that. Now
with the pool-wide space maps recording the vdev ID, this behavior
can cause problems (e.g. is this entry referring to a segment in
the new vdev or the removed log?). Thus, to simplify things the
ID reuse behavior is gone and now vdev IDs for top-level vdevs
are truly unique within a pool.
= Testing
The illumos implementation of this feature has been used internally
for a year and has been in production for ~6 months. For this patch
specifically there don't seem to be any regressions introduced to
ZTS and I have been running zloop for a week without any related
problems.
= Performance Analysis (Linux Specific)
All performance results and analysis for illumos can be found in
the links of the references. Redoing the same experiments in Linux
gave similar results. Below are the specifics of the Linux run.
After the pool reached stable state the percentage of the time
spent in pass 1 per TXG was 64% on average for the stock bits
while the log spacemap bits stayed at 95% during the experiment
(graph: sdimitro.github.io/img/linux-lsm/PercOfSyncInPassOne.png).
Sync times per TXG were 37.6 seconds on average for the stock
bits and 22.7 seconds for the log spacemap bits (related graph:
sdimitro.github.io/img/linux-lsm/SyncTimePerTXG.png). As a result
the log spacemap bits were able to push more TXGs, which is also
the reason why all graphs quantified per TXG have more entries for
the log spacemap bits.
Another interesting aspect in terms of txg syncs is that the stock
bits had 22% of their TXGs reach sync pass 7, 55% reach sync pass 8,
and 20% reach 9. The log space map bits reached sync pass 4 in 79%
of their TXGs, sync pass 7 in 19%, and sync pass 8 at 1%. This
emphasizes the fact that not only we spend less time on metadata
but we also iterate less times to convergence in spa_sync() dirtying
objects.
[related graphs:
stock- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGStock.png
lsm- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGLSM.png]
Finally, the improvement in IOPs that the userland gains from the
change is approximately 40%. There is a consistent win in IOPS as
you can see from the graphs below but the absolute amount of
improvement that the log spacemap gives varies within each minute
interval.
sdimitro.github.io/img/linux-lsm/StockVsLog3Days.png
sdimitro.github.io/img/linux-lsm/StockVsLog10Hours.png
= Porting to Other Platforms
For people that want to port this commit to other platforms below
is a list of ZoL commits that this patch depends on:
Make zdb results for checkpoint tests consistent
db587941c5ff6dea01932bb78f70db63cf7f38ba
Update vdev_is_spacemap_addressable() for new spacemap encoding
419ba5914552c6185afbe1dd17b3ed4b0d526547
Simplify spa_sync by breaking it up to smaller functions
8dc2197b7b1e4d7ebc1420ea30e51c6541f1d834
Factor metaslab_load_wait() in metaslab_load()
b194fab0fb6caad18711abccaff3c69ad8b3f6d3
Rename range_tree_verify to range_tree_verify_not_present
df72b8bebe0ebac0b20e0750984bad182cb6564a
Change target size of metaslabs from 256GB to 16GB
c853f382db731e15a87512f4ef1101d14d778a55
zdb -L should skip leak detection altogether
21e7cf5da89f55ce98ec1115726b150e19eefe89
vs_alloc can underflow in L2ARC vdevs
7558997d2f808368867ca7e5234e5793446e8f3f
Simplify log vdev removal code
6c926f426a26ffb6d7d8e563e33fc176164175cb
Get rid of space_map_update() for ms_synced_length
425d3237ee88abc53d8522a7139c926d278b4b7f
Introduce auxiliary metaslab histograms
928e8ad47d3478a3d5d01f0dd6ae74a9371af65e
Error path in metaslab_load_impl() forgets to drop ms_sync_lock
8eef997679ba54547f7d361553d21b3291f41ae7
= References
Background, Motivation, and Internals of the Feature
- OpenZFS 2017 Presentation:
youtu.be/jj2IxRkl5bQ
- Slides:
slideshare.net/SerapheimNikolaosDim/zfs-log-spacemaps-project
Flushing Algorithm Internals & Performance Results
(Illumos Specific)
- Blogpost:
sdimitro.github.io/post/zfs-lsm-flushing/
- OpenZFS 2018 Presentation:
youtu.be/x6D2dHRjkxw
- Slides:
slideshare.net/SerapheimNikolaosDim/zfs-log-spacemap-flushing-algorithm
Upstream Delphix Issues:
DLPX-51539, DLPX-59659, DLPX-57783, DLPX-61438, DLPX-41227, DLPX-59320
DLPX-63385
Reviewed-by: Sean Eric Fagan <sef@ixsystems.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: George Wilson <gwilson@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8442
2019-07-16 17:11:49 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_min_metaslabs_to_flush\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum number of metaslabs to flush per dirty TXG
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
2014-07-19 20:19:24 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_metaslab_fragmentation_threshold\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Allow metaslabs to keep their active state as long as their fragmentation
|
|
|
|
percentage is less than or equal to this value. An active metaslab that
|
|
|
|
exceeds this threshold will no longer keep its active status allowing
|
|
|
|
better metaslabs to be selected.
|
|
|
|
.sp
|
|
|
|
Default value: \fB70\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_mg_fragmentation_threshold\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Metaslab groups are considered eligible for allocations if their
|
2015-12-30 17:44:46 +00:00
|
|
|
fragmentation metric (measured as a percentage) is less than or equal to
|
2014-07-19 20:19:24 +00:00
|
|
|
this value. If a metaslab group exceeds this threshold then it will be
|
|
|
|
skipped unless all metaslab groups within the metaslab class have also
|
|
|
|
crossed this threshold.
|
|
|
|
.sp
|
2019-06-06 20:08:41 +00:00
|
|
|
Default value: \fB95\fR.
|
2014-07-19 20:19:24 +00:00
|
|
|
.RE
|
|
|
|
|
2014-07-10 03:36:03 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_mg_noalloc_threshold\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Defines a threshold at which metaslab groups should be eligible for
|
|
|
|
allocations. The value is expressed as a percentage of free space
|
|
|
|
beyond which a metaslab group is always eligible for allocations.
|
|
|
|
If a metaslab group's free space is less than or equal to the
|
2015-12-17 01:45:15 +00:00
|
|
|
threshold, the allocator will avoid allocating to that group
|
2014-07-10 03:36:03 +00:00
|
|
|
unless all groups in the pool have reached the threshold. Once all
|
|
|
|
groups have reached the threshold, all groups are allowed to accept
|
|
|
|
allocations. The default value of 0 disables the feature and causes
|
|
|
|
all metaslab groups to be eligible for allocations.
|
|
|
|
|
2017-08-10 22:45:25 +00:00
|
|
|
This parameter allows one to deal with pools having heavily imbalanced
|
2014-07-10 03:36:03 +00:00
|
|
|
vdevs such as would be the case when a new vdev has been added.
|
|
|
|
Setting the threshold to a non-zero percentage will stop allocations
|
|
|
|
from being made to vdevs that aren't filled to the specified percentage
|
|
|
|
and allow lesser filled vdevs to acquire more allocations than they
|
|
|
|
otherwise would under the old \fBzfs_mg_alloc_failures\fR facility.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2018-09-06 01:33:36 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_ddt_data_is_special\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
If enabled, ZFS will place DDT data into the special allocation class.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_user_indirect_is_special\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
If enabled, ZFS will place user data (both file and zvol) indirect blocks
|
|
|
|
into the special allocation class.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 03:20:35 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_multihost_history\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Historical statistics for the last N multihost updates will be available in
|
|
|
|
\fB/proc/spl/kstat/zfs/<pool>/multihost\fR
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_multihost_interval\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Used to control the frequency of multihost writes which are performed when the
|
MMP interval and fail_intervals in uberblock
When Multihost is enabled, and a pool is imported, uberblock writes
include ub_mmp_delay to allow an importing node to calculate the
duration of an activity test. This value, is not enough information.
If zfs_multihost_fail_intervals > 0 on the node with the pool imported,
the safe minimum duration of the activity test is well defined, but does
not depend on ub_mmp_delay:
zfs_multihost_fail_intervals * zfs_multihost_interval
and if zfs_multihost_fail_intervals == 0 on that node, there is no such
well defined safe duration, but the importing host cannot tell whether
mmp_delay is high due to I/O delays, or due to a very large
zfs_multihost_interval setting on the host which last imported the pool.
As a result, it may use a far longer period for the activity test than
is necessary.
This patch renames ub_mmp_sequence to ub_mmp_config and uses it to
record the zfs_multihost_interval and zfs_multihost_fail_intervals
values, as well as the mmp sequence. This allows a shorter activity
test duration to be calculated by the importing host in most situations.
These values are also added to the multihost_history kstat records.
It calculates the activity test duration differently depending on
whether the new fields are present or not; for importing pools with
only ub_mmp_delay, it uses
(zfs_multihost_interval + ub_mmp_delay) * zfs_multihost_import_intervals
Which results in an activity test duration less sensitive to the leaf
count.
In addition, it makes a few other improvements:
* It updates the "sequence" part of ub_mmp_config when MMP writes
in between syncs occur. This allows an importing host to detect MMP
on the remote host sooner, when the pool is idle, as it is not limited
to the granularity of ub_timestamp (1 second).
* It issues writes immediately when zfs_multihost_interval is changed
so remote hosts see the updated value as soon as possible.
* It fixes a bug where setting zfs_multihost_fail_intervals = 1 results
in immediate pool suspension.
* Update tests to verify activity check duration is based on recorded
tunable values, not tunable values on importing host.
* Update tests to verify the expected number of uberblocks have valid
MMP fields - fail_intervals, mmp_interval, mmp_seq (sequence number),
that sequence number is incrementing, and that uberblock values match
tunable settings.
Reviewed-by: Andreas Dilger <andreas.dilger@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #7842
2019-03-21 19:47:57 +00:00
|
|
|
\fBmultihost\fR pool property is on. This is one factor used to determine the
|
|
|
|
length of the activity check during import.
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 03:20:35 +00:00
|
|
|
.sp
|
MMP interval and fail_intervals in uberblock
When Multihost is enabled, and a pool is imported, uberblock writes
include ub_mmp_delay to allow an importing node to calculate the
duration of an activity test. This value, is not enough information.
If zfs_multihost_fail_intervals > 0 on the node with the pool imported,
the safe minimum duration of the activity test is well defined, but does
not depend on ub_mmp_delay:
zfs_multihost_fail_intervals * zfs_multihost_interval
and if zfs_multihost_fail_intervals == 0 on that node, there is no such
well defined safe duration, but the importing host cannot tell whether
mmp_delay is high due to I/O delays, or due to a very large
zfs_multihost_interval setting on the host which last imported the pool.
As a result, it may use a far longer period for the activity test than
is necessary.
This patch renames ub_mmp_sequence to ub_mmp_config and uses it to
record the zfs_multihost_interval and zfs_multihost_fail_intervals
values, as well as the mmp sequence. This allows a shorter activity
test duration to be calculated by the importing host in most situations.
These values are also added to the multihost_history kstat records.
It calculates the activity test duration differently depending on
whether the new fields are present or not; for importing pools with
only ub_mmp_delay, it uses
(zfs_multihost_interval + ub_mmp_delay) * zfs_multihost_import_intervals
Which results in an activity test duration less sensitive to the leaf
count.
In addition, it makes a few other improvements:
* It updates the "sequence" part of ub_mmp_config when MMP writes
in between syncs occur. This allows an importing host to detect MMP
on the remote host sooner, when the pool is idle, as it is not limited
to the granularity of ub_timestamp (1 second).
* It issues writes immediately when zfs_multihost_interval is changed
so remote hosts see the updated value as soon as possible.
* It fixes a bug where setting zfs_multihost_fail_intervals = 1 results
in immediate pool suspension.
* Update tests to verify activity check duration is based on recorded
tunable values, not tunable values on importing host.
* Update tests to verify the expected number of uberblocks have valid
MMP fields - fail_intervals, mmp_interval, mmp_seq (sequence number),
that sequence number is incrementing, and that uberblock values match
tunable settings.
Reviewed-by: Andreas Dilger <andreas.dilger@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #7842
2019-03-21 19:47:57 +00:00
|
|
|
The multihost write period is \fBzfs_multihost_interval / leaf-vdevs\fR
|
|
|
|
milliseconds. On average a multihost write will be issued for each leaf vdev
|
|
|
|
every \fBzfs_multihost_interval\fR milliseconds. In practice, the observed
|
|
|
|
period can vary with the I/O load and this observed value is the delay which is
|
|
|
|
stored in the uberblock.
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 03:20:35 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB1000\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_multihost_import_intervals\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Used to control the duration of the activity test on import. Smaller values of
|
|
|
|
\fBzfs_multihost_import_intervals\fR will reduce the import time but increase
|
|
|
|
the risk of failing to detect an active pool. The total activity check time is
|
MMP interval and fail_intervals in uberblock
When Multihost is enabled, and a pool is imported, uberblock writes
include ub_mmp_delay to allow an importing node to calculate the
duration of an activity test. This value, is not enough information.
If zfs_multihost_fail_intervals > 0 on the node with the pool imported,
the safe minimum duration of the activity test is well defined, but does
not depend on ub_mmp_delay:
zfs_multihost_fail_intervals * zfs_multihost_interval
and if zfs_multihost_fail_intervals == 0 on that node, there is no such
well defined safe duration, but the importing host cannot tell whether
mmp_delay is high due to I/O delays, or due to a very large
zfs_multihost_interval setting on the host which last imported the pool.
As a result, it may use a far longer period for the activity test than
is necessary.
This patch renames ub_mmp_sequence to ub_mmp_config and uses it to
record the zfs_multihost_interval and zfs_multihost_fail_intervals
values, as well as the mmp sequence. This allows a shorter activity
test duration to be calculated by the importing host in most situations.
These values are also added to the multihost_history kstat records.
It calculates the activity test duration differently depending on
whether the new fields are present or not; for importing pools with
only ub_mmp_delay, it uses
(zfs_multihost_interval + ub_mmp_delay) * zfs_multihost_import_intervals
Which results in an activity test duration less sensitive to the leaf
count.
In addition, it makes a few other improvements:
* It updates the "sequence" part of ub_mmp_config when MMP writes
in between syncs occur. This allows an importing host to detect MMP
on the remote host sooner, when the pool is idle, as it is not limited
to the granularity of ub_timestamp (1 second).
* It issues writes immediately when zfs_multihost_interval is changed
so remote hosts see the updated value as soon as possible.
* It fixes a bug where setting zfs_multihost_fail_intervals = 1 results
in immediate pool suspension.
* Update tests to verify activity check duration is based on recorded
tunable values, not tunable values on importing host.
* Update tests to verify the expected number of uberblocks have valid
MMP fields - fail_intervals, mmp_interval, mmp_seq (sequence number),
that sequence number is incrementing, and that uberblock values match
tunable settings.
Reviewed-by: Andreas Dilger <andreas.dilger@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #7842
2019-03-21 19:47:57 +00:00
|
|
|
never allowed to drop below one second.
|
|
|
|
.sp
|
|
|
|
On import the activity check waits a minimum amount of time determined by
|
|
|
|
\fBzfs_multihost_interval * zfs_multihost_import_intervals\fR, or the same
|
|
|
|
product computed on the host which last had the pool imported (whichever is
|
|
|
|
greater). The activity check time may be further extended if the value of mmp
|
|
|
|
delay found in the best uberblock indicates actual multihost updates happened
|
|
|
|
at longer intervals than \fBzfs_multihost_interval\fR. A minimum value of
|
|
|
|
\fB100ms\fR is enforced.
|
|
|
|
.sp
|
|
|
|
A value of 0 is ignored and treated as if it was set to 1.
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 03:20:35 +00:00
|
|
|
.sp
|
Increase default zfs_multihost_fail_intervals and import_intervals
By default, when multihost is enabled for a pool, the pool is
suspended if (zfs_multihost_fail_intervals*zfs_multihost_interval) ms
pass without a successful MMP write. This is the recommended
configuration.
The default value for zfs_multihost_fail_intervals has been 5, and the
default value for zfs_multihost_interval has been 1000, so pool
suspension occurred at 5 seconds.
There have been multiple cases where a single misbehaving device in a
pool triggered a SCSI reset, and all I/O paused for 5-6 seconds. This
in turn caused MMP to suspend the pool.
In the cases observed, the rest of the devices were healthy and the
pool was otherwise correctly performing I/O. The reset was handled
correctly by ZFS, and by suspending the pool MMP made replacing the
device more difficult as well as forcing the host to be rebooted.
Increase the default value of zfs_multihost_fail_intervals to 10, so
that MMP tolerates up to 10 seconds of failed MMP writes before
suspending the pool.
Increase the default value of zfs_multihost_import_intervals to 20, to
maintain the 2:1 safety factor. This results in a force import taking
approximately 20 seconds when MMP is enabled, with default values.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@whamcloud.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #7709
Closes #8495
2019-03-13 16:50:48 +00:00
|
|
|
Default value: \fB20\fR.
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 03:20:35 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_multihost_fail_intervals\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
MMP interval and fail_intervals in uberblock
When Multihost is enabled, and a pool is imported, uberblock writes
include ub_mmp_delay to allow an importing node to calculate the
duration of an activity test. This value, is not enough information.
If zfs_multihost_fail_intervals > 0 on the node with the pool imported,
the safe minimum duration of the activity test is well defined, but does
not depend on ub_mmp_delay:
zfs_multihost_fail_intervals * zfs_multihost_interval
and if zfs_multihost_fail_intervals == 0 on that node, there is no such
well defined safe duration, but the importing host cannot tell whether
mmp_delay is high due to I/O delays, or due to a very large
zfs_multihost_interval setting on the host which last imported the pool.
As a result, it may use a far longer period for the activity test than
is necessary.
This patch renames ub_mmp_sequence to ub_mmp_config and uses it to
record the zfs_multihost_interval and zfs_multihost_fail_intervals
values, as well as the mmp sequence. This allows a shorter activity
test duration to be calculated by the importing host in most situations.
These values are also added to the multihost_history kstat records.
It calculates the activity test duration differently depending on
whether the new fields are present or not; for importing pools with
only ub_mmp_delay, it uses
(zfs_multihost_interval + ub_mmp_delay) * zfs_multihost_import_intervals
Which results in an activity test duration less sensitive to the leaf
count.
In addition, it makes a few other improvements:
* It updates the "sequence" part of ub_mmp_config when MMP writes
in between syncs occur. This allows an importing host to detect MMP
on the remote host sooner, when the pool is idle, as it is not limited
to the granularity of ub_timestamp (1 second).
* It issues writes immediately when zfs_multihost_interval is changed
so remote hosts see the updated value as soon as possible.
* It fixes a bug where setting zfs_multihost_fail_intervals = 1 results
in immediate pool suspension.
* Update tests to verify activity check duration is based on recorded
tunable values, not tunable values on importing host.
* Update tests to verify the expected number of uberblocks have valid
MMP fields - fail_intervals, mmp_interval, mmp_seq (sequence number),
that sequence number is incrementing, and that uberblock values match
tunable settings.
Reviewed-by: Andreas Dilger <andreas.dilger@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #7842
2019-03-21 19:47:57 +00:00
|
|
|
Controls the behavior of the pool when multihost write failures or delays are
|
|
|
|
detected.
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 03:20:35 +00:00
|
|
|
.sp
|
MMP interval and fail_intervals in uberblock
When Multihost is enabled, and a pool is imported, uberblock writes
include ub_mmp_delay to allow an importing node to calculate the
duration of an activity test. This value, is not enough information.
If zfs_multihost_fail_intervals > 0 on the node with the pool imported,
the safe minimum duration of the activity test is well defined, but does
not depend on ub_mmp_delay:
zfs_multihost_fail_intervals * zfs_multihost_interval
and if zfs_multihost_fail_intervals == 0 on that node, there is no such
well defined safe duration, but the importing host cannot tell whether
mmp_delay is high due to I/O delays, or due to a very large
zfs_multihost_interval setting on the host which last imported the pool.
As a result, it may use a far longer period for the activity test than
is necessary.
This patch renames ub_mmp_sequence to ub_mmp_config and uses it to
record the zfs_multihost_interval and zfs_multihost_fail_intervals
values, as well as the mmp sequence. This allows a shorter activity
test duration to be calculated by the importing host in most situations.
These values are also added to the multihost_history kstat records.
It calculates the activity test duration differently depending on
whether the new fields are present or not; for importing pools with
only ub_mmp_delay, it uses
(zfs_multihost_interval + ub_mmp_delay) * zfs_multihost_import_intervals
Which results in an activity test duration less sensitive to the leaf
count.
In addition, it makes a few other improvements:
* It updates the "sequence" part of ub_mmp_config when MMP writes
in between syncs occur. This allows an importing host to detect MMP
on the remote host sooner, when the pool is idle, as it is not limited
to the granularity of ub_timestamp (1 second).
* It issues writes immediately when zfs_multihost_interval is changed
so remote hosts see the updated value as soon as possible.
* It fixes a bug where setting zfs_multihost_fail_intervals = 1 results
in immediate pool suspension.
* Update tests to verify activity check duration is based on recorded
tunable values, not tunable values on importing host.
* Update tests to verify the expected number of uberblocks have valid
MMP fields - fail_intervals, mmp_interval, mmp_seq (sequence number),
that sequence number is incrementing, and that uberblock values match
tunable settings.
Reviewed-by: Andreas Dilger <andreas.dilger@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #7842
2019-03-21 19:47:57 +00:00
|
|
|
When \fBzfs_multihost_fail_intervals = 0\fR, multihost write failures or delays
|
|
|
|
are ignored. The failures will still be reported to the ZED which depending on
|
|
|
|
its configuration may take action such as suspending the pool or offlining a
|
|
|
|
device.
|
|
|
|
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 03:20:35 +00:00
|
|
|
.sp
|
MMP interval and fail_intervals in uberblock
When Multihost is enabled, and a pool is imported, uberblock writes
include ub_mmp_delay to allow an importing node to calculate the
duration of an activity test. This value, is not enough information.
If zfs_multihost_fail_intervals > 0 on the node with the pool imported,
the safe minimum duration of the activity test is well defined, but does
not depend on ub_mmp_delay:
zfs_multihost_fail_intervals * zfs_multihost_interval
and if zfs_multihost_fail_intervals == 0 on that node, there is no such
well defined safe duration, but the importing host cannot tell whether
mmp_delay is high due to I/O delays, or due to a very large
zfs_multihost_interval setting on the host which last imported the pool.
As a result, it may use a far longer period for the activity test than
is necessary.
This patch renames ub_mmp_sequence to ub_mmp_config and uses it to
record the zfs_multihost_interval and zfs_multihost_fail_intervals
values, as well as the mmp sequence. This allows a shorter activity
test duration to be calculated by the importing host in most situations.
These values are also added to the multihost_history kstat records.
It calculates the activity test duration differently depending on
whether the new fields are present or not; for importing pools with
only ub_mmp_delay, it uses
(zfs_multihost_interval + ub_mmp_delay) * zfs_multihost_import_intervals
Which results in an activity test duration less sensitive to the leaf
count.
In addition, it makes a few other improvements:
* It updates the "sequence" part of ub_mmp_config when MMP writes
in between syncs occur. This allows an importing host to detect MMP
on the remote host sooner, when the pool is idle, as it is not limited
to the granularity of ub_timestamp (1 second).
* It issues writes immediately when zfs_multihost_interval is changed
so remote hosts see the updated value as soon as possible.
* It fixes a bug where setting zfs_multihost_fail_intervals = 1 results
in immediate pool suspension.
* Update tests to verify activity check duration is based on recorded
tunable values, not tunable values on importing host.
* Update tests to verify the expected number of uberblocks have valid
MMP fields - fail_intervals, mmp_interval, mmp_seq (sequence number),
that sequence number is incrementing, and that uberblock values match
tunable settings.
Reviewed-by: Andreas Dilger <andreas.dilger@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #7842
2019-03-21 19:47:57 +00:00
|
|
|
When \fBzfs_multihost_fail_intervals > 0\fR, the pool will be suspended if
|
|
|
|
\fBzfs_multihost_fail_intervals * zfs_multihost_interval\fR milliseconds pass
|
|
|
|
without a successful mmp write. This guarantees the activity test will see
|
|
|
|
mmp writes if the pool is imported. A value of 1 is ignored and treated as
|
|
|
|
if it was set to 2. This is necessary to prevent the pool from being suspended
|
|
|
|
due to normal, small I/O latency variations.
|
|
|
|
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 03:20:35 +00:00
|
|
|
.sp
|
Increase default zfs_multihost_fail_intervals and import_intervals
By default, when multihost is enabled for a pool, the pool is
suspended if (zfs_multihost_fail_intervals*zfs_multihost_interval) ms
pass without a successful MMP write. This is the recommended
configuration.
The default value for zfs_multihost_fail_intervals has been 5, and the
default value for zfs_multihost_interval has been 1000, so pool
suspension occurred at 5 seconds.
There have been multiple cases where a single misbehaving device in a
pool triggered a SCSI reset, and all I/O paused for 5-6 seconds. This
in turn caused MMP to suspend the pool.
In the cases observed, the rest of the devices were healthy and the
pool was otherwise correctly performing I/O. The reset was handled
correctly by ZFS, and by suspending the pool MMP made replacing the
device more difficult as well as forcing the host to be rebooted.
Increase the default value of zfs_multihost_fail_intervals to 10, so
that MMP tolerates up to 10 seconds of failed MMP writes before
suspending the pool.
Increase the default value of zfs_multihost_import_intervals to 20, to
maintain the 2:1 safety factor. This results in a force import taking
approximately 20 seconds when MMP is enabled, with default values.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@whamcloud.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #7709
Closes #8495
2019-03-13 16:50:48 +00:00
|
|
|
Default value: \fB10\fR.
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 03:20:35 +00:00
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_no_scrub_io\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Set for no scrub I/O. This results in scrubs not actually scrubbing data and
|
|
|
|
simply doing a metadata crawl of the pool instead.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_no_scrub_prefetch\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Set to disable block prefetching for scrubs.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_nocacheflush\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2018-10-23 16:44:37 +00:00
|
|
|
Disable cache flush operations on disks when writing. Setting this will
|
|
|
|
cause pool corruption on power loss if a volatile out-of-order write cache
|
|
|
|
is enabled.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_nopwrite_enabled\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Enable NOP writes
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes (default) and \fB0\fR to disable.
|
|
|
|
.RE
|
|
|
|
|
2017-03-24 21:28:38 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_dmu_offset_next_sync\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Enable forcing txg sync to find holes. When enabled forces ZFS to act
|
|
|
|
like prior versions when SEEK_HOLE or SEEK_DATA flags are used, which
|
|
|
|
when a dnode is dirty causes txg's to be synced so that this data can be
|
|
|
|
found.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR to disable (default).
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2015-03-27 04:31:52 +00:00
|
|
|
\fBzfs_pd_bytes_max\fR (int)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
The number of bytes which should be prefetched during a pool traversal
|
2016-11-15 01:03:57 +00:00
|
|
|
(eg: \fBzfs send\fR or other data crawling operations)
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2015-03-31 18:51:37 +00:00
|
|
|
Default value: \fB52,428,800\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
2017-02-07 17:44:03 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_per_txg_dirty_frees_percent \fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2019-02-12 20:01:08 +00:00
|
|
|
Tunable to control percentage of dirtied indirect blocks from frees allowed
|
|
|
|
into one TXG. After this threshold is crossed, additional frees will wait until
|
|
|
|
the next TXG.
|
2017-02-07 17:44:03 +00:00
|
|
|
A value of zero will disable this throttle.
|
|
|
|
.sp
|
2019-02-12 20:01:08 +00:00
|
|
|
Default value: \fB5\fR, set to \fB0\fR to disable.
|
2017-02-07 17:44:03 +00:00
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_prefetch_disable\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-26 21:10:31 +00:00
|
|
|
This tunable disables predictive prefetch. Note that it leaves "prescient"
|
|
|
|
prefetch (e.g. prefetch for zfs send) intact. Unlike predictive prefetch,
|
|
|
|
prescient prefetch never issues i/os that end up not being needed, so it
|
|
|
|
can't hurt performance.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
2019-04-16 19:38:36 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_qat_checksum_disable\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
This tunable disables qat hardware acceleration for sha256 checksums. It
|
|
|
|
may be set after the zfs modules have been loaded to initialize the qat
|
|
|
|
hardware as long as support is compiled in and the qat driver is present.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_qat_compress_disable\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
This tunable disables qat hardware acceleration for gzip compression. It
|
|
|
|
may be set after the zfs modules have been loaded to initialize the qat
|
|
|
|
hardware as long as support is compiled in and the qat driver is present.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_qat_encrypt_disable\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
This tunable disables qat hardware acceleration for AES-GCM encryption. It
|
|
|
|
may be set after the zfs modules have been loaded to initialize the qat
|
|
|
|
hardware as long as support is compiled in and the qat driver is present.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_read_chunk_size\fR (long)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Bytes to read per chunk
|
|
|
|
.sp
|
|
|
|
Default value: \fB1,048,576\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_read_history\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 03:20:35 +00:00
|
|
|
Historical statistics for the last N reads will be available in
|
|
|
|
\fB/proc/spl/kstat/zfs/<pool>/reads\fR
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2015-12-30 17:44:46 +00:00
|
|
|
Default value: \fB0\fR (no data is kept).
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_read_history_hits\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Include cache hits in read history
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
2020-07-03 18:05:50 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_rebuild_max_segment\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum read segment size to issue when sequentially resilvering a
|
|
|
|
top-level vdev.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1,048,576\fR.
|
|
|
|
.RE
|
|
|
|
|
Distributed Spare (dRAID) Feature
This patch adds a new top-level vdev type called dRAID, which stands
for Distributed parity RAID. This pool configuration allows all dRAID
vdevs to participate when rebuilding to a distributed hot spare device.
This can substantially reduce the total time required to restore full
parity to pool with a failed device.
A dRAID pool can be created using the new top-level `draid` type.
Like `raidz`, the desired redundancy is specified after the type:
`draid[1,2,3]`. No additional information is required to create the
pool and reasonable default values will be chosen based on the number
of child vdevs in the dRAID vdev.
zpool create <pool> draid[1,2,3] <vdevs...>
Unlike raidz, additional optional dRAID configuration values can be
provided as part of the draid type as colon separated values. This
allows administrators to fully specify a layout for either performance
or capacity reasons. The supported options include:
zpool create <pool> \
draid[<parity>][:<data>d][:<children>c][:<spares>s] \
<vdevs...>
- draid[parity] - Parity level (default 1)
- draid[:<data>d] - Data devices per group (default 8)
- draid[:<children>c] - Expected number of child vdevs
- draid[:<spares>s] - Distributed hot spares (default 0)
Abbreviated example `zpool status` output for a 68 disk dRAID pool
with two distributed spares using special allocation classes.
```
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
slag7 ONLINE 0 0 0
draid2:8d:68c:2s-0 ONLINE 0 0 0
L0 ONLINE 0 0 0
L1 ONLINE 0 0 0
...
U25 ONLINE 0 0 0
U26 ONLINE 0 0 0
spare-53 ONLINE 0 0 0
U27 ONLINE 0 0 0
draid2-0-0 ONLINE 0 0 0
U28 ONLINE 0 0 0
U29 ONLINE 0 0 0
...
U42 ONLINE 0 0 0
U43 ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
L5 ONLINE 0 0 0
U5 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
L6 ONLINE 0 0 0
U6 ONLINE 0 0 0
spares
draid2-0-0 INUSE currently in use
draid2-0-1 AVAIL
```
When adding test coverage for the new dRAID vdev type the following
options were added to the ztest command. These options are leverages
by zloop.sh to test a wide range of dRAID configurations.
-K draid|raidz|random - kind of RAID to test
-D <value> - dRAID data drives per group
-S <value> - dRAID distributed hot spares
-R <value> - RAID parity (raidz or dRAID)
The zpool_create, zpool_import, redundancy, replacement and fault
test groups have all been updated provide test coverage for the
dRAID feature.
Co-authored-by: Isaac Huang <he.huang@intel.com>
Co-authored-by: Mark Maybee <mmaybee@cray.com>
Co-authored-by: Don Brady <don.brady@delphix.com>
Co-authored-by: Matthew Ahrens <mahrens@delphix.com>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Mark Maybee <mmaybee@cray.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10102
2020-11-13 21:51:51 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_rebuild_scrub_enabled\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Automatically start a pool scrub when the last active sequential resilver
|
|
|
|
completes in order to verify the checksums of all blocks which have been
|
|
|
|
resilvered. This option is enabled by default and is strongly recommended.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_rebuild_vdev_limit\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum amount of i/o that can be concurrently issued for a sequential
|
|
|
|
resilver per leaf device, given in bytes.
|
|
|
|
.sp
|
|
|
|
Default value: \fB33,554,432\fR.
|
|
|
|
.RE
|
|
|
|
|
OpenZFS 9290 - device removal reduces redundancy of mirrors
Mirrors are supposed to provide redundancy in the face of whole-disk
failure and silent damage (e.g. some data on disk is not right, but ZFS
hasn't detected the whole device as being broken). However, the current
device removal implementation bypasses some of the mirror's redundancy.
Note that in no case is incorrect data returned, but we might get a
checksum error when we should have been able to find the right data.
There are two underlying problems:
1. When we remove a mirror device, we only read one side of the mirror.
Since we can't verify the checksum, this side may be silently bad, but
the good data is on the other side of the mirror (which we didn't read).
This can cause the removal to "bake in" the busted data – all copies of
the data in the new location are the same, busted version, while we left
the good version behind.
The fix for this is to read and copy both sides of the mirror. If the
old and new vdevs are mirrors, we will read both sides of the old
mirror, and write each copy to the corresponding side of the new mirror.
(If the old and new vdevs have a different number of children, we will
do this as best as possible.) Even though we aren't verifying checksums,
this ensures that as long as there's a good copy of the data, we'll have
a good copy after the removal, even if there's silent damage to one side
of the mirror. If we're removing a mirror that has some silent damage,
we'll have exactly the same damage in the new location (assuming that
the new location is also a mirror).
2. When we read from an indirect vdev that points to a mirror vdev, we
only consider one copy of the data. This can lead to reduced effective
redundancy, because we might read a bad copy of the data from one side
of the mirror, and not retry the other, good side of the mirror.
Note that the problem is not with the removal process, but rather after
the removal has completed (having copied correct data to both sides of
the mirror), if one side of the new mirror is silently damaged, we
encounter the problem when reading the relocated data via the indirect
vdev. Also note that the problem doesn't occur when ZFS knows that one
side of the mirror is bad, e.g. when a disk entirely fails or is
offlined.
The impact is that reads (from indirect vdevs that point to mirrors) may
return a checksum error even though the good data exists on one side of
the mirror, and scrub doesn't repair all data on the mirror (if some of
it is pointed to via an indirect vdev).
The fix for this is complicated by "split blocks" - one logical block
may be split into two (or more) pieces with each piece moved to a
different new location. In this case we need to read all versions of
each split (one from each side of the mirror), and figure out which
combination of versions results in the correct checksum, and then repair
the incorrect versions.
This ensures that we supply the same redundancy whether you use device
removal or not. For example, if a mirror has small silent errors on all
of its children, we can still reconstruct the correct data, as long as
those errors are at sufficiently-separated offsets (specifically,
separated by the largest block size - default of 128KB, but up to 16MB).
Porting notes:
* A new indirect vdev check was moved from dsl_scan_needs_resilver_cb()
to dsl_scan_needs_resilver(), which was added to ZoL as part of the
sequential scrub work.
* Passed NULL for zfs_ereport_post_checksum()'s zbookmark_phys_t
parameter. The extra parameter is unique to ZoL.
* When posting indirect checksum errors the ABD can be passed directly,
zfs_ereport_post_checksum() is not yet ABD-aware in OpenZFS.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9290
OpenZFS-commit: https://github.com/openzfs/openzfs/pull/591
Closes #6900
2018-02-13 19:37:56 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2018-03-29 21:50:40 +00:00
|
|
|
\fBzfs_reconstruct_indirect_combinations_max\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12na
|
|
|
|
If an indirect split block contains more than this many possible unique
|
|
|
|
combinations when being reconstructed, consider it too computationally
|
|
|
|
expensive to check them all. Instead, try at most
|
|
|
|
\fBzfs_reconstruct_indirect_combinations_max\fR randomly-selected
|
|
|
|
combinations each time the block is accessed. This allows all segment
|
|
|
|
copies to participate fairly in the reconstruction when all combinations
|
|
|
|
cannot be checked and prevents repeated use of one bad copy.
|
|
|
|
.sp
|
2019-01-16 22:10:02 +00:00
|
|
|
Default value: \fB4096\fR.
|
OpenZFS 9290 - device removal reduces redundancy of mirrors
Mirrors are supposed to provide redundancy in the face of whole-disk
failure and silent damage (e.g. some data on disk is not right, but ZFS
hasn't detected the whole device as being broken). However, the current
device removal implementation bypasses some of the mirror's redundancy.
Note that in no case is incorrect data returned, but we might get a
checksum error when we should have been able to find the right data.
There are two underlying problems:
1. When we remove a mirror device, we only read one side of the mirror.
Since we can't verify the checksum, this side may be silently bad, but
the good data is on the other side of the mirror (which we didn't read).
This can cause the removal to "bake in" the busted data – all copies of
the data in the new location are the same, busted version, while we left
the good version behind.
The fix for this is to read and copy both sides of the mirror. If the
old and new vdevs are mirrors, we will read both sides of the old
mirror, and write each copy to the corresponding side of the new mirror.
(If the old and new vdevs have a different number of children, we will
do this as best as possible.) Even though we aren't verifying checksums,
this ensures that as long as there's a good copy of the data, we'll have
a good copy after the removal, even if there's silent damage to one side
of the mirror. If we're removing a mirror that has some silent damage,
we'll have exactly the same damage in the new location (assuming that
the new location is also a mirror).
2. When we read from an indirect vdev that points to a mirror vdev, we
only consider one copy of the data. This can lead to reduced effective
redundancy, because we might read a bad copy of the data from one side
of the mirror, and not retry the other, good side of the mirror.
Note that the problem is not with the removal process, but rather after
the removal has completed (having copied correct data to both sides of
the mirror), if one side of the new mirror is silently damaged, we
encounter the problem when reading the relocated data via the indirect
vdev. Also note that the problem doesn't occur when ZFS knows that one
side of the mirror is bad, e.g. when a disk entirely fails or is
offlined.
The impact is that reads (from indirect vdevs that point to mirrors) may
return a checksum error even though the good data exists on one side of
the mirror, and scrub doesn't repair all data on the mirror (if some of
it is pointed to via an indirect vdev).
The fix for this is complicated by "split blocks" - one logical block
may be split into two (or more) pieces with each piece moved to a
different new location. In this case we need to read all versions of
each split (one from each side of the mirror), and figure out which
combination of versions results in the correct checksum, and then repair
the incorrect versions.
This ensures that we supply the same redundancy whether you use device
removal or not. For example, if a mirror has small silent errors on all
of its children, we can still reconstruct the correct data, as long as
those errors are at sufficiently-separated offsets (specifically,
separated by the largest block size - default of 128KB, but up to 16MB).
Porting notes:
* A new indirect vdev check was moved from dsl_scan_needs_resilver_cb()
to dsl_scan_needs_resilver(), which was added to ZoL as part of the
sequential scrub work.
* Passed NULL for zfs_ereport_post_checksum()'s zbookmark_phys_t
parameter. The extra parameter is unique to ZoL.
* When posting indirect checksum errors the ABD can be passed directly,
zfs_ereport_post_checksum() is not yet ABD-aware in OpenZFS.
Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://illumos.org/issues/9290
OpenZFS-commit: https://github.com/openzfs/openzfs/pull/591
Closes #6900
2018-02-13 19:37:56 +00:00
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_recover\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Set to attempt to recover from fatal errors. This should only be used as a
|
|
|
|
last resort, as it typically results in leaked space, or worse.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
2018-12-04 17:37:37 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_removal_ignore_errors\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
.sp
|
|
|
|
Ignore hard IO errors during device removal. When set, if a device encounters
|
|
|
|
a hard IO error during the removal process the removal will not be cancelled.
|
|
|
|
This can result in a normally recoverable block becoming permanently damaged
|
|
|
|
and is not recommended. This should only be used as a last resort when the
|
|
|
|
pool cannot be returned to a healthy state prior to removing the device.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2019-06-13 20:12:39 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_removal_suspend_progress\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
.sp
|
|
|
|
This is used by the test suite so that it can ensure that certain actions
|
|
|
|
happen while in the middle of a removal.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_remove_max_segment\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
.sp
|
|
|
|
The largest contiguous segment that we will attempt to allocate when removing
|
|
|
|
a device. This can be no larger than 16MB. If there is a performance
|
|
|
|
problem with attempting to allocate large blocks, consider decreasing this.
|
|
|
|
.sp
|
|
|
|
Default value: \fB16,777,216\fR (16MB).
|
|
|
|
.RE
|
|
|
|
|
2020-01-07 20:41:53 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_resilver_disable_defer\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Disables the \fBresilver_defer\fR feature, causing an operation that would
|
|
|
|
start a resilver to restart one in progress immediately.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR (feature enabled).
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2017-11-16 01:27:01 +00:00
|
|
|
\fBzfs_resilver_min_time_ms\fR (int)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-11-16 01:27:01 +00:00
|
|
|
Resilvers are processed by the sync thread. While resilvering it will spend
|
|
|
|
at least this much time working on a resilver between txg flushes.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2017-11-16 01:27:01 +00:00
|
|
|
Default value: \fB3,000\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
2018-03-13 17:43:14 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_scan_ignore_errors\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
If set to a nonzero value, remove the DTL (dirty time list) upon
|
|
|
|
completion of a pool scan (scrub) even if there were unrepairable
|
|
|
|
errors. It is intended to be used during pool repair or recovery to
|
|
|
|
stop resilvering when the pool is next imported.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2017-11-16 01:27:01 +00:00
|
|
|
\fBzfs_scrub_min_time_ms\fR (int)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-11-16 01:27:01 +00:00
|
|
|
Scrubs are processed by the sync thread. While scrubbing it will spend
|
|
|
|
at least this much time working on a scrub between txg flushes.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2017-11-16 01:27:01 +00:00
|
|
|
Default value: \fB1,000\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2017-11-16 01:27:01 +00:00
|
|
|
\fBzfs_scan_checkpoint_intval\fR (int)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-11-16 01:27:01 +00:00
|
|
|
To preserve progress across reboots the sequential scan algorithm periodically
|
|
|
|
needs to stop metadata scanning and issue all the verifications I/Os to disk.
|
|
|
|
The frequency of this flushing is determined by the
|
2018-06-25 16:50:01 +00:00
|
|
|
\fBzfs_scan_checkpoint_intval\fR tunable.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2017-11-16 01:27:01 +00:00
|
|
|
Default value: \fB7200\fR seconds (every 2 hours).
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2017-11-16 01:27:01 +00:00
|
|
|
\fBzfs_scan_fill_weight\fR (int)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-11-16 01:27:01 +00:00
|
|
|
This tunable affects how scrub and resilver I/O segments are ordered. A higher
|
|
|
|
number indicates that we care more about how filled in a segment is, while a
|
|
|
|
lower number indicates we care more about the size of the extent without
|
|
|
|
considering the gaps within a segment. This value is only tunable upon module
|
|
|
|
insertion. Changing the value afterwards will have no affect on scrub or
|
|
|
|
resilver performance.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2017-11-16 01:27:01 +00:00
|
|
|
Default value: \fB3\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2017-11-16 01:27:01 +00:00
|
|
|
\fBzfs_scan_issue_strategy\fR (int)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-11-16 01:27:01 +00:00
|
|
|
Determines the order that data will be verified while scrubbing or resilvering.
|
|
|
|
If set to \fB1\fR, data will be verified as sequentially as possible, given the
|
|
|
|
amount of memory reserved for scrubbing (see \fBzfs_scan_mem_lim_fact\fR). This
|
|
|
|
may improve scrub performance if the pool's data is very fragmented. If set to
|
|
|
|
\fB2\fR, the largest mostly-contiguous chunk of found data will be verified
|
|
|
|
first. By deferring scrubbing of small segments, we may later find adjacent data
|
|
|
|
to coalesce and increase the segment size. If set to \fB0\fR, zfs will use
|
|
|
|
strategy \fB1\fR during normal verification and strategy \fB2\fR while taking a
|
|
|
|
checkpoint.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2017-11-16 01:27:01 +00:00
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_scan_legacy\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
A value of 0 indicates that scrubs and resilvers will gather metadata in
|
|
|
|
memory before issuing sequential I/O. A value of 1 indicates that the legacy
|
|
|
|
algorithm will be used where I/O is initiated as soon as it is discovered.
|
|
|
|
Changing this value to 0 will not affect scrubs or resilvers that are already
|
|
|
|
in progress.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_scan_max_ext_gap\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Indicates the largest gap in bytes between scrub / resilver I/Os that will still
|
|
|
|
be considered sequential for sorting purposes. Changing this value will not
|
|
|
|
affect scrubs or resilvers that are already in progress.
|
|
|
|
.sp
|
|
|
|
Default value: \fB2097152 (2 MB)\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_scan_mem_lim_fact\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum fraction of RAM used for I/O sorting by sequential scan algorithm.
|
|
|
|
This tunable determines the hard limit for I/O sorting memory usage.
|
|
|
|
When the hard limit is reached we stop scanning metadata and start issuing
|
|
|
|
data verification I/O. This is done until we get below the soft limit.
|
|
|
|
.sp
|
|
|
|
Default value: \fB20\fR which is 5% of RAM (1/20).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_scan_mem_lim_soft_fact\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The fraction of the hard limit used to determined the soft limit for I/O sorting
|
2019-08-30 16:41:35 +00:00
|
|
|
by the sequential scan algorithm. When we cross this limit from below no action
|
2017-11-16 01:27:01 +00:00
|
|
|
is taken. When we cross this limit from above it is because we are issuing
|
|
|
|
verification I/O. In this case (unless the metadata scan is done) we stop
|
|
|
|
issuing verification I/O and start scanning metadata again until we get to the
|
|
|
|
hard limit.
|
|
|
|
.sp
|
|
|
|
Default value: \fB20\fR which is 5% of the hard limit (1/20).
|
|
|
|
.RE
|
|
|
|
|
2020-01-07 20:41:53 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_scan_strict_mem_lim\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Enforces tight memory limits on pool scans when a sequential scan is in
|
|
|
|
progress. When disabled the memory limit may be exceeded by fast disks.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_scan_suspend_progress\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Freezes a scrub/resilver in progress without actually pausing it. Intended for
|
|
|
|
testing/debugging.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
|
2017-11-16 01:27:01 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_scan_vdev_limit\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum amount of data that can be concurrently issued at once for scrubs and
|
|
|
|
resilvers per leaf device, given in bytes.
|
|
|
|
.sp
|
|
|
|
Default value: \fB41943040\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
2013-12-17 21:53:52 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_send_corrupt_data\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Allow sending of corrupt data (ignore read/checksum errors when sending data)
|
2013-12-17 21:53:52 +00:00
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
2019-05-07 22:18:44 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_send_unmodified_spill_blocks\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Include unmodified spill blocks in the send stream. Under certain circumstances
|
|
|
|
previous versions of ZFS could incorrectly remove the spill block from an
|
|
|
|
existing object. Including unmodified copies of the spill blocks creates a
|
|
|
|
backwards compatible stream which will recreate a spill block if it was
|
|
|
|
incorrectly removed.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes (default) and \fB0\fR for no.
|
|
|
|
.RE
|
|
|
|
|
Implement Redacted Send/Receive
Redacted send/receive allows users to send subsets of their data to
a target system. One possible use case for this feature is to not
transmit sensitive information to a data warehousing, test/dev, or
analytics environment. Another is to save space by not replicating
unimportant data within a given dataset, for example in backup tools
like zrepl.
Redacted send/receive is a three-stage process. First, a clone (or
clones) is made of the snapshot to be sent to the target. In this
clone (or clones), all unnecessary or unwanted data is removed or
modified. This clone is then snapshotted to create the "redaction
snapshot" (or snapshots). Second, the new zfs redact command is used
to create a redaction bookmark. The redaction bookmark stores the
list of blocks in a snapshot that were modified by the redaction
snapshot(s). Finally, the redaction bookmark is passed as a parameter
to zfs send. When sending to the snapshot that was redacted, the
redaction bookmark is used to filter out blocks that contain sensitive
or unwanted information, and those blocks are not included in the send
stream. When sending from the redaction bookmark, the blocks it
contains are considered as candidate blocks in addition to those
blocks in the destination snapshot that were modified since the
creation_txg of the redaction bookmark. This step is necessary to
allow the target to rehydrate data in the case where some blocks are
accidentally or unnecessarily modified in the redaction snapshot.
The changes to bookmarks to enable fast space estimation involve
adding deadlists to bookmarks. There is also logic to manage the
life cycles of these deadlists.
The new size estimation process operates in cases where previously
an accurate estimate could not be provided. In those cases, a send
is performed where no data blocks are read, reducing the runtime
significantly and providing a byte-accurate size estimate.
Reviewed-by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Prashanth Sreenivasa <pks@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Chris Williamson <chris.williamson@delphix.com>
Reviewed-by: Pavel Zhakarov <pavel.zakharov@delphix.com>
Reviewed-by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #7958
2019-06-19 16:48:13 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_send_no_prefetch_queue_ff\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The fill fraction of the \fBzfs send\fR internal queues. The fill fraction
|
|
|
|
controls the timing with which internal threads are woken up.
|
|
|
|
.sp
|
|
|
|
Default value: \fB20\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_send_no_prefetch_queue_length\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The maximum number of bytes allowed in \fBzfs send\fR's internal queues.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1,048,576\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_send_queue_ff\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The fill fraction of the \fBzfs send\fR prefetch queue. The fill fraction
|
|
|
|
controls the timing with which internal threads are woken up.
|
|
|
|
.sp
|
|
|
|
Default value: \fB20\fR.
|
|
|
|
.RE
|
|
|
|
|
2018-04-09 02:41:15 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_send_queue_length\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
Implement Redacted Send/Receive
Redacted send/receive allows users to send subsets of their data to
a target system. One possible use case for this feature is to not
transmit sensitive information to a data warehousing, test/dev, or
analytics environment. Another is to save space by not replicating
unimportant data within a given dataset, for example in backup tools
like zrepl.
Redacted send/receive is a three-stage process. First, a clone (or
clones) is made of the snapshot to be sent to the target. In this
clone (or clones), all unnecessary or unwanted data is removed or
modified. This clone is then snapshotted to create the "redaction
snapshot" (or snapshots). Second, the new zfs redact command is used
to create a redaction bookmark. The redaction bookmark stores the
list of blocks in a snapshot that were modified by the redaction
snapshot(s). Finally, the redaction bookmark is passed as a parameter
to zfs send. When sending to the snapshot that was redacted, the
redaction bookmark is used to filter out blocks that contain sensitive
or unwanted information, and those blocks are not included in the send
stream. When sending from the redaction bookmark, the blocks it
contains are considered as candidate blocks in addition to those
blocks in the destination snapshot that were modified since the
creation_txg of the redaction bookmark. This step is necessary to
allow the target to rehydrate data in the case where some blocks are
accidentally or unnecessarily modified in the redaction snapshot.
The changes to bookmarks to enable fast space estimation involve
adding deadlists to bookmarks. There is also logic to manage the
life cycles of these deadlists.
The new size estimation process operates in cases where previously
an accurate estimate could not be provided. In those cases, a send
is performed where no data blocks are read, reducing the runtime
significantly and providing a byte-accurate size estimate.
Reviewed-by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Prashanth Sreenivasa <pks@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Chris Williamson <chris.williamson@delphix.com>
Reviewed-by: Pavel Zhakarov <pavel.zakharov@delphix.com>
Reviewed-by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #7958
2019-06-19 16:48:13 +00:00
|
|
|
The maximum number of bytes allowed that will be prefetched by \fBzfs send\fR.
|
|
|
|
This value must be at least twice the maximum block size in use.
|
2018-04-09 02:41:15 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB16,777,216\fR.
|
|
|
|
.RE
|
|
|
|
|
Implement Redacted Send/Receive
Redacted send/receive allows users to send subsets of their data to
a target system. One possible use case for this feature is to not
transmit sensitive information to a data warehousing, test/dev, or
analytics environment. Another is to save space by not replicating
unimportant data within a given dataset, for example in backup tools
like zrepl.
Redacted send/receive is a three-stage process. First, a clone (or
clones) is made of the snapshot to be sent to the target. In this
clone (or clones), all unnecessary or unwanted data is removed or
modified. This clone is then snapshotted to create the "redaction
snapshot" (or snapshots). Second, the new zfs redact command is used
to create a redaction bookmark. The redaction bookmark stores the
list of blocks in a snapshot that were modified by the redaction
snapshot(s). Finally, the redaction bookmark is passed as a parameter
to zfs send. When sending to the snapshot that was redacted, the
redaction bookmark is used to filter out blocks that contain sensitive
or unwanted information, and those blocks are not included in the send
stream. When sending from the redaction bookmark, the blocks it
contains are considered as candidate blocks in addition to those
blocks in the destination snapshot that were modified since the
creation_txg of the redaction bookmark. This step is necessary to
allow the target to rehydrate data in the case where some blocks are
accidentally or unnecessarily modified in the redaction snapshot.
The changes to bookmarks to enable fast space estimation involve
adding deadlists to bookmarks. There is also logic to manage the
life cycles of these deadlists.
The new size estimation process operates in cases where previously
an accurate estimate could not be provided. In those cases, a send
is performed where no data blocks are read, reducing the runtime
significantly and providing a byte-accurate size estimate.
Reviewed-by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Prashanth Sreenivasa <pks@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Chris Williamson <chris.williamson@delphix.com>
Reviewed-by: Pavel Zhakarov <pavel.zakharov@delphix.com>
Reviewed-by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #7958
2019-06-19 16:48:13 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_recv_queue_ff\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The fill fraction of the \fBzfs receive\fR queue. The fill fraction
|
|
|
|
controls the timing with which internal threads are woken up.
|
|
|
|
.sp
|
|
|
|
Default value: \fB20\fR.
|
|
|
|
.RE
|
|
|
|
|
2018-04-09 02:41:15 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_recv_queue_length\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The maximum number of bytes allowed in the \fBzfs receive\fR queue. This value
|
|
|
|
must be at least twice the maximum block size in use.
|
|
|
|
.sp
|
|
|
|
Default value: \fB16,777,216\fR.
|
|
|
|
.RE
|
|
|
|
|
Improve zfs receive performance by batching writes
For each WRITE record in the stream, `zfs receive` creates a DMU
transaction (`dmu_tx_create()`) and writes this block's data into the
object. If per-block overheads (as opposed to per-byte overheads)
dominate performance (as is often the case with small recordsize), the
per-dmu-transaction overheads can be significant. For example, in some
workloads the `receieve_writer` thread is 100% on CPU, and more than
half of its CPU time is in these per-tx routines (e.g.
dmu_tx_hold_write, dmu_tx_assign, dmu_tx_commit).
To improve performance of `zfs receive`, this commit batches WRITE
records which are to nearby offsets of the same object, and uses one DMU
transaction to write them all. By default the batch size is 1MB, which
for recordsize=8K reduces the number of DMU transactions by 128x for
full send streams (incrementals will depend on how "clumpy" the changed
blocks are).
This commit improves the performance of `dd if=stream | zfs recv`
from 78,800 blocks/sec to 98,100 blocks/sec (25% improvement).
Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #10099
2020-03-16 18:51:56 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_recv_write_batch_size\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The maximum amount of data (in bytes) that \fBzfs receive\fR will write in
|
|
|
|
one DMU transaction. This is the uncompressed size, even when receiving a
|
|
|
|
compressed send stream. This setting will not reduce the write size below
|
|
|
|
a single block. Capped at a maximum of 32MB
|
|
|
|
.sp
|
|
|
|
Default value: \fB1MB\fR.
|
|
|
|
.RE
|
|
|
|
|
Implement Redacted Send/Receive
Redacted send/receive allows users to send subsets of their data to
a target system. One possible use case for this feature is to not
transmit sensitive information to a data warehousing, test/dev, or
analytics environment. Another is to save space by not replicating
unimportant data within a given dataset, for example in backup tools
like zrepl.
Redacted send/receive is a three-stage process. First, a clone (or
clones) is made of the snapshot to be sent to the target. In this
clone (or clones), all unnecessary or unwanted data is removed or
modified. This clone is then snapshotted to create the "redaction
snapshot" (or snapshots). Second, the new zfs redact command is used
to create a redaction bookmark. The redaction bookmark stores the
list of blocks in a snapshot that were modified by the redaction
snapshot(s). Finally, the redaction bookmark is passed as a parameter
to zfs send. When sending to the snapshot that was redacted, the
redaction bookmark is used to filter out blocks that contain sensitive
or unwanted information, and those blocks are not included in the send
stream. When sending from the redaction bookmark, the blocks it
contains are considered as candidate blocks in addition to those
blocks in the destination snapshot that were modified since the
creation_txg of the redaction bookmark. This step is necessary to
allow the target to rehydrate data in the case where some blocks are
accidentally or unnecessarily modified in the redaction snapshot.
The changes to bookmarks to enable fast space estimation involve
adding deadlists to bookmarks. There is also logic to manage the
life cycles of these deadlists.
The new size estimation process operates in cases where previously
an accurate estimate could not be provided. In those cases, a send
is performed where no data blocks are read, reducing the runtime
significantly and providing a byte-accurate size estimate.
Reviewed-by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Prashanth Sreenivasa <pks@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Chris Williamson <chris.williamson@delphix.com>
Reviewed-by: Pavel Zhakarov <pavel.zakharov@delphix.com>
Reviewed-by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #7958
2019-06-19 16:48:13 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_override_estimate_recordsize\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Setting this variable overrides the default logic for estimating block
|
|
|
|
sizes when doing a zfs send. The default heuristic is that the average
|
|
|
|
block size will be the current recordsize. Override this value if most data
|
|
|
|
in your dataset is not of that size and you require accurate zfs send size
|
|
|
|
estimates.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_sync_pass_deferred_free\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Flushing of data to disk is done in passes. Defer frees starting in this pass
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB2\fR.
|
|
|
|
.RE
|
|
|
|
|
2016-12-16 22:11:29 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_spa_discard_memory_limit\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum memory used for prefetching a checkpoint's space map on each
|
|
|
|
vdev while discarding the checkpoint.
|
|
|
|
.sp
|
|
|
|
Default value: \fB16,777,216\fR.
|
|
|
|
.RE
|
|
|
|
|
2019-05-07 22:34:42 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_special_class_metadata_reserve_pct\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Only allow small data blocks to be allocated on the special and dedup vdev
|
|
|
|
types when the available free space percentage on these vdevs exceeds this
|
|
|
|
value. This ensures reserved space is available for pool meta data as the
|
|
|
|
special vdevs approach capacity.
|
|
|
|
.sp
|
|
|
|
Default value: \fB25\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_sync_pass_dont_compress\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2020-08-25 04:25:28 +00:00
|
|
|
Starting in this sync pass, we disable compression (including of metadata).
|
2019-06-13 20:10:19 +00:00
|
|
|
With the default setting, in practice, we don't have this many sync passes,
|
|
|
|
so this has no effect.
|
|
|
|
.sp
|
|
|
|
The original intent was that disabling compression would help the sync passes
|
|
|
|
to converge. However, in practice disabling compression increases the average
|
|
|
|
number of sync passes, because when we turn compression off, a lot of block's
|
|
|
|
size will change and thus we have to re-allocate (not overwrite) them. It
|
|
|
|
also increases the number of 128KB allocations (e.g. for indirect blocks and
|
|
|
|
spacemaps) because these will not be compressed. The 128K allocations are
|
|
|
|
especially detrimental to performance on highly fragmented systems, which may
|
|
|
|
have very few free segments of this size, and may need to load new metaslabs
|
|
|
|
to satisfy 128K allocations.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2019-06-13 20:10:19 +00:00
|
|
|
Default value: \fB8\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_sync_pass_rewrite\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Rewrite new block pointers starting in this pass
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB2\fR.
|
|
|
|
.RE
|
|
|
|
|
2017-10-26 19:57:53 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_sync_taskq_batch_pct\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
This controls the number of threads used by the dp_sync_taskq. The default
|
|
|
|
value of 75% will create a maximum of one thread per cpu.
|
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB75\fR%.
|
2017-10-26 19:57:53 +00:00
|
|
|
.RE
|
|
|
|
|
2019-03-29 16:13:20 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2020-01-07 20:41:53 +00:00
|
|
|
\fBzfs_trim_extent_bytes_max\fR (uint)
|
2019-03-29 16:13:20 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum size of TRIM command. Ranges larger than this will be split in to
|
|
|
|
chunks no larger than \fBzfs_trim_extent_bytes_max\fR bytes before being
|
|
|
|
issued to the device.
|
|
|
|
.sp
|
|
|
|
Default value: \fB134,217,728\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2020-01-07 20:41:53 +00:00
|
|
|
\fBzfs_trim_extent_bytes_min\fR (uint)
|
2019-03-29 16:13:20 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Minimum size of TRIM commands. TRIM ranges smaller than this will be skipped
|
|
|
|
unless they're part of a larger range which was broken in to chunks. This is
|
|
|
|
done because it's common for these small TRIMs to negatively impact overall
|
|
|
|
performance. This value can be set to 0 to TRIM all unallocated space.
|
|
|
|
.sp
|
|
|
|
Default value: \fB32,768\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2020-01-07 20:41:53 +00:00
|
|
|
\fBzfs_trim_metaslab_skip\fR (uint)
|
2019-03-29 16:13:20 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Skip uninitialized metaslabs during the TRIM process. This option is useful
|
|
|
|
for pools constructed from large thinly-provisioned devices where TRIM
|
|
|
|
operations are slow. As a pool ages an increasing fraction of the pools
|
|
|
|
metaslabs will be initialized progressively degrading the usefulness of
|
|
|
|
this option. This setting is stored when starting a manual TRIM and will
|
|
|
|
persist for the duration of the requested TRIM.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2020-01-07 20:41:53 +00:00
|
|
|
\fBzfs_trim_queue_limit\fR (uint)
|
2019-03-29 16:13:20 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum number of queued TRIMs outstanding per leaf vdev. The number of
|
|
|
|
concurrent TRIM commands issued to the device is controlled by the
|
|
|
|
\fBzfs_vdev_trim_min_active\fR and \fBzfs_vdev_trim_max_active\fR module
|
|
|
|
options.
|
|
|
|
.sp
|
|
|
|
Default value: \fB10\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2020-01-07 20:41:53 +00:00
|
|
|
\fBzfs_trim_txg_batch\fR (uint)
|
2019-03-29 16:13:20 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The number of transaction groups worth of frees which should be aggregated
|
|
|
|
before TRIM operations are issued to the device. This setting represents a
|
|
|
|
trade-off between issuing larger, more efficient TRIM operations and the
|
|
|
|
delay before the recently trimmed space is available for use by the device.
|
|
|
|
.sp
|
|
|
|
Increasing this value will allow frees to be aggregated for a longer time.
|
|
|
|
This will result is larger TRIM operations and potentially increased memory
|
|
|
|
usage. Decreasing this value will have the opposite effect. The default
|
|
|
|
value of 32 was determined to be a reasonable compromise.
|
|
|
|
.sp
|
|
|
|
Default value: \fB32\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_txg_history\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
Multi-modifier protection (MMP)
Add multihost=on|off pool property to control MMP. When enabled
a new thread writes uberblocks to the last slot in each label, at a
set frequency, to indicate to other hosts the pool is actively imported.
These uberblocks are the last synced uberblock with an updated
timestamp. Property defaults to off.
During tryimport, find the "best" uberblock (newest txg and timestamp)
repeatedly, checking for change in the found uberblock. Include the
results of the activity test in the config returned by tryimport.
These results are reported to user in "zpool import".
Allow the user to control the period between MMP writes, and the
duration of the activity test on import, via a new module parameter
zfs_multihost_interval. The period is specified in milliseconds. The
activity test duration is calculated from this value, and from the
mmp_delay in the "best" uberblock found initially.
Add a kstat interface to export statistics about Multiple Modifier
Protection (MMP) updates. Include the last synced txg number, the
timestamp, the delay since the last MMP update, the VDEV GUID, the VDEV
label that received the last MMP update, and the VDEV path. Abbreviated
output below.
$ cat /proc/spl/kstat/zfs/mypool/multihost
31 0 0x01 10 880 105092382393521 105144180101111
txg timestamp mmp_delay vdev_guid vdev_label vdev_path
20468 261337 250274925 68396651780 3 /dev/sda
20468 261339 252023374 6267402363293 1 /dev/sdc
20468 261340 252000858 6698080955233 1 /dev/sdx
20468 261341 251980635 783892869810 2 /dev/sdy
20468 261342 253385953 8923255792467 3 /dev/sdd
20468 261344 253336622 042125143176 0 /dev/sdab
20468 261345 253310522 1200778101278 2 /dev/sde
20468 261346 253286429 0950576198362 2 /dev/sdt
20468 261347 253261545 96209817917 3 /dev/sds
20468 261349 253238188 8555725937673 3 /dev/sdb
Add a new tunable zfs_multihost_history to specify the number of MMP
updates to store history for. By default it is set to zero meaning that
no MMP statistics are stored.
When using ztest to generate activity, for automated tests of the MMP
function, some test functions interfere with the test. For example, the
pool is exported to run zdb and then imported again. Add a new ztest
function, "-M", to alter ztest behavior to prevent this.
Add new tests to verify the new functionality. Tests provided by
Giuseppe Di Natale.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes #745
Closes #6279
2017-07-08 03:20:35 +00:00
|
|
|
Historical statistics for the last N txgs will be available in
|
|
|
|
\fB/proc/spl/kstat/zfs/<pool>/txgs\fR
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
2017-10-30 20:15:10 +00:00
|
|
|
Default value: \fB0\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_txg_timeout\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Flush dirty data to disk at least every N seconds (maximum txg duration)
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB5\fR.
|
|
|
|
.RE
|
|
|
|
|
2019-03-29 16:13:20 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_aggregate_trim\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Allow TRIM I/Os to be aggregated. This is normally not helpful because
|
|
|
|
the extents to be trimmed will have been already been aggregated by the
|
|
|
|
metaslab. This option is provided for debugging and performance analysis.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_aggregation_limit\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Max vdev I/O aggregation size
|
|
|
|
.sp
|
2019-03-13 19:00:10 +00:00
|
|
|
Default value: \fB1,048,576\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_aggregation_limit_non_rotating\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Max vdev I/O aggregation size for non-rotating media
|
|
|
|
.sp
|
2013-11-16 06:52:54 +00:00
|
|
|
Default value: \fB131,072\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_cache_bshift\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Shift size to inflate reads too
|
|
|
|
.sp
|
2015-12-30 17:44:46 +00:00
|
|
|
Default value: \fB16\fR (effectively 65536).
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_cache_max\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2017-10-30 20:15:10 +00:00
|
|
|
Inflate reads smaller than this value to meet the \fBzfs_vdev_cache_bshift\fR
|
|
|
|
size (default 64k).
|
2015-12-30 17:44:46 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB16384\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_cache_size\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Total size of the per-disk cache in bytes.
|
|
|
|
.sp
|
|
|
|
Currently this feature is disabled as it has been found to not be helpful
|
|
|
|
for performance and in some cases harmful.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
FreeBSD r256956: Improve ZFS N-way mirror read performance by using load and locality information.
The existing algorithm selects a preferred leaf vdev based on offset of the zio
request modulo the number of members in the mirror. It assumes the devices are
of equal performance and that spreading the requests randomly over both drives
will be sufficient to saturate them. In practice this results in the leaf vdevs
being under utilized.
The new algorithm takes into the following additional factors:
* Load of the vdevs (number outstanding I/O requests)
* The locality of last queued I/O vs the new I/O request.
Within the locality calculation additional knowledge about the underlying vdev
is considered such as; is the device backing the vdev a rotating media device.
This results in performance increases across the board as well as significant
increases for predominantly streaming loads and for configurations which don't
have evenly performing devices.
The following are results from a setup with 3 Way Mirror with 2 x HD's and
1 x SSD from a basic test running multiple parrallel dd's.
With pre-fetch disabled (vfs.zfs.prefetch_disable=1):
== Stripe Balanced (default) ==
Read 15360MB using bs: 1048576, readers: 3, took 161 seconds @ 95 MB/s
== Load Balanced (zfslinux) ==
Read 15360MB using bs: 1048576, readers: 3, took 297 seconds @ 51 MB/s
== Load Balanced (locality freebsd) ==
Read 15360MB using bs: 1048576, readers: 3, took 54 seconds @ 284 MB/s
With pre-fetch enabled (vfs.zfs.prefetch_disable=0):
== Stripe Balanced (default) ==
Read 15360MB using bs: 1048576, readers: 3, took 91 seconds @ 168 MB/s
== Load Balanced (zfslinux) ==
Read 15360MB using bs: 1048576, readers: 3, took 108 seconds @ 142 MB/s
== Load Balanced (locality freebsd) ==
Read 15360MB using bs: 1048576, readers: 3, took 48 seconds @ 320 MB/s
In addition to the performance changes the code was also restructured, with
the help of Justin Gibbs, to provide a more logical flow which also ensures
vdevs loads are only calculated from the set of valid candidates.
The following additional sysctls where added to allow the administrator
to tune the behaviour of the load algorithm:
* vfs.zfs.vdev.mirror.rotating_inc
* vfs.zfs.vdev.mirror.rotating_seek_inc
* vfs.zfs.vdev.mirror.rotating_seek_offset
* vfs.zfs.vdev.mirror.non_rotating_inc
* vfs.zfs.vdev.mirror.non_rotating_seek_inc
These changes where based on work started by the zfsonlinux developers:
https://github.com/zfsonlinux/zfs/pull/1487
Reviewed by: gibbs, mav, will
MFC after: 2 weeks
Sponsored by: Multiplay
References:
https://github.com/freebsd/freebsd@5c7a6f5d
https://github.com/freebsd/freebsd@31b7f68d
https://github.com/freebsd/freebsd@e186f564
Performance Testing:
https://github.com/zfsonlinux/zfs/pull/4334#issuecomment-189057141
Porting notes:
- The tunables were adjusted to have ZoL-style names.
- The code was modified to use ZoL's vd_nonrot.
- Fixes were done to make cstyle.pl happy
- Merge conflicts were handled manually
- freebsd/freebsd@e186f564bc946f82c76e0b34c2f0370ed9aea022 by my
collegue Andriy Gapon has been included. It applied perfectly, but
added a cstyle regression.
- This replaces 556011dbec2d10579819078559a77630fc559112 entirely.
- A typo "IO'a" has been corrected to say "IO's"
- Descriptions of new tunables were added to man/man5/zfs-module-parameters.5.
Ported-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #4334
2016-02-13 01:47:22 +00:00
|
|
|
\fBzfs_vdev_mirror_rotating_inc\fR (int)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
FreeBSD r256956: Improve ZFS N-way mirror read performance by using load and locality information.
The existing algorithm selects a preferred leaf vdev based on offset of the zio
request modulo the number of members in the mirror. It assumes the devices are
of equal performance and that spreading the requests randomly over both drives
will be sufficient to saturate them. In practice this results in the leaf vdevs
being under utilized.
The new algorithm takes into the following additional factors:
* Load of the vdevs (number outstanding I/O requests)
* The locality of last queued I/O vs the new I/O request.
Within the locality calculation additional knowledge about the underlying vdev
is considered such as; is the device backing the vdev a rotating media device.
This results in performance increases across the board as well as significant
increases for predominantly streaming loads and for configurations which don't
have evenly performing devices.
The following are results from a setup with 3 Way Mirror with 2 x HD's and
1 x SSD from a basic test running multiple parrallel dd's.
With pre-fetch disabled (vfs.zfs.prefetch_disable=1):
== Stripe Balanced (default) ==
Read 15360MB using bs: 1048576, readers: 3, took 161 seconds @ 95 MB/s
== Load Balanced (zfslinux) ==
Read 15360MB using bs: 1048576, readers: 3, took 297 seconds @ 51 MB/s
== Load Balanced (locality freebsd) ==
Read 15360MB using bs: 1048576, readers: 3, took 54 seconds @ 284 MB/s
With pre-fetch enabled (vfs.zfs.prefetch_disable=0):
== Stripe Balanced (default) ==
Read 15360MB using bs: 1048576, readers: 3, took 91 seconds @ 168 MB/s
== Load Balanced (zfslinux) ==
Read 15360MB using bs: 1048576, readers: 3, took 108 seconds @ 142 MB/s
== Load Balanced (locality freebsd) ==
Read 15360MB using bs: 1048576, readers: 3, took 48 seconds @ 320 MB/s
In addition to the performance changes the code was also restructured, with
the help of Justin Gibbs, to provide a more logical flow which also ensures
vdevs loads are only calculated from the set of valid candidates.
The following additional sysctls where added to allow the administrator
to tune the behaviour of the load algorithm:
* vfs.zfs.vdev.mirror.rotating_inc
* vfs.zfs.vdev.mirror.rotating_seek_inc
* vfs.zfs.vdev.mirror.rotating_seek_offset
* vfs.zfs.vdev.mirror.non_rotating_inc
* vfs.zfs.vdev.mirror.non_rotating_seek_inc
These changes where based on work started by the zfsonlinux developers:
https://github.com/zfsonlinux/zfs/pull/1487
Reviewed by: gibbs, mav, will
MFC after: 2 weeks
Sponsored by: Multiplay
References:
https://github.com/freebsd/freebsd@5c7a6f5d
https://github.com/freebsd/freebsd@31b7f68d
https://github.com/freebsd/freebsd@e186f564
Performance Testing:
https://github.com/zfsonlinux/zfs/pull/4334#issuecomment-189057141
Porting notes:
- The tunables were adjusted to have ZoL-style names.
- The code was modified to use ZoL's vd_nonrot.
- Fixes were done to make cstyle.pl happy
- Merge conflicts were handled manually
- freebsd/freebsd@e186f564bc946f82c76e0b34c2f0370ed9aea022 by my
collegue Andriy Gapon has been included. It applied perfectly, but
added a cstyle regression.
- This replaces 556011dbec2d10579819078559a77630fc559112 entirely.
- A typo "IO'a" has been corrected to say "IO's"
- Descriptions of new tunables were added to man/man5/zfs-module-parameters.5.
Ported-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #4334
2016-02-13 01:47:22 +00:00
|
|
|
A number by which the balancing algorithm increments the load calculation for
|
|
|
|
the purpose of selecting the least busy mirror member when an I/O immediately
|
|
|
|
follows its predecessor on rotational vdevs for the purpose of making decisions
|
|
|
|
based on load.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
FreeBSD r256956: Improve ZFS N-way mirror read performance by using load and locality information.
The existing algorithm selects a preferred leaf vdev based on offset of the zio
request modulo the number of members in the mirror. It assumes the devices are
of equal performance and that spreading the requests randomly over both drives
will be sufficient to saturate them. In practice this results in the leaf vdevs
being under utilized.
The new algorithm takes into the following additional factors:
* Load of the vdevs (number outstanding I/O requests)
* The locality of last queued I/O vs the new I/O request.
Within the locality calculation additional knowledge about the underlying vdev
is considered such as; is the device backing the vdev a rotating media device.
This results in performance increases across the board as well as significant
increases for predominantly streaming loads and for configurations which don't
have evenly performing devices.
The following are results from a setup with 3 Way Mirror with 2 x HD's and
1 x SSD from a basic test running multiple parrallel dd's.
With pre-fetch disabled (vfs.zfs.prefetch_disable=1):
== Stripe Balanced (default) ==
Read 15360MB using bs: 1048576, readers: 3, took 161 seconds @ 95 MB/s
== Load Balanced (zfslinux) ==
Read 15360MB using bs: 1048576, readers: 3, took 297 seconds @ 51 MB/s
== Load Balanced (locality freebsd) ==
Read 15360MB using bs: 1048576, readers: 3, took 54 seconds @ 284 MB/s
With pre-fetch enabled (vfs.zfs.prefetch_disable=0):
== Stripe Balanced (default) ==
Read 15360MB using bs: 1048576, readers: 3, took 91 seconds @ 168 MB/s
== Load Balanced (zfslinux) ==
Read 15360MB using bs: 1048576, readers: 3, took 108 seconds @ 142 MB/s
== Load Balanced (locality freebsd) ==
Read 15360MB using bs: 1048576, readers: 3, took 48 seconds @ 320 MB/s
In addition to the performance changes the code was also restructured, with
the help of Justin Gibbs, to provide a more logical flow which also ensures
vdevs loads are only calculated from the set of valid candidates.
The following additional sysctls where added to allow the administrator
to tune the behaviour of the load algorithm:
* vfs.zfs.vdev.mirror.rotating_inc
* vfs.zfs.vdev.mirror.rotating_seek_inc
* vfs.zfs.vdev.mirror.rotating_seek_offset
* vfs.zfs.vdev.mirror.non_rotating_inc
* vfs.zfs.vdev.mirror.non_rotating_seek_inc
These changes where based on work started by the zfsonlinux developers:
https://github.com/zfsonlinux/zfs/pull/1487
Reviewed by: gibbs, mav, will
MFC after: 2 weeks
Sponsored by: Multiplay
References:
https://github.com/freebsd/freebsd@5c7a6f5d
https://github.com/freebsd/freebsd@31b7f68d
https://github.com/freebsd/freebsd@e186f564
Performance Testing:
https://github.com/zfsonlinux/zfs/pull/4334#issuecomment-189057141
Porting notes:
- The tunables were adjusted to have ZoL-style names.
- The code was modified to use ZoL's vd_nonrot.
- Fixes were done to make cstyle.pl happy
- Merge conflicts were handled manually
- freebsd/freebsd@e186f564bc946f82c76e0b34c2f0370ed9aea022 by my
collegue Andriy Gapon has been included. It applied perfectly, but
added a cstyle regression.
- This replaces 556011dbec2d10579819078559a77630fc559112 entirely.
- A typo "IO'a" has been corrected to say "IO's"
- Descriptions of new tunables were added to man/man5/zfs-module-parameters.5.
Ported-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #4334
2016-02-13 01:47:22 +00:00
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_mirror_rotating_seek_inc\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
A number by which the balancing algorithm increments the load calculation for
|
|
|
|
the purpose of selecting the least busy mirror member when an I/O lacks
|
|
|
|
locality as defined by the zfs_vdev_mirror_rotating_seek_offset. I/Os within
|
|
|
|
this that are not immediately following the previous I/O are incremented by
|
|
|
|
half.
|
|
|
|
.sp
|
|
|
|
Default value: \fB5\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_mirror_rotating_seek_offset\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The maximum distance for the last queued I/O in which the balancing algorithm
|
|
|
|
considers an I/O to have locality.
|
|
|
|
See the section "ZFS I/O SCHEDULER".
|
|
|
|
.sp
|
|
|
|
Default value: \fB1048576\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_mirror_non_rotating_inc\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
A number by which the balancing algorithm increments the load calculation for
|
|
|
|
the purpose of selecting the least busy mirror member on non-rotational vdevs
|
|
|
|
when I/Os do not immediately follow one another.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_mirror_non_rotating_seek_inc\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
A number by which the balancing algorithm increments the load calculation for
|
|
|
|
the purpose of selecting the least busy mirror member when an I/O lacks
|
|
|
|
locality as defined by the zfs_vdev_mirror_rotating_seek_offset. I/Os within
|
|
|
|
this that are not immediately following the previous I/O are incremented by
|
|
|
|
half.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_read_gap_limit\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Aggregate read I/O operations if the gap on-disk between them is within this
|
|
|
|
threshold.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB32,768\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_write_gap_limit\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Aggregate write I/O over gap
|
|
|
|
.sp
|
|
|
|
Default value: \fB4,096\fR.
|
|
|
|
.RE
|
|
|
|
|
SIMD implementation of vdev_raidz generate and reconstruct routines
This is a new implementation of RAIDZ1/2/3 routines using x86_64
scalar, SSE, and AVX2 instruction sets. Included are 3 parity
generation routines (P, PQ, and PQR) and 7 reconstruction routines,
for all RAIDZ level. On module load, a quick benchmark of supported
routines will select the fastest for each operation and they will
be used at runtime. Original implementation is still present and
can be selected via module parameter.
Patch contains:
- specialized gen/rec routines for all RAIDZ levels,
- new scalar raidz implementation (unrolled),
- two x86_64 SIMD implementations (SSE and AVX2 instructions sets),
- fastest routines selected on module load (benchmark).
- cmd/raidz_test - verify and benchmark all implementations
- added raidz_test to the ZFS Test Suite
New zfs module parameters:
- zfs_vdev_raidz_impl (str): selects the implementation to use. On
module load, the parameter will only accept first 3 options, and
the other implementations can be set once module is finished
loading. Possible values for this option are:
"fastest" - use the fastest math available
"original" - use the original raidz code
"scalar" - new scalar impl
"sse" - new SSE impl if available
"avx2" - new AVX2 impl if available
See contents of `/sys/module/zfs/parameters/zfs_vdev_raidz_impl` to
get the list of supported values. If an implementation is not supported
on the system, it will not be shown. Currently selected option is
enclosed in `[]`.
Signed-off-by: Gvozden Neskovic <neskovic@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #4328
2016-04-25 08:04:31 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_raidz_impl\fR (string)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2016-07-17 17:41:11 +00:00
|
|
|
Parameter for selecting raidz parity implementation to use.
|
SIMD implementation of vdev_raidz generate and reconstruct routines
This is a new implementation of RAIDZ1/2/3 routines using x86_64
scalar, SSE, and AVX2 instruction sets. Included are 3 parity
generation routines (P, PQ, and PQR) and 7 reconstruction routines,
for all RAIDZ level. On module load, a quick benchmark of supported
routines will select the fastest for each operation and they will
be used at runtime. Original implementation is still present and
can be selected via module parameter.
Patch contains:
- specialized gen/rec routines for all RAIDZ levels,
- new scalar raidz implementation (unrolled),
- two x86_64 SIMD implementations (SSE and AVX2 instructions sets),
- fastest routines selected on module load (benchmark).
- cmd/raidz_test - verify and benchmark all implementations
- added raidz_test to the ZFS Test Suite
New zfs module parameters:
- zfs_vdev_raidz_impl (str): selects the implementation to use. On
module load, the parameter will only accept first 3 options, and
the other implementations can be set once module is finished
loading. Possible values for this option are:
"fastest" - use the fastest math available
"original" - use the original raidz code
"scalar" - new scalar impl
"sse" - new SSE impl if available
"avx2" - new AVX2 impl if available
See contents of `/sys/module/zfs/parameters/zfs_vdev_raidz_impl` to
get the list of supported values. If an implementation is not supported
on the system, it will not be shown. Currently selected option is
enclosed in `[]`.
Signed-off-by: Gvozden Neskovic <neskovic@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #4328
2016-04-25 08:04:31 +00:00
|
|
|
|
|
|
|
Options marked (always) below may be selected on module load as they are
|
|
|
|
supported on all systems.
|
|
|
|
The remaining options may only be set after the module is loaded, as they
|
|
|
|
are available only if the implementations are compiled in and supported
|
|
|
|
on the running system.
|
|
|
|
|
|
|
|
Once the module is loaded, the content of
|
|
|
|
/sys/module/zfs/parameters/zfs_vdev_raidz_impl will show available options
|
|
|
|
with the currently selected one enclosed in [].
|
|
|
|
Possible options are:
|
|
|
|
fastest - (always) implementation selected using built-in benchmark
|
|
|
|
original - (always) original raidz implementation
|
|
|
|
scalar - (always) scalar raidz implementation
|
2016-06-28 17:49:53 +00:00
|
|
|
sse2 - implementation using SSE2 instruction set (64bit x86 only)
|
|
|
|
ssse3 - implementation using SSSE3 instruction set (64bit x86 only)
|
SIMD implementation of vdev_raidz generate and reconstruct routines
This is a new implementation of RAIDZ1/2/3 routines using x86_64
scalar, SSE, and AVX2 instruction sets. Included are 3 parity
generation routines (P, PQ, and PQR) and 7 reconstruction routines,
for all RAIDZ level. On module load, a quick benchmark of supported
routines will select the fastest for each operation and they will
be used at runtime. Original implementation is still present and
can be selected via module parameter.
Patch contains:
- specialized gen/rec routines for all RAIDZ levels,
- new scalar raidz implementation (unrolled),
- two x86_64 SIMD implementations (SSE and AVX2 instructions sets),
- fastest routines selected on module load (benchmark).
- cmd/raidz_test - verify and benchmark all implementations
- added raidz_test to the ZFS Test Suite
New zfs module parameters:
- zfs_vdev_raidz_impl (str): selects the implementation to use. On
module load, the parameter will only accept first 3 options, and
the other implementations can be set once module is finished
loading. Possible values for this option are:
"fastest" - use the fastest math available
"original" - use the original raidz code
"scalar" - new scalar impl
"sse" - new SSE impl if available
"avx2" - new AVX2 impl if available
See contents of `/sys/module/zfs/parameters/zfs_vdev_raidz_impl` to
get the list of supported values. If an implementation is not supported
on the system, it will not be shown. Currently selected option is
enclosed in `[]`.
Signed-off-by: Gvozden Neskovic <neskovic@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #4328
2016-04-25 08:04:31 +00:00
|
|
|
avx2 - implementation using AVX2 instruction set (64bit x86 only)
|
2016-11-02 19:40:23 +00:00
|
|
|
avx512f - implementation using AVX512F instruction set (64bit x86 only)
|
|
|
|
avx512bw - implementation using AVX512F & AVX512BW instruction sets (64bit x86 only)
|
Add parity generation/rebuild using 128-bits NEON for Aarch64
This re-use the framework established for SSE2, SSSE3 and
AVX2. However, GCC is using FP registers on Aarch64, so
unlike SSE/AVX2 we can't rely on the registers being left alone
between ASM statements. So instead, the NEON code uses
C variables and GCC extended ASM syntax. Note that since
the kernel explicitly disable vector registers, they
have to be locally re-enabled explicitly.
As we use the variable's number to define the symbolic
name, and GCC won't allow duplicate symbolic names,
numbers have to be unique. Even when the code is not
going to be used (e.g. the case for 4 registers when
using the macro with only 2). Only the actually used
variables should be declared, otherwise the build
will fails in debug mode.
This requires the replacement of the XOR(X,X) syntax
by a new ZERO(X) macro, which does the same thing but
without repeating the argument. And perhaps someday
there will be a machine where there is a more efficient
way to zero a register than XOR with itself. This affects
scalar, SSE2, SSSE3 and AVX2 as they need the new macro.
It's possible to write faster implementations (different
scheduling, different unrolling, interleaving NEON and
scalar, ...) for various cores, but this one has the
advantage of fitting in the current state of the code,
and thus is likely easier to review/check/merge.
The only difference between aarch64-neon and aarch64-neonx2
is that aarch64-neonx2 unroll some functions some more.
Reviewed-by: Gvozden Neskovic <neskovic@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Romain Dolbeau <romain.dolbeau@atos.net>
Closes #4801
2016-10-03 16:44:00 +00:00
|
|
|
aarch64_neon - implementation using NEON (Aarch64/64 bit ARMv8 only)
|
|
|
|
aarch64_neonx2 - implementation using NEON with more unrolling (Aarch64/64 bit ARMv8 only)
|
2020-01-23 19:01:24 +00:00
|
|
|
powerpc_altivec - implementation using Altivec (PowerPC only)
|
SIMD implementation of vdev_raidz generate and reconstruct routines
This is a new implementation of RAIDZ1/2/3 routines using x86_64
scalar, SSE, and AVX2 instruction sets. Included are 3 parity
generation routines (P, PQ, and PQR) and 7 reconstruction routines,
for all RAIDZ level. On module load, a quick benchmark of supported
routines will select the fastest for each operation and they will
be used at runtime. Original implementation is still present and
can be selected via module parameter.
Patch contains:
- specialized gen/rec routines for all RAIDZ levels,
- new scalar raidz implementation (unrolled),
- two x86_64 SIMD implementations (SSE and AVX2 instructions sets),
- fastest routines selected on module load (benchmark).
- cmd/raidz_test - verify and benchmark all implementations
- added raidz_test to the ZFS Test Suite
New zfs module parameters:
- zfs_vdev_raidz_impl (str): selects the implementation to use. On
module load, the parameter will only accept first 3 options, and
the other implementations can be set once module is finished
loading. Possible values for this option are:
"fastest" - use the fastest math available
"original" - use the original raidz code
"scalar" - new scalar impl
"sse" - new SSE impl if available
"avx2" - new AVX2 impl if available
See contents of `/sys/module/zfs/parameters/zfs_vdev_raidz_impl` to
get the list of supported values. If an implementation is not supported
on the system, it will not be shown. Currently selected option is
enclosed in `[]`.
Signed-off-by: Gvozden Neskovic <neskovic@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #4328
2016-04-25 08:04:31 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fBfastest\fR.
|
|
|
|
.RE
|
|
|
|
|
2020-01-07 20:41:53 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_vdev_scheduler\fR (charp)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
\fBDEPRECATED\fR: This option exists for compatibility with older user
|
|
|
|
configurations. It does nothing except print a warning to the kernel log if
|
|
|
|
set.
|
|
|
|
.sp
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_zevent_cols\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
When zevents are logged to the console use this as the word wrap width.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB80\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_zevent_console\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Log events to the console
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_zevent_len_max\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Max event queue length. A value of 0 will result in a calculated value which
|
|
|
|
increases with the number of CPUs in the system (minimum 64 events). Events
|
|
|
|
in the queue can be viewed with the \fBzpool events\fR command.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2017-10-26 19:57:53 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
2020-09-04 17:34:28 +00:00
|
|
|
.na
|
|
|
|
\fBzfs_zevent_retain_max\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Maximum recent zevent records to retain for duplicate checking. Setting
|
|
|
|
this value to zero disables duplicate detection.
|
|
|
|
.sp
|
|
|
|
Default value: \fB2000\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_zevent_retain_expire_secs\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Lifespan for a recent ereport that was retained for duplicate checking.
|
|
|
|
.sp
|
|
|
|
Default value: \fB900\fR.
|
|
|
|
.RE
|
|
|
|
|
2017-10-26 19:57:53 +00:00
|
|
|
.na
|
|
|
|
\fBzfs_zil_clean_taskq_maxalloc\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The maximum number of taskq entries that are allowed to be cached. When this
|
OpenZFS 8909 - 8585 can cause a use-after-free kernel panic
Authored by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: John Kennedy <jwk404@gmail.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Brad Lewis <brad.lewis@delphix.com>
Reviewed by: Igor Kozhukhov <igor@dilos.org>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Robert Mustacchi <rm@joyent.com>
Ported-by: Prakash Surya <prakash.surya@delphix.com>
PROBLEM
=======
There's a race condition that exists if `zil_free_lwb` races with either
`zil_commit_waiter_timeout` and/or `zil_lwb_flush_vdevs_done`.
Here's an example panic due to this bug:
> ::status
debugging crash dump vmcore.0 (64-bit) from ip-10-110-205-40
operating system: 5.11 dlpx-5.2.2.0_2017-12-04-17-28-32b6ba51fb (i86pc)
image uuid: 4af0edfb-e58e-6ed8-cafc-d3e9167c7513
panic message:
BAD TRAP: type=e (#pf Page fault) rp=ffffff0010555970 addr=60 occurred in module "zfs" due to a NULL pointer dereference
dump content: kernel pages only
> $c
zio_shrink+0x12()
zil_lwb_write_issue+0x30d(ffffff03dcd15cc0, ffffff03e0730e20)
zil_commit_waiter_timeout+0xa2(ffffff03dcd15cc0, ffffff03d97ffcf8)
zil_commit_waiter+0xf3(ffffff03dcd15cc0, ffffff03d97ffcf8)
zil_commit+0x80(ffffff03dcd15cc0, 9a9)
zfs_write+0xc34(ffffff03dc38b140, ffffff0010555e60, 40, ffffff03e00fb758, 0)
fop_write+0x5b(ffffff03dc38b140, ffffff0010555e60, 40, ffffff03e00fb758, 0)
write+0x250(42, fffffd7ff4832000, 2000)
sys_syscall+0x177()
If there's an outstanding lwb that's in `zil_commit_waiter_timeout`
waiting to timeout, waiting on it's waiter's CV, we must be sure not to
call `zil_free_lwb`. If we end up calling `zil_free_lwb`, then that LWB
may be freed and can result in a use-after-free situation where the
stale lwb pointer stored in the `zil_commit_waiter_t` structure of the
thread waiting on the waiter's CV is used.
A similar situation can occur if an lwb is issued to disk, and thus in
the `LWB_STATE_ISSUED` state, and `zil_free_lwb` is called while the
disk is servicing that lwb. In this situation, the lwb will be freed by
`zil_free_lwb`, which will result in a use-after-free situation when the
lwb's zio completes, and `zil_lwb_flush_vdevs_done` is called.
This race condition is prevented in `zil_close` by calling `zil_commit`
before `zil_free_lwb` is called, which will ensure all outstanding (i.e.
all lwb's in the `LWB_STATE_OPEN` and/or `LWB_STATE_ISSUED` states)
reach the `LWB_STATE_DONE` state before the lwb's are freed
(`zil_commit` will not return untill all the lwb's are
`LWB_STATE_DONE`).
Further, this race condition is prevented in `zil_sync` by only calling
`zil_free_lwb` for lwb's that do not have their `lwb_buf` pointer set.
All lwb's not in the `LWB_STATE_DONE` state will have a non-null value
for this pointer; the pointer is only cleared in
`zil_lwb_flush_vdevs_done`, at which point the lwb's state will be
changed to `LWB_STATE_DONE`.
This race *is* present in `zil_suspend`, leading to this bug.
At first glance, it would appear as though this would not be true
because `zil_suspend` will call `zil_commit`, just like `zil_close`, but
the problem is that `zil_suspend` will set the zilog's `zl_suspend`
field prior to calling `zil_commit`. Further, in `zil_commit`, if
`zl_suspend` is set, `zil_commit` will take a special branch of logic
and use `txg_wait_synced` instead of performing the normal `zil_commit`
logic.
This call to `txg_wait_synced` might be good enough for the data to
reach disk safely before it returns, but it does not ensure that all
outstanding lwb's reach the `LWB_STATE_DONE` state before it returns.
This is because, if there's an lwb "stuck" in
`zil_commit_waiter_timeout`, waiting for it's lwb to timeout, it will
maintain a non-null value for it's `lwb_buf` field and thus `zil_sync`
will not free that lwb. Thus, even though the lwb's data is already on
disk, the lwb will be left lingering, waiting on the CV, and will
eventually timeout and be issued to disk even though the write is
unnecessary.
So, after `zil_commit` is called from `zil_suspend`, we incorrectly
assume that there are not outstanding lwb's, and proceed to free all
lwb's found on the zilog's lwb list. As a result, we free the lwb that
will later be used `zil_commit_waiter_timeout`.
SOLUTION
========
The solution to this, is to ensure all outstanding lwb's complete before
calling `zil_free_lwb` via `zil_destroy` in `zil_suspend`. This patch
accomplishes this goal by forcing the normal `zil_commit` logic when
called from `zil_sync`.
Now, `zil_suspend` will call `zil_commit_impl` which will always use the
normal logic of waiting/issuing lwb's to disk before it returns. As a
result, any lwb's outstanding when `zil_commit_impl` is called will be
guaranteed to reach the `LWB_STATE_DONE` state by the time it returns.
Further, no new lwb's will be created via `zil_commit` since the zilog's
`zl_suspend` flag will be set. This will force all new callers of
`zil_commit` to use `txg_wait_synced` instead of creating and issuing
new lwb's.
Thus, all lwb's left on the zilog's lwb list when `zil_destroy` is
called will be in the `LWB_STATE_DONE` state, and we'll avoid this race
condition.
OpenZFS-issue: https://www.illumos.org/issues/8909
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/ece62b6f8d
Closes #6940
2017-12-07 19:26:32 +00:00
|
|
|
limit is exceeded transaction records (itxs) will be cleaned synchronously.
|
2017-10-26 19:57:53 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB1048576\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_zil_clean_taskq_minalloc\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
The number of taskq entries that are pre-populated when the taskq is first
|
|
|
|
created and are immediately available for use.
|
|
|
|
.sp
|
|
|
|
Default value: \fB1024\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzfs_zil_clean_taskq_nthr_pct\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
This controls the number of threads used by the dp_zil_clean_taskq. The default
|
|
|
|
value of 100% will create a maximum of one thread per cpu.
|
|
|
|
.sp
|
2018-01-09 19:51:11 +00:00
|
|
|
Default value: \fB100\fR%.
|
2017-10-26 19:57:53 +00:00
|
|
|
.RE
|
|
|
|
|
2019-06-10 18:48:42 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzil_maxblocksize\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
This sets the maximum block size used by the ZIL. On very fragmented pools,
|
|
|
|
lowering this (typically to 36KB) can improve performance.
|
|
|
|
.sp
|
|
|
|
Default value: \fB131072\fR (128KB).
|
|
|
|
.RE
|
|
|
|
|
2018-10-23 16:44:37 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzil_nocacheflush\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Disable the cache flush commands that are normally sent to the disk(s) by
|
|
|
|
the ZIL after an LWB write has completed. Setting this will cause ZIL
|
|
|
|
corruption on power loss if a volatile out-of-order write cache is enabled.
|
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzil_replay_disable\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Disable intent logging replay. Can be disabled for recovery from corrupted
|
|
|
|
ZIL
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
OpenZFS 7578 - Fix/improve some aspects of ZIL writing
- After some ZIL changes 6 years ago zil_slog_limit got partially broken
due to zl_itx_list_sz not updated when async itx'es upgraded to sync.
Actually because of other changes about that time zl_itx_list_sz is not
really required to implement the functionality, so this patch removes
some unneeded broken code and variables.
- Original idea of zil_slog_limit was to reduce chance of SLOG abuse by
single heavy logger, that increased latency for other (more latency critical)
loggers, by pushing heavy log out into the main pool instead of SLOG. Beside
huge latency increase for heavy writers, this implementation caused double
write of all data, since the log records were explicitly prepared for SLOG.
Since we now have I/O scheduler, I've found it can be much more efficient
to reduce priority of heavy logger SLOG writes from ZIO_PRIORITY_SYNC_WRITE
to ZIO_PRIORITY_ASYNC_WRITE, while still leave them on SLOG.
- Existing ZIL implementation had problem with space efficiency when it
has to write large chunks of data into log blocks of limited size. In some
cases efficiency stopped to almost as low as 50%. In case of ZIL stored on
spinning rust, that also reduced log write speed in half, since head had to
uselessly fly over allocated but not written areas. This change improves
the situation by offloading problematic operations from z*_log_write() to
zil_lwb_commit(), which knows real situation of log blocks allocation and
can split large requests into pieces much more efficiently. Also as side
effect it removes one of two data copy operations done by ZIL code WR_COPIED
case.
- While there, untangle and unify code of z*_log_write() functions.
Also zfs_log_write() alike to zvol_log_write() can now handle writes crossing
block boundary, that may also improve efficiency if ZPL is made to do that.
Sponsored by: iXsystems, Inc.
Authored by: Alexander Motin <mav@FreeBSD.org>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Andriy Gapon <avg@FreeBSD.org>
Reviewed by: Steven Hartland <steven.hartland@multiplay.co.uk>
Reviewed by: Brad Lewis <brad.lewis@delphix.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Richard Yao <ryao@gentoo.org>
Ported-by: Giuseppe Di Natale <dinatale2@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/7578
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/aeb13ac
Closes #6191
2017-06-09 16:15:37 +00:00
|
|
|
\fBzil_slog_bulk\fR (ulong)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
OpenZFS 7578 - Fix/improve some aspects of ZIL writing
- After some ZIL changes 6 years ago zil_slog_limit got partially broken
due to zl_itx_list_sz not updated when async itx'es upgraded to sync.
Actually because of other changes about that time zl_itx_list_sz is not
really required to implement the functionality, so this patch removes
some unneeded broken code and variables.
- Original idea of zil_slog_limit was to reduce chance of SLOG abuse by
single heavy logger, that increased latency for other (more latency critical)
loggers, by pushing heavy log out into the main pool instead of SLOG. Beside
huge latency increase for heavy writers, this implementation caused double
write of all data, since the log records were explicitly prepared for SLOG.
Since we now have I/O scheduler, I've found it can be much more efficient
to reduce priority of heavy logger SLOG writes from ZIO_PRIORITY_SYNC_WRITE
to ZIO_PRIORITY_ASYNC_WRITE, while still leave them on SLOG.
- Existing ZIL implementation had problem with space efficiency when it
has to write large chunks of data into log blocks of limited size. In some
cases efficiency stopped to almost as low as 50%. In case of ZIL stored on
spinning rust, that also reduced log write speed in half, since head had to
uselessly fly over allocated but not written areas. This change improves
the situation by offloading problematic operations from z*_log_write() to
zil_lwb_commit(), which knows real situation of log blocks allocation and
can split large requests into pieces much more efficiently. Also as side
effect it removes one of two data copy operations done by ZIL code WR_COPIED
case.
- While there, untangle and unify code of z*_log_write() functions.
Also zfs_log_write() alike to zvol_log_write() can now handle writes crossing
block boundary, that may also improve efficiency if ZPL is made to do that.
Sponsored by: iXsystems, Inc.
Authored by: Alexander Motin <mav@FreeBSD.org>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Andriy Gapon <avg@FreeBSD.org>
Reviewed by: Steven Hartland <steven.hartland@multiplay.co.uk>
Reviewed by: Brad Lewis <brad.lewis@delphix.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Richard Yao <ryao@gentoo.org>
Ported-by: Giuseppe Di Natale <dinatale2@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/7578
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/aeb13ac
Closes #6191
2017-06-09 16:15:37 +00:00
|
|
|
Limit SLOG write size per commit executed with synchronous priority.
|
|
|
|
Any writes above that will be executed with lower (asynchronous) priority
|
|
|
|
to limit potential SLOG device abuse by single active ZIL writer.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
OpenZFS 7578 - Fix/improve some aspects of ZIL writing
- After some ZIL changes 6 years ago zil_slog_limit got partially broken
due to zl_itx_list_sz not updated when async itx'es upgraded to sync.
Actually because of other changes about that time zl_itx_list_sz is not
really required to implement the functionality, so this patch removes
some unneeded broken code and variables.
- Original idea of zil_slog_limit was to reduce chance of SLOG abuse by
single heavy logger, that increased latency for other (more latency critical)
loggers, by pushing heavy log out into the main pool instead of SLOG. Beside
huge latency increase for heavy writers, this implementation caused double
write of all data, since the log records were explicitly prepared for SLOG.
Since we now have I/O scheduler, I've found it can be much more efficient
to reduce priority of heavy logger SLOG writes from ZIO_PRIORITY_SYNC_WRITE
to ZIO_PRIORITY_ASYNC_WRITE, while still leave them on SLOG.
- Existing ZIL implementation had problem with space efficiency when it
has to write large chunks of data into log blocks of limited size. In some
cases efficiency stopped to almost as low as 50%. In case of ZIL stored on
spinning rust, that also reduced log write speed in half, since head had to
uselessly fly over allocated but not written areas. This change improves
the situation by offloading problematic operations from z*_log_write() to
zil_lwb_commit(), which knows real situation of log blocks allocation and
can split large requests into pieces much more efficiently. Also as side
effect it removes one of two data copy operations done by ZIL code WR_COPIED
case.
- While there, untangle and unify code of z*_log_write() functions.
Also zfs_log_write() alike to zvol_log_write() can now handle writes crossing
block boundary, that may also improve efficiency if ZPL is made to do that.
Sponsored by: iXsystems, Inc.
Authored by: Alexander Motin <mav@FreeBSD.org>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Andriy Gapon <avg@FreeBSD.org>
Reviewed by: Steven Hartland <steven.hartland@multiplay.co.uk>
Reviewed by: Brad Lewis <brad.lewis@delphix.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Richard Yao <ryao@gentoo.org>
Ported-by: Giuseppe Di Natale <dinatale2@llnl.gov>
OpenZFS-issue: https://www.illumos.org/issues/7578
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/aeb13ac
Closes #6191
2017-06-09 16:15:37 +00:00
|
|
|
Default value: \fB786,432\fR.
|
2013-11-16 06:52:54 +00:00
|
|
|
.RE
|
|
|
|
|
2019-02-15 20:44:24 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzio_deadman_log_all\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
If non-zero, the zio deadman will produce debugging messages (see
|
|
|
|
\fBzfs_dbgmsg_enable\fR) for all zios, rather than only for leaf
|
|
|
|
zios possessing a vdev. This is meant to be used by developers to gain
|
|
|
|
diagnostic information for hang conditions which don't involve a mutex
|
|
|
|
or other locking primitive; typically conditions in which a thread in
|
|
|
|
the zio pipeline is looping indefinitely.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2018-08-29 18:33:33 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzio_decompress_fail_fraction\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
If non-zero, this value represents the denominator of the probability that zfs
|
|
|
|
should induce a decompression failure. For instance, for a 5% decompression
|
|
|
|
failure rate, this value should be set to 20.
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
2018-11-09 00:47:24 +00:00
|
|
|
\fBzio_slow_io_ms\fR (int)
|
2013-11-16 06:52:54 +00:00
|
|
|
.ad
|
|
|
|
.RS 12n
|
2018-11-09 00:47:24 +00:00
|
|
|
When an I/O operation takes more than \fBzio_slow_io_ms\fR milliseconds to
|
|
|
|
complete is marked as a slow I/O. Each slow I/O causes a delay zevent. Slow
|
|
|
|
I/O counters can be seen with "zpool status -s".
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB30,000\fR.
|
|
|
|
.RE
|
|
|
|
|
2016-10-14 00:59:18 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzio_dva_throttle_enabled\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2018-11-09 00:47:24 +00:00
|
|
|
Throttle block allocations in the I/O pipeline. This allows for
|
2016-10-14 00:59:18 +00:00
|
|
|
dynamic allocation distribution when devices are imbalanced.
|
2017-04-25 04:01:04 +00:00
|
|
|
When enabled, the maximum number of pending allocations per top-level vdev
|
|
|
|
is limited by \fBzfs_vdev_queue_depth_pct\fR.
|
2016-10-14 00:59:18 +00:00
|
|
|
.sp
|
2016-12-08 20:57:42 +00:00
|
|
|
Default value: \fB1\fR.
|
2016-10-14 00:59:18 +00:00
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzio_requeue_io_start_cut_in_line\fR (int)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Prioritize requeued I/O
|
|
|
|
.sp
|
|
|
|
Default value: \fB0\fR.
|
|
|
|
.RE
|
|
|
|
|
2015-12-16 19:22:32 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzio_taskq_batch_pct\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Percentage of online CPUs (or CPU cores, etc) which will run a worker thread
|
2018-11-09 00:47:24 +00:00
|
|
|
for I/O. These workers are responsible for I/O work such as compression and
|
2015-12-16 19:22:32 +00:00
|
|
|
checksum calculations. Fractional number of CPUs will be rounded down.
|
|
|
|
.sp
|
|
|
|
The default value of 75 was chosen to avoid using all CPUs which can result in
|
|
|
|
latency issues and inconsistent application performance, especially when high
|
|
|
|
compression is enabled.
|
|
|
|
.sp
|
|
|
|
Default value: \fB75\fR.
|
|
|
|
.RE
|
|
|
|
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzvol_inhibit_dev\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Do not create zvol device nodes. This may slightly improve startup time on
|
|
|
|
systems with a very large number of zvols.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Use \fB1\fR for yes and \fB0\fR for no (default).
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzvol_major\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Major number for zvol block devices
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB230\fR.
|
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzvol_max_discard_blocks\fR (ulong)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
2015-12-30 17:44:46 +00:00
|
|
|
Discard (aka TRIM) operations done on zvols will be done in batches of this
|
|
|
|
many blocks, where block size is determined by the \fBvolblocksize\fR property
|
|
|
|
of a zvol.
|
2013-11-16 06:52:54 +00:00
|
|
|
.sp
|
|
|
|
Default value: \fB16,384\fR.
|
|
|
|
.RE
|
|
|
|
|
2015-08-18 20:51:20 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzvol_prefetch_bytes\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
When adding a zvol to the system prefetch \fBzvol_prefetch_bytes\fR
|
|
|
|
from the start and end of the volume. Prefetching these regions
|
|
|
|
of the volume is desirable because they are likely to be accessed
|
|
|
|
immediately by \fBblkid(8)\fR or by the kernel scanning for a partition
|
|
|
|
table.
|
|
|
|
.sp
|
|
|
|
Default value: \fB131,072\fR.
|
|
|
|
.RE
|
|
|
|
|
2017-02-23 00:08:04 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzvol_request_sync\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
When processing I/O requests for a zvol submit them synchronously. This
|
|
|
|
effectively limits the queue depth to 1 for each I/O submitter. When set
|
|
|
|
to 0 requests are handled asynchronously by a thread pool. The number of
|
|
|
|
requests which can be handled concurrently is controller by \fBzvol_threads\fR.
|
|
|
|
.sp
|
2017-05-03 00:37:14 +00:00
|
|
|
Default value: \fB0\fR.
|
2017-02-23 00:08:04 +00:00
|
|
|
.RE
|
|
|
|
|
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzvol_threads\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Max number of threads which can handle zvol I/O requests concurrently.
|
|
|
|
.sp
|
|
|
|
Default value: \fB32\fR.
|
|
|
|
.RE
|
|
|
|
|
2017-07-12 20:05:37 +00:00
|
|
|
.sp
|
|
|
|
.ne 2
|
|
|
|
.na
|
|
|
|
\fBzvol_volmode\fR (uint)
|
|
|
|
.ad
|
|
|
|
.RS 12n
|
|
|
|
Defines zvol block devices behaviour when \fBvolmode\fR is set to \fBdefault\fR.
|
|
|
|
Valid values are \fB1\fR (full), \fB2\fR (dev) and \fB3\fR (none).
|
|
|
|
.sp
|
|
|
|
Default value: \fB1\fR.
|
|
|
|
.RE
|
|
|
|
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
.SH ZFS I/O SCHEDULER
|
|
|
|
ZFS issues I/O operations to leaf vdevs to satisfy and complete I/Os.
|
|
|
|
The I/O scheduler determines when and in what order those operations are
|
|
|
|
issued. The I/O scheduler divides operations into five I/O classes
|
|
|
|
prioritized in the following order: sync read, sync write, async read,
|
|
|
|
async write, and scrub/resilver. Each queue defines the minimum and
|
|
|
|
maximum number of concurrent operations that may be issued to the
|
|
|
|
device. In addition, the device has an aggregate maximum,
|
|
|
|
\fBzfs_vdev_max_active\fR. Note that the sum of the per-queue minimums
|
|
|
|
must not exceed the aggregate maximum. If the sum of the per-queue
|
|
|
|
maximums exceeds the aggregate maximum, then the number of active I/Os
|
|
|
|
may reach \fBzfs_vdev_max_active\fR, in which case no further I/Os will
|
|
|
|
be issued regardless of whether all per-queue minimums have been met.
|
|
|
|
.sp
|
|
|
|
For many physical devices, throughput increases with the number of
|
|
|
|
concurrent operations, but latency typically suffers. Further, physical
|
|
|
|
devices typically have a limit at which more concurrent operations have no
|
|
|
|
effect on throughput or can actually cause it to decrease.
|
|
|
|
.sp
|
|
|
|
The scheduler selects the next operation to issue by first looking for an
|
|
|
|
I/O class whose minimum has not been satisfied. Once all are satisfied and
|
|
|
|
the aggregate maximum has not been hit, the scheduler looks for classes
|
|
|
|
whose maximum has not been satisfied. Iteration through the I/O classes is
|
|
|
|
done in the order specified above. No further operations are issued if the
|
|
|
|
aggregate maximum number of concurrent operations has been hit or if there
|
|
|
|
are no operations queued for an I/O class that has not hit its maximum.
|
|
|
|
Every time an I/O is queued or an operation completes, the I/O scheduler
|
|
|
|
looks for new operations to issue.
|
|
|
|
.sp
|
|
|
|
In general, smaller max_active's will lead to lower latency of synchronous
|
|
|
|
operations. Larger max_active's may lead to higher overall throughput,
|
|
|
|
depending on underlying storage.
|
|
|
|
.sp
|
|
|
|
The ratio of the queues' max_actives determines the balance of performance
|
|
|
|
between reads, writes, and scrubs. E.g., increasing
|
|
|
|
\fBzfs_vdev_scrub_max_active\fR will cause the scrub or resilver to complete
|
|
|
|
more quickly, but reads and writes to have higher latency and lower throughput.
|
|
|
|
.sp
|
|
|
|
All I/O classes have a fixed maximum number of outstanding operations
|
|
|
|
except for the async write class. Asynchronous writes represent the data
|
|
|
|
that is committed to stable storage during the syncing stage for
|
|
|
|
transaction groups. Transaction groups enter the syncing state
|
|
|
|
periodically so the number of queued async writes will quickly burst up
|
|
|
|
and then bleed down to zero. Rather than servicing them as quickly as
|
|
|
|
possible, the I/O scheduler changes the maximum number of active async
|
|
|
|
write I/Os according to the amount of dirty data in the pool. Since
|
|
|
|
both throughput and latency typically increase with the number of
|
|
|
|
concurrent operations issued to physical devices, reducing the
|
|
|
|
burstiness in the number of concurrent operations also stabilizes the
|
|
|
|
response time of operations from other -- and in particular synchronous
|
|
|
|
-- queues. In broad strokes, the I/O scheduler will issue more
|
|
|
|
concurrent operations from the async write queue as there's more dirty
|
|
|
|
data in the pool.
|
|
|
|
.sp
|
|
|
|
Async Writes
|
|
|
|
.sp
|
|
|
|
The number of concurrent operations issued for the async write I/O class
|
|
|
|
follows a piece-wise linear function defined by a few adjustable points.
|
|
|
|
.nf
|
|
|
|
|
|
|
|
| o---------| <-- zfs_vdev_async_write_max_active
|
|
|
|
^ | /^ |
|
|
|
|
| | / | |
|
|
|
|
active | / | |
|
|
|
|
I/O | / | |
|
|
|
|
count | / | |
|
|
|
|
| / | |
|
|
|
|
|-------o | | <-- zfs_vdev_async_write_min_active
|
|
|
|
0|_______^______|_________|
|
|
|
|
0% | | 100% of zfs_dirty_data_max
|
|
|
|
| |
|
|
|
|
| `-- zfs_vdev_async_write_active_max_dirty_percent
|
|
|
|
`--------- zfs_vdev_async_write_active_min_dirty_percent
|
|
|
|
|
|
|
|
.fi
|
|
|
|
Until the amount of dirty data exceeds a minimum percentage of the dirty
|
|
|
|
data allowed in the pool, the I/O scheduler will limit the number of
|
|
|
|
concurrent operations to the minimum. As that threshold is crossed, the
|
|
|
|
number of concurrent operations issued increases linearly to the maximum at
|
|
|
|
the specified maximum percentage of the dirty data allowed in the pool.
|
|
|
|
.sp
|
|
|
|
Ideally, the amount of dirty data on a busy pool will stay in the sloped
|
|
|
|
part of the function between \fBzfs_vdev_async_write_active_min_dirty_percent\fR
|
|
|
|
and \fBzfs_vdev_async_write_active_max_dirty_percent\fR. If it exceeds the
|
|
|
|
maximum percentage, this indicates that the rate of incoming data is
|
|
|
|
greater than the rate that the backend storage can handle. In this case, we
|
|
|
|
must further throttle incoming writes, as described in the next section.
|
|
|
|
|
|
|
|
.SH ZFS TRANSACTION DELAY
|
|
|
|
We delay transactions when we've determined that the backend storage
|
|
|
|
isn't able to accommodate the rate of incoming writes.
|
|
|
|
.sp
|
|
|
|
If there is already a transaction waiting, we delay relative to when
|
|
|
|
that transaction will finish waiting. This way the calculated delay time
|
|
|
|
is independent of the number of threads concurrently executing
|
|
|
|
transactions.
|
|
|
|
.sp
|
|
|
|
If we are the only waiter, wait relative to when the transaction
|
|
|
|
started, rather than the current time. This credits the transaction for
|
|
|
|
"time already served", e.g. reading indirect blocks.
|
|
|
|
.sp
|
|
|
|
The minimum time for a transaction to take is calculated as:
|
|
|
|
.nf
|
|
|
|
min_time = zfs_delay_scale * (dirty - min) / (max - dirty)
|
|
|
|
min_time is then capped at 100 milliseconds.
|
|
|
|
.fi
|
|
|
|
.sp
|
|
|
|
The delay has two degrees of freedom that can be adjusted via tunables. The
|
|
|
|
percentage of dirty data at which we start to delay is defined by
|
|
|
|
\fBzfs_delay_min_dirty_percent\fR. This should typically be at or above
|
|
|
|
\fBzfs_vdev_async_write_active_max_dirty_percent\fR so that we only start to
|
|
|
|
delay after writing at full speed has failed to keep up with the incoming write
|
|
|
|
rate. The scale of the curve is defined by \fBzfs_delay_scale\fR. Roughly speaking,
|
|
|
|
this variable determines the amount of delay at the midpoint of the curve.
|
|
|
|
.sp
|
|
|
|
.nf
|
|
|
|
delay
|
|
|
|
10ms +-------------------------------------------------------------*+
|
|
|
|
| *|
|
|
|
|
9ms + *+
|
|
|
|
| *|
|
|
|
|
8ms + *+
|
|
|
|
| * |
|
|
|
|
7ms + * +
|
|
|
|
| * |
|
|
|
|
6ms + * +
|
|
|
|
| * |
|
|
|
|
5ms + * +
|
|
|
|
| * |
|
|
|
|
4ms + * +
|
|
|
|
| * |
|
|
|
|
3ms + * +
|
|
|
|
| * |
|
|
|
|
2ms + (midpoint) * +
|
|
|
|
| | ** |
|
|
|
|
1ms + v *** +
|
|
|
|
| zfs_delay_scale ----------> ******** |
|
|
|
|
0 +-------------------------------------*********----------------+
|
|
|
|
0% <- zfs_dirty_data_max -> 100%
|
|
|
|
.fi
|
|
|
|
.sp
|
|
|
|
Note that since the delay is added to the outstanding time remaining on the
|
|
|
|
most recent transaction, the delay is effectively the inverse of IOPS.
|
|
|
|
Here the midpoint of 500us translates to 2000 IOPS. The shape of the curve
|
|
|
|
was chosen such that small changes in the amount of accumulated dirty data
|
|
|
|
in the first 3/4 of the curve yield relatively small differences in the
|
|
|
|
amount of delay.
|
|
|
|
.sp
|
|
|
|
The effects can be easier to understand when the amount of delay is
|
|
|
|
represented on a log scale:
|
|
|
|
.sp
|
|
|
|
.nf
|
|
|
|
delay
|
|
|
|
100ms +-------------------------------------------------------------++
|
|
|
|
+ +
|
|
|
|
| |
|
|
|
|
+ *+
|
|
|
|
10ms + *+
|
|
|
|
+ ** +
|
|
|
|
| (midpoint) ** |
|
|
|
|
+ | ** +
|
|
|
|
1ms + v **** +
|
|
|
|
+ zfs_delay_scale ----------> ***** +
|
|
|
|
| **** |
|
|
|
|
+ **** +
|
|
|
|
100us + ** +
|
|
|
|
+ * +
|
|
|
|
| * |
|
|
|
|
+ * +
|
|
|
|
10us + * +
|
|
|
|
+ +
|
|
|
|
| |
|
|
|
|
+ +
|
|
|
|
+--------------------------------------------------------------+
|
|
|
|
0% <- zfs_dirty_data_max -> 100%
|
|
|
|
.fi
|
|
|
|
.sp
|
|
|
|
Note here that only as the amount of dirty data approaches its limit does
|
|
|
|
the delay start to increase rapidly. The goal of a properly tuned system
|
|
|
|
should be to keep the amount of dirty data out of that range by first
|
|
|
|
ensuring that the appropriate limits are set for the I/O scheduler to reach
|
|
|
|
optimal throughput on the backend storage, and then by changing the value
|
|
|
|
of \fBzfs_delay_scale\fR to increase the steepness of the curve.
|