2012-02-04 05:44:53 +00:00
|
|
|
.\"
|
|
|
|
|
.\" This file and its contents are supplied under the terms of the
|
|
|
|
|
.\" Common Development and Distribution License ("CDDL"), version 1.0.
|
|
|
|
|
.\" You may only use this file in accordance with the terms of version
|
|
|
|
|
.\" 1.0 of the CDDL.
|
|
|
|
|
.\"
|
|
|
|
|
.\" A full copy of the text of the CDDL should have accompanied this
|
|
|
|
|
.\" source. A copy of the CDDL is also available via the Internet at
|
|
|
|
|
.\" http://www.illumos.org/license/CDDL.
|
|
|
|
|
.\"
|
|
|
|
|
.\" Copyright 2012, Richard Lowe.
|
Extend zdb to print inconsistencies in livelists and metaslabs
Livelists and spacemaps are data structures that are logs of allocations
and frees. Livelists entries are block pointers (blkptr_t). Spacemaps
entries are ranges of numbers, most often used as to track
allocated/freed regions of metaslabs/vdevs.
These data structures can become self-inconsistent, for example if a
block or range can be "double allocated" (two allocation records without
an intervening free) or "double freed" (two free records without an
intervening allocation).
ZDB (as well as zfs running in the kernel) can detect these
inconsistencies when loading livelists and metaslab. However, it
generally halts processing when the error is detected.
When analyzing an on-disk problem, we often want to know the entire set
of inconsistencies, which is not possible with the current behavior.
This commit adds a new flag, `zdb -y`, which analyzes the livelist and
metaslab data structures and displays all of their inconsistencies.
Note that this is different from the leak detection performed by
`zdb -b`, which checks for inconsistencies between the spacemaps and the
tree of block pointers, but assumes the spacemaps are self-consistent.
The specific checks added are:
Verify livelists by iterating through each sublivelists and:
- report leftover FREEs
- report double ALLOCs and double FREEs
- record leftover ALLOCs together with their TXG [see Cross Check]
Verify spacemaps by iterating over each metaslab and:
- iterate over spacemap and then the metaslab's entries in the
spacemap log, then report any double FREEs and double ALLOCs
Verify that livelists are consistenet with spacemaps. The space
referenced by livelists (after using the FREE's to cancel out
corresponding ALLOCs) should be allocated, according to the spacemaps.
Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: Sara Hartse <sara.hartse@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
External-issue: DLPX-66031
Closes #10515
2020-07-15 00:51:05 +00:00
|
|
|
.\" Copyright (c) 2012, 2019 by Delphix. All rights reserved.
|
2017-04-13 16:40:56 +00:00
|
|
|
.\" Copyright 2017 Nexenta Systems, Inc.
|
|
|
|
|
.\" Copyright (c) 2017 Lawrence Livermore National Security, LLC.
|
|
|
|
|
.\" Copyright (c) 2017 Intel Corporation.
|
2012-02-04 05:44:53 +00:00
|
|
|
.\"
|
2021-01-28 05:36:01 +00:00
|
|
|
.Dd October 7, 2020
|
2021-05-27 00:46:40 +00:00
|
|
|
.Dt ZDB 8
|
2020-08-21 18:55:47 +00:00
|
|
|
.Os
|
2021-05-27 00:46:40 +00:00
|
|
|
.
|
2017-04-13 16:40:56 +00:00
|
|
|
.Sh NAME
|
|
|
|
|
.Nm zdb
|
2021-05-27 00:46:40 +00:00
|
|
|
.Nd display ZFS storage pool debugging and consistency information
|
2017-04-13 16:40:56 +00:00
|
|
|
.Sh SYNOPSIS
|
|
|
|
|
.Nm
|
Extend zdb to print inconsistencies in livelists and metaslabs
Livelists and spacemaps are data structures that are logs of allocations
and frees. Livelists entries are block pointers (blkptr_t). Spacemaps
entries are ranges of numbers, most often used as to track
allocated/freed regions of metaslabs/vdevs.
These data structures can become self-inconsistent, for example if a
block or range can be "double allocated" (two allocation records without
an intervening free) or "double freed" (two free records without an
intervening allocation).
ZDB (as well as zfs running in the kernel) can detect these
inconsistencies when loading livelists and metaslab. However, it
generally halts processing when the error is detected.
When analyzing an on-disk problem, we often want to know the entire set
of inconsistencies, which is not possible with the current behavior.
This commit adds a new flag, `zdb -y`, which analyzes the livelist and
metaslab data structures and displays all of their inconsistencies.
Note that this is different from the leak detection performed by
`zdb -b`, which checks for inconsistencies between the spacemaps and the
tree of block pointers, but assumes the spacemaps are self-consistent.
The specific checks added are:
Verify livelists by iterating through each sublivelists and:
- report leftover FREEs
- report double ALLOCs and double FREEs
- record leftover ALLOCs together with their TXG [see Cross Check]
Verify spacemaps by iterating over each metaslab and:
- iterate over spacemap and then the metaslab's entries in the
spacemap log, then report any double FREEs and double ALLOCs
Verify that livelists are consistenet with spacemaps. The space
referenced by livelists (after using the FREE's to cancel out
corresponding ALLOCs) should be allocated, according to the spacemaps.
Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: Sara Hartse <sara.hartse@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
External-issue: DLPX-66031
Closes #10515
2020-07-15 00:51:05 +00:00
|
|
|
.Op Fl AbcdDFGhikLMPsvXYy
|
2021-05-27 00:46:40 +00:00
|
|
|
.Op Fl e Oo Fl V Oc Oo Fl p Ar path Oc Ns …
|
2017-04-13 16:40:56 +00:00
|
|
|
.Op Fl I Ar inflight I/Os
|
2021-05-27 00:46:40 +00:00
|
|
|
.Oo Fl o Ar var Ns = Ns Ar value Oc Ns …
|
2017-04-13 16:40:56 +00:00
|
|
|
.Op Fl t Ar txg
|
|
|
|
|
.Op Fl U Ar cache
|
|
|
|
|
.Op Fl x Ar dumpdir
|
2021-05-27 00:46:40 +00:00
|
|
|
.Op Ar poolname Ns Op / Ns Ar dataset | objset ID
|
|
|
|
|
.Op Ar object Ns | Ns Ar range Ns …
|
2017-04-13 16:40:56 +00:00
|
|
|
.Nm
|
|
|
|
|
.Op Fl AdiPv
|
2021-05-27 00:46:40 +00:00
|
|
|
.Op Fl e Oo Fl V Oc Oo Fl p Ar path Oc Ns …
|
2017-04-13 16:40:56 +00:00
|
|
|
.Op Fl U Ar cache
|
2021-05-27 00:46:40 +00:00
|
|
|
.Ar poolname Ns Op Ar / Ns Ar dataset | objset ID
|
|
|
|
|
.Op Ar object Ns | Ns Ar range Ns …
|
2017-04-13 16:40:56 +00:00
|
|
|
.Nm
|
|
|
|
|
.Fl C
|
|
|
|
|
.Op Fl A
|
|
|
|
|
.Op Fl U Ar cache
|
|
|
|
|
.Nm
|
2017-05-01 18:06:07 +00:00
|
|
|
.Fl E
|
|
|
|
|
.Op Fl A
|
2021-05-27 00:46:40 +00:00
|
|
|
.Ar word0 : Ns Ar word1 Ns :…: Ns Ar word15
|
2017-05-01 18:06:07 +00:00
|
|
|
.Nm
|
2017-04-13 16:40:56 +00:00
|
|
|
.Fl l
|
|
|
|
|
.Op Fl Aqu
|
|
|
|
|
.Ar device
|
|
|
|
|
.Nm
|
|
|
|
|
.Fl m
|
2019-01-16 22:10:02 +00:00
|
|
|
.Op Fl AFLPXY
|
2021-05-27 00:46:40 +00:00
|
|
|
.Op Fl e Oo Fl V Oc Oo Fl p Ar path Oc Ns …
|
2017-04-13 16:40:56 +00:00
|
|
|
.Op Fl t Ar txg
|
|
|
|
|
.Op Fl U Ar cache
|
2021-05-27 00:46:40 +00:00
|
|
|
.Ar poolname Op Ar vdev Oo Ar metaslab Oc Ns …
|
2017-04-13 16:40:56 +00:00
|
|
|
.Nm
|
|
|
|
|
.Fl O
|
|
|
|
|
.Ar dataset path
|
|
|
|
|
.Nm
|
2021-01-28 05:36:01 +00:00
|
|
|
.Fl r
|
|
|
|
|
.Ar dataset path destination
|
|
|
|
|
.Nm
|
2017-04-13 16:40:56 +00:00
|
|
|
.Fl R
|
|
|
|
|
.Op Fl A
|
2021-05-27 00:46:40 +00:00
|
|
|
.Op Fl e Oo Fl V Oc Oo Fl p Ar path Oc Ns …
|
2017-04-13 16:40:56 +00:00
|
|
|
.Op Fl U Ar cache
|
2021-05-27 00:46:40 +00:00
|
|
|
.Ar poolname vdev : Ns Ar offset : Ns Oo Ar lsize Ns / Oc Ns Ar psize Ns Op : Ns Ar flags
|
2017-04-13 16:40:56 +00:00
|
|
|
.Nm
|
|
|
|
|
.Fl S
|
|
|
|
|
.Op Fl AP
|
2021-05-27 00:46:40 +00:00
|
|
|
.Op Fl e Oo Fl V Oc Oo Fl p Ar path Oc Ns …
|
2017-04-13 16:40:56 +00:00
|
|
|
.Op Fl U Ar cache
|
|
|
|
|
.Ar poolname
|
2021-05-27 00:46:40 +00:00
|
|
|
.
|
2017-04-13 16:40:56 +00:00
|
|
|
.Sh DESCRIPTION
|
|
|
|
|
The
|
|
|
|
|
.Nm
|
|
|
|
|
utility displays information about a ZFS pool useful for debugging and performs
|
|
|
|
|
some amount of consistency checking.
|
|
|
|
|
It is a not a general purpose tool and options
|
|
|
|
|
.Pq and facilities
|
|
|
|
|
may change.
|
2019-04-15 02:06:34 +00:00
|
|
|
This is not a
|
|
|
|
|
.Xr fsck 8
|
2017-04-13 16:40:56 +00:00
|
|
|
utility.
|
|
|
|
|
.Pp
|
2012-02-04 05:44:53 +00:00
|
|
|
The output of this command in general reflects the on-disk structure of a ZFS
|
2017-04-13 16:40:56 +00:00
|
|
|
pool, and is inherently unstable.
|
|
|
|
|
The precise output of most invocations is not documented, a knowledge of ZFS
|
|
|
|
|
internals is assumed.
|
|
|
|
|
.Pp
|
|
|
|
|
If the
|
|
|
|
|
.Ar dataset
|
|
|
|
|
argument does not contain any
|
|
|
|
|
.Qq Sy /
|
|
|
|
|
or
|
|
|
|
|
.Qq Sy @
|
|
|
|
|
characters, it is interpreted as a pool name.
|
|
|
|
|
The root dataset can be specified as
|
2021-05-27 00:46:40 +00:00
|
|
|
.Qq Ar pool Ns / .
|
2017-04-13 16:40:56 +00:00
|
|
|
.Pp
|
2012-02-04 05:44:53 +00:00
|
|
|
When operating on an imported and active pool it is possible, though unlikely,
|
|
|
|
|
that zdb may interpret inconsistent pool data and behave erratically.
|
2021-05-27 00:46:40 +00:00
|
|
|
.
|
2017-04-13 16:40:56 +00:00
|
|
|
.Sh OPTIONS
|
2012-02-04 05:44:53 +00:00
|
|
|
Display options:
|
2017-04-13 16:40:56 +00:00
|
|
|
.Bl -tag -width Ds
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl b , -block-stats
|
2017-04-13 16:40:56 +00:00
|
|
|
Display statistics regarding the number, size
|
|
|
|
|
.Pq logical, physical and allocated
|
|
|
|
|
and deduplication of blocks.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl c , -checksum
|
2012-02-04 05:44:53 +00:00
|
|
|
Verify the checksum of all metadata blocks while printing block statistics
|
2017-04-13 16:40:56 +00:00
|
|
|
.Po see
|
|
|
|
|
.Fl b
|
|
|
|
|
.Pc .
|
|
|
|
|
.Pp
|
2012-02-04 05:44:53 +00:00
|
|
|
If specified multiple times, verify the checksums of all blocks.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl C , -config
|
2017-04-13 16:40:56 +00:00
|
|
|
Display information about the configuration.
|
|
|
|
|
If specified with no other options, instead display information about the cache
|
|
|
|
|
file
|
|
|
|
|
.Pq Pa /etc/zfs/zpool.cache .
|
|
|
|
|
To specify the cache file to display, see
|
|
|
|
|
.Fl U .
|
|
|
|
|
.Pp
|
|
|
|
|
If specified multiple times, and a pool name is also specified display both the
|
|
|
|
|
cached configuration and the on-disk configuration.
|
|
|
|
|
If specified multiple times with
|
|
|
|
|
.Fl e
|
|
|
|
|
also display the configuration that would be used were the pool to be imported.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl d , -datasets
|
2017-04-13 16:40:56 +00:00
|
|
|
Display information about datasets.
|
|
|
|
|
Specified once, displays basic dataset information: ID, create transaction,
|
|
|
|
|
size, and object count.
|
|
|
|
|
.Pp
|
2012-02-04 05:44:53 +00:00
|
|
|
If specified multiple times provides greater and greater verbosity.
|
2017-04-13 16:40:56 +00:00
|
|
|
.Pp
|
2020-01-24 19:00:46 +00:00
|
|
|
If object IDs or object ID ranges are specified, display information about
|
|
|
|
|
those specific objects or ranges only.
|
|
|
|
|
.Pp
|
|
|
|
|
An object ID range is specified in terms of a colon-separated tuple of
|
|
|
|
|
the form
|
2021-05-27 00:46:40 +00:00
|
|
|
.Ao start Ac : Ns Ao end Ac Ns Op : Ns Ao flags Ac .
|
2020-01-24 19:00:46 +00:00
|
|
|
The fields
|
|
|
|
|
.Ar start
|
|
|
|
|
and
|
|
|
|
|
.Ar end
|
2020-06-10 04:24:09 +00:00
|
|
|
are integer object identifiers that denote the upper and lower bounds
|
2021-05-27 00:46:40 +00:00
|
|
|
of the range.
|
|
|
|
|
An
|
2020-01-24 19:00:46 +00:00
|
|
|
.Ar end
|
2021-05-27 00:46:40 +00:00
|
|
|
value of -1 specifies a range with no upper bound.
|
|
|
|
|
The
|
2020-01-24 19:00:46 +00:00
|
|
|
.Ar flags
|
|
|
|
|
field optionally specifies a set of flags, described below, that control
|
2021-05-27 00:46:40 +00:00
|
|
|
which object types are dumped.
|
|
|
|
|
By default, all object types are dumped.
|
|
|
|
|
A minus sign
|
2020-01-24 19:00:46 +00:00
|
|
|
.Pq -
|
|
|
|
|
negates the effect of the flag that follows it and has no effect unless
|
|
|
|
|
preceded by the
|
|
|
|
|
.Ar A
|
2021-05-27 00:46:40 +00:00
|
|
|
flag.
|
|
|
|
|
For example, the range 0:-1:A-d will dump all object types except for directories.
|
2020-01-24 19:00:46 +00:00
|
|
|
.Pp
|
2021-05-27 00:46:40 +00:00
|
|
|
.Bl -tag -compact -width Ds
|
2020-01-24 19:00:46 +00:00
|
|
|
.It Sy A
|
|
|
|
|
Dump all objects (this is the default)
|
|
|
|
|
.It Sy d
|
|
|
|
|
Dump ZFS directory objects
|
|
|
|
|
.It Sy f
|
|
|
|
|
Dump ZFS plain file objects
|
|
|
|
|
.It Sy m
|
|
|
|
|
Dump SPA space map objects
|
|
|
|
|
.It Sy z
|
|
|
|
|
Dump ZAP objects
|
|
|
|
|
.It Sy -
|
|
|
|
|
Negate the effect of next flag
|
|
|
|
|
.El
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl D , -dedup-stats
|
2017-04-13 16:40:56 +00:00
|
|
|
Display deduplication statistics, including the deduplication ratio
|
|
|
|
|
.Pq Sy dedup ,
|
|
|
|
|
compression ratio
|
|
|
|
|
.Pq Sy compress ,
|
|
|
|
|
inflation due to the zfs copies property
|
|
|
|
|
.Pq Sy copies ,
|
|
|
|
|
and an overall effective ratio
|
|
|
|
|
.Pq Sy dedup No * Sy compress No / Sy copies .
|
|
|
|
|
.It Fl DD
|
|
|
|
|
Display a histogram of deduplication statistics, showing the allocated
|
|
|
|
|
.Pq physically present on disk
|
|
|
|
|
and referenced
|
|
|
|
|
.Pq logically referenced in the pool
|
|
|
|
|
block counts and sizes by reference count.
|
|
|
|
|
.It Fl DDD
|
|
|
|
|
Display the statistics independently for each deduplication table.
|
|
|
|
|
.It Fl DDDD
|
|
|
|
|
Dump the contents of the deduplication tables describing duplicate blocks.
|
|
|
|
|
.It Fl DDDDD
|
|
|
|
|
Also dump the contents of the deduplication tables describing unique blocks.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl E , -embedded-block-pointer Ns = Ns Ar word0 : Ns Ar word1 Ns :…: Ns Ar word15
|
2017-05-01 18:06:07 +00:00
|
|
|
Decode and display block from an embedded block pointer specified by the
|
|
|
|
|
.Ar word
|
|
|
|
|
arguments.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl h , -history
|
2017-04-13 16:40:56 +00:00
|
|
|
Display pool history similar to
|
|
|
|
|
.Nm zpool Cm history ,
|
|
|
|
|
but include internal changes, transaction, and dataset information.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl i , -intent-logs
|
2017-04-13 16:40:56 +00:00
|
|
|
Display information about intent log
|
|
|
|
|
.Pq ZIL
|
|
|
|
|
entries relating to each dataset.
|
|
|
|
|
If specified multiple times, display counts of each intent log transaction type.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl k , -checkpointed-state
|
2016-12-16 22:11:29 +00:00
|
|
|
Examine the checkpointed state of the pool.
|
|
|
|
|
Note, the on disk format of the pool is not reverted to the checkpointed state.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl l , -label Ns = Ns Ar device
|
2020-04-10 17:33:35 +00:00
|
|
|
Read the vdev labels and L2ARC header from the specified device.
|
2017-04-13 16:40:56 +00:00
|
|
|
.Nm Fl l
|
|
|
|
|
will return 0 if valid label was found, 1 if error occurred, and 2 if no valid
|
2021-05-27 00:46:40 +00:00
|
|
|
labels were found.
|
|
|
|
|
The presence of L2ARC header is indicated by a specific
|
|
|
|
|
sequence (L2ARC_DEV_HDR_MAGIC).
|
|
|
|
|
If there is an accounting error in the size or the number of L2ARC log blocks
|
2020-05-07 23:34:03 +00:00
|
|
|
.Nm Fl l
|
2021-05-27 00:46:40 +00:00
|
|
|
will return 1.
|
|
|
|
|
Each unique configuration is displayed only once.
|
2017-04-13 16:40:56 +00:00
|
|
|
.It Fl ll Ar device
|
2021-05-27 00:46:40 +00:00
|
|
|
In addition display label space usage stats.
|
|
|
|
|
If a valid L2ARC header was found
|
2020-04-10 17:33:35 +00:00
|
|
|
also display the properties of log blocks used for restoring L2ARC contents
|
|
|
|
|
(persistent L2ARC).
|
2017-04-13 16:40:56 +00:00
|
|
|
.It Fl lll Ar device
|
2021-05-27 00:46:40 +00:00
|
|
|
Display every configuration, unique or not.
|
|
|
|
|
If a valid L2ARC header was found
|
2020-04-10 17:33:35 +00:00
|
|
|
also display the properties of log entries in log blocks used for restoring
|
|
|
|
|
L2ARC contents (persistent L2ARC).
|
2017-04-13 16:40:56 +00:00
|
|
|
.Pp
|
|
|
|
|
If the
|
|
|
|
|
.Fl q
|
2020-04-10 17:33:35 +00:00
|
|
|
option is also specified, don't print the labels or the L2ARC header.
|
2017-04-13 16:40:56 +00:00
|
|
|
.Pp
|
|
|
|
|
If the
|
|
|
|
|
.Fl u
|
2021-05-27 00:46:40 +00:00
|
|
|
option is also specified, also display the uberblocks on this device.
|
|
|
|
|
Specify multiple times to increase verbosity.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl L , -disable-leak-tracking
|
2019-01-30 17:54:27 +00:00
|
|
|
Disable leak detection and the loading of space maps.
|
2017-04-13 16:40:56 +00:00
|
|
|
By default,
|
|
|
|
|
.Nm
|
2012-02-04 05:44:53 +00:00
|
|
|
verifies that all non-free blocks are referenced, which can be very expensive.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl m , -metaslabs
|
Log Spacemap Project
= Motivation
At Delphix we've seen a lot of customer systems where fragmentation
is over 75% and random writes take a performance hit because a lot
of time is spend on I/Os that update on-disk space accounting metadata.
Specifically, we seen cases where 20% to 40% of sync time is spend
after sync pass 1 and ~30% of the I/Os on the system is spent updating
spacemaps.
The problem is that these pools have existed long enough that we've
touched almost every metaslab at least once, and random writes
scatter frees across all metaslabs every TXG, thus appending to
their spacemaps and resulting in many I/Os. To give an example,
assuming that every VDEV has 200 metaslabs and our writes fit within
a single spacemap block (generally 4K) we have 200 I/Os. Then if we
assume 2 levels of indirection, we need 400 additional I/Os and
since we are talking about metadata for which we keep 2 extra copies
for redundancy we need to triple that number, leading to a total of
1800 I/Os per VDEV every TXG.
We could try and decrease the number of metaslabs so we have less
I/Os per TXG but then each metaslab would cover a wider range on
disk and thus would take more time to be loaded in memory from disk.
In addition, after it's loaded, it's range tree would consume more
memory.
Another idea would be to just increase the spacemap block size
which would allow us to fit more entries within an I/O block
resulting in fewer I/Os per metaslab and a speedup in loading time.
The problem is still that we don't deal with the number of I/Os
going up as the number of metaslabs is increasing and the fact
is that we generally write a lot to a few metaslabs and a little
to the rest of them. Thus, just increasing the block size would
actually waste bandwidth because we won't be utilizing our bigger
block size.
= About this patch
This patch introduces the Log Spacemap project which provides the
solution to the above problem while taking into account all the
aforementioned tradeoffs. The details on how it achieves that can
be found in the references sections below and in the code (see
Big Theory Statement in spa_log_spacemap.c).
Even though the change is fairly constraint within the metaslab
and lower-level SPA codepaths, there is a side-change that is
user-facing. The change is that VDEV IDs from VDEV holes will no
longer be reused. To give some background and reasoning for this,
when a log device is removed and its VDEV structure was replaced
with a hole (or was compacted; if at the end of the vdev array),
its vdev_id could be reused by devices added after that. Now
with the pool-wide space maps recording the vdev ID, this behavior
can cause problems (e.g. is this entry referring to a segment in
the new vdev or the removed log?). Thus, to simplify things the
ID reuse behavior is gone and now vdev IDs for top-level vdevs
are truly unique within a pool.
= Testing
The illumos implementation of this feature has been used internally
for a year and has been in production for ~6 months. For this patch
specifically there don't seem to be any regressions introduced to
ZTS and I have been running zloop for a week without any related
problems.
= Performance Analysis (Linux Specific)
All performance results and analysis for illumos can be found in
the links of the references. Redoing the same experiments in Linux
gave similar results. Below are the specifics of the Linux run.
After the pool reached stable state the percentage of the time
spent in pass 1 per TXG was 64% on average for the stock bits
while the log spacemap bits stayed at 95% during the experiment
(graph: sdimitro.github.io/img/linux-lsm/PercOfSyncInPassOne.png).
Sync times per TXG were 37.6 seconds on average for the stock
bits and 22.7 seconds for the log spacemap bits (related graph:
sdimitro.github.io/img/linux-lsm/SyncTimePerTXG.png). As a result
the log spacemap bits were able to push more TXGs, which is also
the reason why all graphs quantified per TXG have more entries for
the log spacemap bits.
Another interesting aspect in terms of txg syncs is that the stock
bits had 22% of their TXGs reach sync pass 7, 55% reach sync pass 8,
and 20% reach 9. The log space map bits reached sync pass 4 in 79%
of their TXGs, sync pass 7 in 19%, and sync pass 8 at 1%. This
emphasizes the fact that not only we spend less time on metadata
but we also iterate less times to convergence in spa_sync() dirtying
objects.
[related graphs:
stock- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGStock.png
lsm- sdimitro.github.io/img/linux-lsm/NumberOfPassesPerTXGLSM.png]
Finally, the improvement in IOPs that the userland gains from the
change is approximately 40%. There is a consistent win in IOPS as
you can see from the graphs below but the absolute amount of
improvement that the log spacemap gives varies within each minute
interval.
sdimitro.github.io/img/linux-lsm/StockVsLog3Days.png
sdimitro.github.io/img/linux-lsm/StockVsLog10Hours.png
= Porting to Other Platforms
For people that want to port this commit to other platforms below
is a list of ZoL commits that this patch depends on:
Make zdb results for checkpoint tests consistent
db587941c5ff6dea01932bb78f70db63cf7f38ba
Update vdev_is_spacemap_addressable() for new spacemap encoding
419ba5914552c6185afbe1dd17b3ed4b0d526547
Simplify spa_sync by breaking it up to smaller functions
8dc2197b7b1e4d7ebc1420ea30e51c6541f1d834
Factor metaslab_load_wait() in metaslab_load()
b194fab0fb6caad18711abccaff3c69ad8b3f6d3
Rename range_tree_verify to range_tree_verify_not_present
df72b8bebe0ebac0b20e0750984bad182cb6564a
Change target size of metaslabs from 256GB to 16GB
c853f382db731e15a87512f4ef1101d14d778a55
zdb -L should skip leak detection altogether
21e7cf5da89f55ce98ec1115726b150e19eefe89
vs_alloc can underflow in L2ARC vdevs
7558997d2f808368867ca7e5234e5793446e8f3f
Simplify log vdev removal code
6c926f426a26ffb6d7d8e563e33fc176164175cb
Get rid of space_map_update() for ms_synced_length
425d3237ee88abc53d8522a7139c926d278b4b7f
Introduce auxiliary metaslab histograms
928e8ad47d3478a3d5d01f0dd6ae74a9371af65e
Error path in metaslab_load_impl() forgets to drop ms_sync_lock
8eef997679ba54547f7d361553d21b3291f41ae7
= References
Background, Motivation, and Internals of the Feature
- OpenZFS 2017 Presentation:
youtu.be/jj2IxRkl5bQ
- Slides:
slideshare.net/SerapheimNikolaosDim/zfs-log-spacemaps-project
Flushing Algorithm Internals & Performance Results
(Illumos Specific)
- Blogpost:
sdimitro.github.io/post/zfs-lsm-flushing/
- OpenZFS 2018 Presentation:
youtu.be/x6D2dHRjkxw
- Slides:
slideshare.net/SerapheimNikolaosDim/zfs-log-spacemap-flushing-algorithm
Upstream Delphix Issues:
DLPX-51539, DLPX-59659, DLPX-57783, DLPX-61438, DLPX-41227, DLPX-59320
DLPX-63385
Reviewed-by: Sean Eric Fagan <sef@ixsystems.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: George Wilson <gwilson@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Closes #8442
2019-07-16 17:11:49 +00:00
|
|
|
Display the offset, spacemap, free space of each metaslab, all the log
|
|
|
|
|
spacemaps and their obsolete entry statistics.
|
2017-04-13 16:40:56 +00:00
|
|
|
.It Fl mm
|
|
|
|
|
Also display information about the on-disk free space histogram associated with
|
|
|
|
|
each metaslab.
|
|
|
|
|
.It Fl mmm
|
|
|
|
|
Display the maximum contiguous free space, the in-core free space histogram, and
|
|
|
|
|
the percentage of free space in each space map.
|
|
|
|
|
.It Fl mmmm
|
|
|
|
|
Display every spacemap record.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl M , -metaslab-groups
|
2021-11-30 18:26:45 +00:00
|
|
|
Display all "normal" vdev metaslab group information - per-vdev metaslab count, fragmentation,
|
|
|
|
|
and free space histogram, as well as overall pool fragmentation and histogram.
|
2017-04-13 16:40:56 +00:00
|
|
|
.It Fl MM
|
2021-11-30 18:26:45 +00:00
|
|
|
"Special" vdevs are added to -M's normal output.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl O , -object-lookups Ns = Ns Ar dataset path
|
2017-04-13 16:40:56 +00:00
|
|
|
Look up the specified
|
|
|
|
|
.Ar path
|
|
|
|
|
inside of the
|
|
|
|
|
.Ar dataset
|
|
|
|
|
and display its metadata and indirect blocks.
|
|
|
|
|
Specified
|
|
|
|
|
.Ar path
|
|
|
|
|
must be relative to the root of
|
|
|
|
|
.Ar dataset .
|
|
|
|
|
This option can be combined with
|
|
|
|
|
.Fl v
|
|
|
|
|
for increasing verbosity.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl r , -copy-object Ns = Ns Ar dataset path destination
|
2021-01-28 05:36:01 +00:00
|
|
|
Copy the specified
|
|
|
|
|
.Ar path
|
|
|
|
|
inside of the
|
|
|
|
|
.Ar dataset
|
|
|
|
|
to the specified destination.
|
|
|
|
|
Specified
|
|
|
|
|
.Ar path
|
|
|
|
|
must be relative to the root of
|
|
|
|
|
.Ar dataset .
|
|
|
|
|
This option can be combined with
|
|
|
|
|
.Fl v
|
|
|
|
|
for increasing verbosity.
|
2017-08-24 17:30:42 +00:00
|
|
|
.It Xo
|
2022-01-06 18:54:32 +00:00
|
|
|
.Fl R , -read-block Ns = Ns Ar poolname vdev : Ns Ar offset : Ns Oo Ar lsize Ns / Oc Ns Ar psize Ns Op : Ns Ar flags
|
2017-08-24 17:30:42 +00:00
|
|
|
.Xc
|
2017-04-13 16:40:56 +00:00
|
|
|
Read and display a block from the specified device.
|
|
|
|
|
By default the block is displayed as a hex dump, but see the description of the
|
|
|
|
|
.Sy r
|
|
|
|
|
flag, below.
|
|
|
|
|
.Pp
|
|
|
|
|
The block is specified in terms of a colon-separated tuple
|
|
|
|
|
.Ar vdev
|
|
|
|
|
.Pq an integer vdev identifier
|
|
|
|
|
.Ar offset
|
|
|
|
|
.Pq the offset within the vdev
|
|
|
|
|
.Ar size
|
2019-12-10 23:51:58 +00:00
|
|
|
.Pq the physical size, or logical size / physical size
|
|
|
|
|
of the block to read and, optionally,
|
2017-04-13 16:40:56 +00:00
|
|
|
.Ar flags
|
|
|
|
|
.Pq a set of flags, described below .
|
|
|
|
|
.Pp
|
|
|
|
|
.Bl -tag -compact -width "b offset"
|
|
|
|
|
.It Sy b Ar offset
|
2020-02-10 22:00:05 +00:00
|
|
|
Print block pointer at hex offset
|
2019-11-27 18:08:18 +00:00
|
|
|
.It Sy c
|
|
|
|
|
Calculate and display checksums
|
2017-04-13 16:40:56 +00:00
|
|
|
.It Sy d
|
2021-05-27 00:46:40 +00:00
|
|
|
Decompress the block.
|
|
|
|
|
Set environment variable
|
2019-08-30 16:41:35 +00:00
|
|
|
.Nm ZDB_NO_ZLE
|
2018-02-02 00:19:36 +00:00
|
|
|
to skip zle when guessing.
|
2017-04-13 16:40:56 +00:00
|
|
|
.It Sy e
|
2012-02-04 05:44:53 +00:00
|
|
|
Byte swap the block
|
2017-04-13 16:40:56 +00:00
|
|
|
.It Sy g
|
2012-02-04 05:44:53 +00:00
|
|
|
Dump gang block header
|
2017-04-13 16:40:56 +00:00
|
|
|
.It Sy i
|
2012-02-04 05:44:53 +00:00
|
|
|
Dump indirect block
|
2017-04-13 16:40:56 +00:00
|
|
|
.It Sy r
|
2012-02-04 05:44:53 +00:00
|
|
|
Dump raw uninterpreted block data
|
2019-12-10 23:51:58 +00:00
|
|
|
.It Sy v
|
|
|
|
|
Verbose output for guessing compression algorithm
|
2017-04-13 16:40:56 +00:00
|
|
|
.El
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl s , -io-stats
|
2017-04-13 16:40:56 +00:00
|
|
|
Report statistics on
|
|
|
|
|
.Nm zdb
|
|
|
|
|
I/O.
|
|
|
|
|
Display operation counts, bandwidth, and error counts of I/O to the pool from
|
|
|
|
|
.Nm .
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl S , -simulate-dedup
|
2012-02-04 05:44:53 +00:00
|
|
|
Simulate the effects of deduplication, constructing a DDT and then display
|
2017-04-13 16:40:56 +00:00
|
|
|
that DDT as with
|
|
|
|
|
.Fl DD .
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl u , -uberblock
|
2012-02-04 05:44:53 +00:00
|
|
|
Display the current uberblock.
|
2017-04-13 16:40:56 +00:00
|
|
|
.El
|
|
|
|
|
.Pp
|
2012-02-04 05:44:53 +00:00
|
|
|
Other options:
|
2017-04-13 16:40:56 +00:00
|
|
|
.Bl -tag -width Ds
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl A , -ignore-assertions
|
2012-02-04 05:44:53 +00:00
|
|
|
Do not abort should any assertion fail.
|
2017-04-13 16:40:56 +00:00
|
|
|
.It Fl AA
|
2012-02-04 05:44:53 +00:00
|
|
|
Enable panic recovery, certain errors which would otherwise be fatal are
|
|
|
|
|
demoted to warnings.
|
2017-04-13 16:40:56 +00:00
|
|
|
.It Fl AAA
|
2012-02-04 05:44:53 +00:00
|
|
|
Do not abort if asserts fail and also enable panic recovery.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl e , -exported Ns = Ns Oo Fl p Ar path Oc Ns …
|
2017-04-13 16:40:56 +00:00
|
|
|
Operate on an exported pool, not present in
|
|
|
|
|
.Pa /etc/zfs/zpool.cache .
|
|
|
|
|
The
|
|
|
|
|
.Fl p
|
|
|
|
|
flag specifies the path under which devices are to be searched.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl x , -dump-blocks Ns = Ns Ar dumpdir
|
2016-01-01 13:42:58 +00:00
|
|
|
All blocks accessed will be copied to files in the specified directory.
|
|
|
|
|
The blocks will be placed in sparse files whose name is the same as
|
2017-04-13 16:40:56 +00:00
|
|
|
that of the file or device read.
|
|
|
|
|
.Nm
|
|
|
|
|
can be then run on the generated files.
|
|
|
|
|
Note that the
|
|
|
|
|
.Fl bbc
|
|
|
|
|
flags are sufficient to access
|
|
|
|
|
.Pq and thus copy
|
2016-01-01 13:42:58 +00:00
|
|
|
all metadata on the pool.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl F , -automatic-rewind
|
2012-02-04 05:44:53 +00:00
|
|
|
Attempt to make an unreadable pool readable by trying progressively older
|
|
|
|
|
transactions.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl G , -dump-debug-msg
|
2017-04-13 16:40:56 +00:00
|
|
|
Dump the contents of the zfs_dbgmsg buffer before exiting
|
|
|
|
|
.Nm .
|
|
|
|
|
zfs_dbgmsg is a buffer used by ZFS to dump advanced debug information.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl I , -inflight Ns = Ns Ar inflight I/Os
|
2017-04-13 16:40:56 +00:00
|
|
|
Limit the number of outstanding checksum I/Os to the specified value.
|
|
|
|
|
The default value is 200.
|
|
|
|
|
This option affects the performance of the
|
|
|
|
|
.Fl c
|
2013-05-02 23:36:32 +00:00
|
|
|
option.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl o , -option Ns = Ns Ar var Ns = Ns Ar value …
|
2017-04-13 16:40:56 +00:00
|
|
|
Set the given global libzpool variable to the provided value.
|
|
|
|
|
The value must be an unsigned 32-bit integer.
|
|
|
|
|
Currently only little-endian systems are supported to avoid accidentally setting
|
|
|
|
|
the high 32 bits of 64-bit variables.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl P , -parseable
|
2021-05-27 00:46:40 +00:00
|
|
|
Print numbers in an unscaled form more amenable to parsing, e.g.\&
|
|
|
|
|
.Sy 1000000
|
|
|
|
|
rather than
|
|
|
|
|
.Sy 1M .
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl t , -txg Ns = Ns Ar transaction
|
2017-04-13 16:40:56 +00:00
|
|
|
Specify the highest transaction to use when searching for uberblocks.
|
|
|
|
|
See also the
|
|
|
|
|
.Fl u
|
|
|
|
|
and
|
|
|
|
|
.Fl l
|
|
|
|
|
options for a means to see the available uberblocks and their associated
|
|
|
|
|
transaction numbers.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl U , -cachefile Ns = Ns Ar cachefile
|
2017-04-13 16:40:56 +00:00
|
|
|
Use a cache file other than
|
|
|
|
|
.Pa /etc/zfs/zpool.cache .
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl v , -verbose
|
2017-04-13 16:40:56 +00:00
|
|
|
Enable verbosity.
|
|
|
|
|
Specify multiple times for increased verbosity.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl V , -verbatim
|
2017-04-13 21:28:46 +00:00
|
|
|
Attempt verbatim import.
|
|
|
|
|
This mimics the behavior of the kernel when loading a pool from a cachefile.
|
|
|
|
|
Only usable with
|
|
|
|
|
.Fl e .
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl X , -extreme-rewind
|
2017-04-13 16:40:56 +00:00
|
|
|
Attempt
|
|
|
|
|
.Qq extreme
|
|
|
|
|
transaction rewind, that is attempt the same recovery as
|
|
|
|
|
.Fl F
|
|
|
|
|
but read transactions otherwise deemed too old.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl Y , -all-reconstruction
|
2019-01-16 22:10:02 +00:00
|
|
|
Attempt all possible combinations when reconstructing indirect split blocks.
|
|
|
|
|
This flag disables the individual I/O deadman timer in order to allow as
|
|
|
|
|
much time as required for the attempted reconstruction.
|
2022-01-06 18:54:32 +00:00
|
|
|
.It Fl y , -livelist
|
Extend zdb to print inconsistencies in livelists and metaslabs
Livelists and spacemaps are data structures that are logs of allocations
and frees. Livelists entries are block pointers (blkptr_t). Spacemaps
entries are ranges of numbers, most often used as to track
allocated/freed regions of metaslabs/vdevs.
These data structures can become self-inconsistent, for example if a
block or range can be "double allocated" (two allocation records without
an intervening free) or "double freed" (two free records without an
intervening allocation).
ZDB (as well as zfs running in the kernel) can detect these
inconsistencies when loading livelists and metaslab. However, it
generally halts processing when the error is detected.
When analyzing an on-disk problem, we often want to know the entire set
of inconsistencies, which is not possible with the current behavior.
This commit adds a new flag, `zdb -y`, which analyzes the livelist and
metaslab data structures and displays all of their inconsistencies.
Note that this is different from the leak detection performed by
`zdb -b`, which checks for inconsistencies between the spacemaps and the
tree of block pointers, but assumes the spacemaps are self-consistent.
The specific checks added are:
Verify livelists by iterating through each sublivelists and:
- report leftover FREEs
- report double ALLOCs and double FREEs
- record leftover ALLOCs together with their TXG [see Cross Check]
Verify spacemaps by iterating over each metaslab and:
- iterate over spacemap and then the metaslab's entries in the
spacemap log, then report any double FREEs and double ALLOCs
Verify that livelists are consistenet with spacemaps. The space
referenced by livelists (after using the FREE's to cancel out
corresponding ALLOCs) should be allocated, according to the spacemaps.
Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: Sara Hartse <sara.hartse@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
External-issue: DLPX-66031
Closes #10515
2020-07-15 00:51:05 +00:00
|
|
|
Perform validation for livelists that are being deleted.
|
|
|
|
|
Scans through the livelist and metaslabs, checking for duplicate entries
|
|
|
|
|
and compares the two, checking for potential double frees.
|
|
|
|
|
If it encounters issues, warnings will be printed, but the command will not
|
|
|
|
|
necessarily fail.
|
2017-04-13 16:40:56 +00:00
|
|
|
.El
|
|
|
|
|
.Pp
|
2012-02-04 05:44:53 +00:00
|
|
|
Specifying a display option more than once enables verbosity for only that
|
|
|
|
|
option, with more occurrences enabling more verbosity.
|
2017-04-13 16:40:56 +00:00
|
|
|
.Pp
|
2012-02-04 05:44:53 +00:00
|
|
|
If no options are specified, all information about the named pool will be
|
|
|
|
|
displayed at default verbosity.
|
2021-05-27 00:46:40 +00:00
|
|
|
.
|
2017-04-13 16:40:56 +00:00
|
|
|
.Sh EXAMPLES
|
|
|
|
|
.Bl -tag -width Ds
|
|
|
|
|
.It Xo
|
2021-05-27 00:46:40 +00:00
|
|
|
.Sy Example 1 :
|
2017-04-13 16:40:56 +00:00
|
|
|
Display the configuration of imported pool
|
2021-05-27 00:46:40 +00:00
|
|
|
.Ar rpool
|
2017-04-13 16:40:56 +00:00
|
|
|
.Xc
|
|
|
|
|
.Bd -literal
|
2021-05-27 00:46:40 +00:00
|
|
|
.No # Nm zdb Fl C Ar rpool
|
2012-02-04 05:44:53 +00:00
|
|
|
MOS Configuration:
|
|
|
|
|
version: 28
|
|
|
|
|
name: 'rpool'
|
2021-05-27 00:46:40 +00:00
|
|
|
…
|
2017-04-13 16:40:56 +00:00
|
|
|
.Ed
|
|
|
|
|
.It Xo
|
2021-05-27 00:46:40 +00:00
|
|
|
.Sy Example 2 :
|
2017-04-13 16:40:56 +00:00
|
|
|
Display basic dataset information about
|
2021-05-27 00:46:40 +00:00
|
|
|
.Ar rpool
|
2017-04-13 16:40:56 +00:00
|
|
|
.Xc
|
|
|
|
|
.Bd -literal
|
2021-05-27 00:46:40 +00:00
|
|
|
.No # Nm zdb Fl d Ar rpool
|
2012-02-04 05:44:53 +00:00
|
|
|
Dataset mos [META], ID 0, cr_txg 4, 26.9M, 1051 objects
|
|
|
|
|
Dataset rpool/swap [ZVOL], ID 59, cr_txg 356, 486M, 2 objects
|
2021-05-27 00:46:40 +00:00
|
|
|
…
|
2017-04-13 16:40:56 +00:00
|
|
|
.Ed
|
|
|
|
|
.It Xo
|
2021-05-27 00:46:40 +00:00
|
|
|
.Sy Example 3 :
|
2017-04-13 16:40:56 +00:00
|
|
|
Display basic information about object 0 in
|
2021-05-27 00:46:40 +00:00
|
|
|
.Ar rpool/export/home
|
2017-04-13 16:40:56 +00:00
|
|
|
.Xc
|
|
|
|
|
.Bd -literal
|
2021-05-27 00:46:40 +00:00
|
|
|
.No # Nm zdb Fl d Ar rpool/export/home 0
|
2012-02-04 05:44:53 +00:00
|
|
|
Dataset rpool/export/home [ZPL], ID 137, cr_txg 1546, 32K, 8 objects
|
|
|
|
|
|
|
|
|
|
Object lvl iblk dblk dsize lsize %full type
|
|
|
|
|
0 7 16K 16K 15.0K 16K 25.00 DMU dnode
|
2017-04-13 16:40:56 +00:00
|
|
|
.Ed
|
|
|
|
|
.It Xo
|
2021-05-27 00:46:40 +00:00
|
|
|
.Sy Example 4 :
|
2017-04-13 16:40:56 +00:00
|
|
|
Display the predicted effect of enabling deduplication on
|
2021-05-27 00:46:40 +00:00
|
|
|
.Ar rpool
|
2017-04-13 16:40:56 +00:00
|
|
|
.Xc
|
|
|
|
|
.Bd -literal
|
2021-05-27 00:46:40 +00:00
|
|
|
.No # Nm zdb Fl S Ar rpool
|
2012-02-04 05:44:53 +00:00
|
|
|
Simulated DDT histogram:
|
|
|
|
|
|
2015-12-17 01:45:15 +00:00
|
|
|
bucket allocated referenced
|
2012-02-04 05:44:53 +00:00
|
|
|
______ ______________________________ ______________________________
|
|
|
|
|
refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE
|
|
|
|
|
------ ------ ----- ----- ----- ------ ----- ----- -----
|
|
|
|
|
1 694K 27.1G 15.0G 15.0G 694K 27.1G 15.0G 15.0G
|
|
|
|
|
2 35.0K 1.33G 699M 699M 74.7K 2.79G 1.45G 1.45G
|
2021-05-27 00:46:40 +00:00
|
|
|
…
|
2012-02-04 05:44:53 +00:00
|
|
|
dedup = 1.11, compress = 1.80, copies = 1.00, dedup * compress / copies = 2.00
|
2017-04-13 16:40:56 +00:00
|
|
|
.Ed
|
|
|
|
|
.El
|
2021-05-27 00:46:40 +00:00
|
|
|
.
|
2017-04-13 16:40:56 +00:00
|
|
|
.Sh SEE ALSO
|
|
|
|
|
.Xr zfs 8 ,
|
|
|
|
|
.Xr zpool 8
|
