2017-04-06 00:18:22 +00:00
|
|
|
#
|
2019-12-19 19:53:55 +00:00
|
|
|
# Copyright (c) 2016, 2019 by Delphix. All rights reserved.
|
2017-04-06 00:18:22 +00:00
|
|
|
# These variables are used by zfs-tests.sh to constrain which utilities
|
|
|
|
|
# may be used by the suite. The suite will create a directory which is
|
|
|
|
|
# the only element of $PATH and create symlinks from that dir to the
|
|
|
|
|
# binaries listed below.
|
|
|
|
|
#
|
|
|
|
|
# Please keep the contents of each variable sorted for ease of reading
|
|
|
|
|
# and maintenance.
|
|
|
|
|
#
|
2019-12-18 20:29:43 +00:00
|
|
|
export SYSTEM_FILES_COMMON='arp
|
2017-04-06 00:18:22 +00:00
|
|
|
awk
|
2018-01-29 22:33:57 +00:00
|
|
|
base64
|
2017-04-06 00:18:22 +00:00
|
|
|
basename
|
|
|
|
|
bc
|
|
|
|
|
bunzip2
|
|
|
|
|
bzcat
|
|
|
|
|
cat
|
|
|
|
|
chgrp
|
|
|
|
|
chmod
|
|
|
|
|
chown
|
2020-02-16 20:49:49 +00:00
|
|
|
cksum
|
2017-04-06 00:18:22 +00:00
|
|
|
cmp
|
|
|
|
|
cp
|
|
|
|
|
cpio
|
|
|
|
|
cut
|
|
|
|
|
date
|
|
|
|
|
dd
|
|
|
|
|
df
|
|
|
|
|
diff
|
|
|
|
|
dirname
|
2017-04-18 16:44:17 +00:00
|
|
|
dmesg
|
2017-04-06 00:18:22 +00:00
|
|
|
du
|
|
|
|
|
echo
|
|
|
|
|
egrep
|
|
|
|
|
expr
|
|
|
|
|
false
|
|
|
|
|
file
|
|
|
|
|
find
|
|
|
|
|
fio
|
|
|
|
|
getconf
|
|
|
|
|
getent
|
|
|
|
|
getfacl
|
|
|
|
|
grep
|
|
|
|
|
gunzip
|
|
|
|
|
gzip
|
|
|
|
|
head
|
|
|
|
|
hostname
|
|
|
|
|
id
|
|
|
|
|
iostat
|
|
|
|
|
kill
|
|
|
|
|
ksh
|
|
|
|
|
ln
|
|
|
|
|
logname
|
|
|
|
|
ls
|
|
|
|
|
mkdir
|
|
|
|
|
mknod
|
|
|
|
|
mktemp
|
|
|
|
|
mount
|
|
|
|
|
mv
|
|
|
|
|
net
|
OpenZFS 9102 - zfs should be able to initialize storage devices
PROBLEM
========
The first access to a block incurs a performance penalty on some platforms
(e.g. AWS's EBS, VMware VMDKs). Therefore we recommend that volumes are
"thick provisioned", where supported by the platform (VMware). This can
create a large delay in getting a new virtual machines up and running (or
adding storage to an existing Engine). If the thick provision step is
omitted, write performance will be suboptimal until all blocks on the LUN
have been written.
SOLUTION
=========
This feature introduces a way to 'initialize' the disks at install or in the
background to make sure we don't incur this first read penalty.
When an entire LUN is added to ZFS, we make all space available immediately,
and allow ZFS to find unallocated space and zero it out. This works with
concurrent writes to arbitrary offsets, ensuring that we don't zero out
something that has been (or is in the middle of being) written. This scheme
can also be applied to existing pools (affecting only free regions on the
vdev). Detailed design:
- new subcommand:zpool initialize [-cs] <pool> [<vdev> ...]
- start, suspend, or cancel initialization
- Creates new open-context thread for each vdev
- Thread iterates through all metaslabs in this vdev
- Each metaslab:
- select a metaslab
- load the metaslab
- mark the metaslab as being zeroed
- walk all free ranges within that metaslab and translate
them to ranges on the leaf vdev
- issue a "zeroing" I/O on the leaf vdev that corresponds to
a free range on the metaslab we're working on
- continue until all free ranges for this metaslab have been
"zeroed"
- reset/unmark the metaslab being zeroed
- if more metaslabs exist, then repeat above tasks.
- if no more metaslabs, then we're done.
- progress for the initialization is stored on-disk in the vdev’s
leaf zap object. The following information is stored:
- the last offset that has been initialized
- the state of the initialization process (i.e. active,
suspended, or canceled)
- the start time for the initialization
- progress is reported via the zpool status command and shows
information for each of the vdevs that are initializing
Porting notes:
- Added zfs_initialize_value module parameter to set the pattern
written by "zpool initialize".
- Added zfs_vdev_{initializing,removal}_{min,max}_active module options.
Authored by: George Wilson <george.wilson@delphix.com>
Reviewed by: John Wren Kennedy <john.kennedy@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: loli10K <ezomori.nozomu@gmail.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Richard Lowe <richlowe@richlowe.net>
Signed-off-by: Tim Chase <tim@chase2k.com>
Ported-by: Tim Chase <tim@chase2k.com>
OpenZFS-issue: https://www.illumos.org/issues/9102
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/c3963210eb
Closes #8230
2018-12-19 14:54:59 +00:00
|
|
|
od
|
2017-04-06 00:18:22 +00:00
|
|
|
openssl
|
2020-06-25 01:45:44 +00:00
|
|
|
pamtester
|
2017-04-06 00:18:22 +00:00
|
|
|
pax
|
|
|
|
|
pgrep
|
|
|
|
|
ping
|
|
|
|
|
pkill
|
2018-02-23 19:38:05 +00:00
|
|
|
printenv
|
2017-04-06 00:18:22 +00:00
|
|
|
printf
|
|
|
|
|
ps
|
|
|
|
|
pwd
|
|
|
|
|
python
|
2019-09-16 17:44:51 +00:00
|
|
|
python2
|
2018-02-28 16:52:34 +00:00
|
|
|
python3
|
2017-04-06 00:18:22 +00:00
|
|
|
quotaon
|
|
|
|
|
readlink
|
|
|
|
|
rm
|
|
|
|
|
rmdir
|
2018-02-11 23:11:59 +00:00
|
|
|
scp
|
2019-12-20 00:26:07 +00:00
|
|
|
script
|
2017-04-06 00:18:22 +00:00
|
|
|
sed
|
|
|
|
|
seq
|
|
|
|
|
setfacl
|
|
|
|
|
sh
|
|
|
|
|
sleep
|
|
|
|
|
sort
|
2018-02-11 23:11:59 +00:00
|
|
|
ssh
|
2017-04-06 00:18:22 +00:00
|
|
|
stat
|
|
|
|
|
strings
|
|
|
|
|
su
|
|
|
|
|
sudo
|
|
|
|
|
sum
|
Enable additional test cases
Enable additional test cases, in most cases this required a few
minor modifications to the test scripts. In a few cases a real
bug was uncovered and fixed. And in a handful of cases where pools
are layered on pools the test case will be skipped until this is
supported. Details below for each test case.
* zpool_add_004_pos - Skip test on Linux until adding zvols to pools
is fully supported and deadlock free.
* zpool_add_005_pos.ksh - Skip dumpadm portion of the test which isn't
relevant for Linux. The find_vfstab_dev, find_mnttab_dev, and
save_dump_dev functions were updated accordingly for Linux. Add
O_EXCL to the in-use check to prevent the -f (force) option from
working for mounted filesystems and improve the resulting error.
* zpool_add_006_pos - Update test case such that it doesn't depend
on nested pools. Switch to truncate from mkfile to reduce space
requirements and speed up the test case.
* zpool_clear_001_pos - Speed up test case by filling filesystem to
25% capacity.
* zpool_create_002_pos, zpool_create_004_pos - Use sparse files for
file vdevs in order to avoid increasing the partition size.
* zpool_create_006_pos - 6ba1ce9 allows raidz+mirror configs with
similar redundancy. Updating the valid_args and forced_args cases.
* zpool_create_008_pos - Disable overlapping partition portion.
* zpool_create_011_neg - Fix to correctly create the extra partition.
Modified zpool_vdev.c to use fstat64_blk() wrapper which includes
the st_size even for block devices.
* zpool_create_012_neg - Updated to properly find swap devices.
* zpool_create_014_neg, zpool_create_015_neg - Updated to use
swap_setup() and swap_cleanup() wrappers which do the right thing
on Linux and Illumos. Removed '-n' option which succeeds under
Linux due to differences in the in-use checks.
* zpool_create_016_pos.ksh - Skipped test case isn't useful.
* zpool_create_020_pos - Added missing / to cleanup() function.
Remove cache file prior to test to ensure a clean environment
and avoid false positives.
* zpool_destroy_001_pos - Removed test case which creates a pool on
a zvol. This is more likely to deadlock under Linux and has never
been completely supported on any platform.
* zpool_destroy_002_pos - 'zpool destroy -f' is unsupported on Linux.
Mount point must not be busy in order to unmount them.
* zfs_destroy_001_pos - Handle EBUSY error which can occur with
volumes when racing with udev.
* zpool_expand_001_pos, zpool_expand_003_neg - Skip test on Linux
until adding zvols to pools is fully supported and deadlock free.
The test could be modified to use loop-back devices but it would
be preferable to use the test case as is for improved coverage.
* zpool_export_004_pos - Updated test case to such that it doesn't
depend on nested pools. Normal file vdev under /var/tmp are fine.
* zpool_import_all_001_pos - Updated to skip partition 1, which is
known as slice 2, on Illumos. This prevents overwriting the
default TESTPOOL which was causing the failure.
* zpool_import_002_pos, zpool_import_012_pos - No changes needed.
* zpool_remove_003_pos - No changes needed
* zpool_upgrade_002_pos, zpool_upgrade_004_pos - Root cause addressed
by upstream OpenZFS commit 3b7f360.
* zpool_upgrade_007_pos - Disabled in test case due to known failure.
Opened issue https://github.com/zfsonlinux/zfs/issues/6112
* zvol_misc_002_pos - Updated to to use ext2.
* zvol_misc_001_neg, zvol_misc_003_neg, zvol_misc_004_pos,
zvol_misc_005_neg, zvol_misc_006_pos - Moved to skip list, these
test case could be updated to use Linux's crash dump facility.
* zvol_swap_* - Updated to use swap_setup/swap_cleanup helpers.
File creation switched from /tmp to /var/tmp. Enabled minimal
useful tests for Linux, skip test cases which aren't applicable.
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #3484
Issue #5634
Issue #2437
Issue #5202
Issue #4034
Closes #6095
2017-05-11 21:27:57 +00:00
|
|
|
swapoff
|
2017-04-06 00:18:22 +00:00
|
|
|
swapon
|
|
|
|
|
sync
|
|
|
|
|
tail
|
|
|
|
|
tar
|
2018-02-08 16:16:23 +00:00
|
|
|
tee
|
2017-04-06 00:18:22 +00:00
|
|
|
timeout
|
|
|
|
|
touch
|
|
|
|
|
tr
|
|
|
|
|
true
|
|
|
|
|
truncate
|
|
|
|
|
umask
|
|
|
|
|
umount
|
|
|
|
|
uname
|
|
|
|
|
uuidgen
|
|
|
|
|
vmstat
|
|
|
|
|
wait
|
|
|
|
|
wc
|
|
|
|
|
which
|
|
|
|
|
xargs'
|
|
|
|
|
|
2019-12-18 20:29:43 +00:00
|
|
|
export SYSTEM_FILES_FREEBSD='chflags
|
|
|
|
|
compress
|
2020-02-27 17:38:34 +00:00
|
|
|
diskinfo
|
2019-12-18 20:29:43 +00:00
|
|
|
dumpon
|
2020-02-12 21:04:51 +00:00
|
|
|
env
|
2019-12-18 20:29:43 +00:00
|
|
|
fsck
|
2020-01-10 21:24:59 +00:00
|
|
|
getextattr
|
2019-12-18 20:29:43 +00:00
|
|
|
gpart
|
2020-08-01 15:44:54 +00:00
|
|
|
jail
|
|
|
|
|
jexec
|
|
|
|
|
jls
|
2020-01-10 21:24:59 +00:00
|
|
|
lsextattr
|
2019-12-18 20:29:43 +00:00
|
|
|
md5
|
|
|
|
|
mdconfig
|
|
|
|
|
mkfifo
|
|
|
|
|
newfs
|
|
|
|
|
pw
|
2020-01-10 21:24:59 +00:00
|
|
|
rmextattr
|
|
|
|
|
setextattr
|
2019-12-18 20:29:43 +00:00
|
|
|
sha256
|
2020-02-21 23:59:20 +00:00
|
|
|
showmount
|
2019-12-18 20:29:43 +00:00
|
|
|
swapctl
|
|
|
|
|
sysctl
|
|
|
|
|
uncompress'
|
|
|
|
|
|
|
|
|
|
export SYSTEM_FILES_LINUX='attr
|
bash scripts: use /usr/bin/env for bash shebangs
Not all systems / distros have a `/bin/bash`, and these scripts are
more difficult to run at development time.
For example, my system is NixOS which doesn't have a /bin/bash. This
is not a problem for NixOS building ZFS as a package: the build
environment automatically replaces these shebangs with corrected
paths.
The problem is much more annoying at development time: either the
scripts don't run, or I correct them for my local machine and deal with
a perpetually dirty work tree.
Before committing this patch I confirmed there are existing scripts
which use `/usr/bin/env` to locate bash, so I am thinking this is a
safe transformation.
There are a handful of other shebangs in this repository which don't
work on my system. This patch is useful on its own specifically for
`commitcheck.sh`, otherwise I can't validate my commits before
submission.
Here are the remaining shebangs which NixOS systems won't have:
1274 #!/bin/ksh -p
91 #!/bin/ksh
89 #! /bin/ksh -p
2 #!/bin/sed -f
1 #!/usr/bin/perl -w
1 #!/usr/bin/ksh
1 #!/bin/nawk -f
plus this which will create an invalid shebang in
`tests/zfs-tests/tests/functional/mv_files/mv_files_common.kshlib`:
echo "#!/bin/ksh" > $TEST_BASE_DIR/exitsZero.ksh
I chose to leave those alone for now, and gauge the interest in this
much smaller patch first.
The fixes for these are easy enough by simply using `/usr/bin/env ksh`:
91 #!/bin/ksh
1 #!/usr/bin/ksh
The fix for the other set is much trickier. Quoting the GNU coreutils
manual:
Most operating systems (e.g. GNU/Linux, BSDs) treat all text after
the first space as a single argument. When using env in a script it
is thus not possible to specify multiple arguments.
and not all `env`'s support arguments.
Mine (GNU Coreutils 8.31) does, though this feature is new since
April 2018, GNU Coreutils 8.30:
https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=668306ed86c8c79b0af0db8b9c882654ebb66db2
and worse, requires the -S argument:
-S, --split-string=S process and split S into separate arguments;
used to pass multiple arguments on shebang
lines
Example:
$ seq 1 2 | $(nix-build '<nixpkgs>' -A coreutils)/bin/env "sort -nr"
/nix/[...]-coreutils-8.31/bin/env: ‘sort -nr’: No such file or directory
/nix/[...]-coreutils-8.31/bin/env: use -[v]S to pass options in shebang lines
$ seq 1 2 | $(nix-build '<nixpkgs>' -A coreutils)/bin/env "-S sort -nr"
2
1
GNU Coreutils says FreeBSD's `env` does, though I wonder if FreeBSD's
would be unhappy with the `-S`:
https://www.gnu.org/software/coreutils/manual/html_node/env-invocation.html#env-invocation
BusyBox v1.30.1 does not, and does not have a `-S`-like option:
$ seq 1 2 | $(nix-build '<nixpkgs>' -A busybox)/bin/env "sort -nr"
env: can't execute 'sort -nr': No such file or directory
Toybox 0.8.1 also does not, and also does not have a `-S` option:
$ seq 1 2 | $(nix-build '<nixpkgs>' -A toybox)/bin/env "sort -nr"
env: exec sort -nr: No such file or directory
---
At any rate, if this patch merges and the remaining ~1,500 are updated,
the much larger patch should probably include a checkstyle-like test
asserting all new shebangs use `/usr/bin/env`. I also don't mind
dealing with NixOS weirdness if the project would prefer that.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Graham Christensen <graham@grahamc.com>
Closes #9893
2020-02-10 21:13:46 +00:00
|
|
|
bash
|
2019-12-18 20:29:43 +00:00
|
|
|
blkid
|
|
|
|
|
blockdev
|
|
|
|
|
chattr
|
2020-01-06 19:14:19 +00:00
|
|
|
dmidecode
|
2019-12-18 20:29:43 +00:00
|
|
|
exportfs
|
|
|
|
|
fallocate
|
|
|
|
|
fdisk
|
2020-02-18 19:23:41 +00:00
|
|
|
free
|
2019-12-18 20:29:43 +00:00
|
|
|
getfattr
|
|
|
|
|
groupadd
|
|
|
|
|
groupdel
|
|
|
|
|
groupmod
|
|
|
|
|
hostid
|
|
|
|
|
losetup
|
|
|
|
|
lsattr
|
|
|
|
|
lsblk
|
|
|
|
|
lscpu
|
|
|
|
|
lsmod
|
|
|
|
|
lsscsi
|
|
|
|
|
md5sum
|
|
|
|
|
mkswap
|
|
|
|
|
modprobe
|
|
|
|
|
mpstat
|
|
|
|
|
nproc
|
|
|
|
|
parted
|
|
|
|
|
perf
|
|
|
|
|
setenforce
|
|
|
|
|
setfattr
|
|
|
|
|
sha256sum
|
|
|
|
|
udevadm
|
|
|
|
|
useradd
|
|
|
|
|
userdel
|
|
|
|
|
usermod'
|
|
|
|
|
|
2017-04-06 00:18:22 +00:00
|
|
|
export ZFS_FILES='zdb
|
|
|
|
|
zfs
|
|
|
|
|
zhack
|
|
|
|
|
zinject
|
|
|
|
|
zpool
|
|
|
|
|
ztest
|
|
|
|
|
raidz_test
|
pyzfs: python3 support (build system)
Almost all of the Python code in the respository has been updated
to be compatibile with Python 2.6, Python 3.4, or newer. The only
exceptions are arc_summery3.py which requires Python 3, and pyzfs
which requires at least Python 2.7. This allows us to maintain a
single version of the code and support most default versions of
python. This change does the following:
* Sets the default shebang for all Python scripts to python3. If
only Python 2 is available, then at install time scripts which
are compatible with Python 2 will have their shebangs replaced
with /usr/bin/python. This is done for compatibility until
Python 2 goes end of life. Since only the installed versions
are changed this means Python 3 must be installed on the system
for test-runner when testing in-tree.
* Added --with-python=<2|3|3.4,etc> configure option which sets
the PYTHON environment variable to target a specific python
version. By default the newest installed version of Python
will be used or the preferred distribution version when
creating pacakges.
* Fixed --enable-pyzfs configure checks so they are run when
--enable-pyzfs=check and --enable-pyzfs=yes.
* Enabled pyzfs for Python 3.4 and newer, which is now supported.
* Renamed pyzfs package to python<VERSION>-pyzfs and updated to
install in the appropriate site location. For example, when
building with --with-python=3.4 a python34-pyzfs will be
created which installs in /usr/lib/python3.4/site-packages/.
* Renamed the following python scripts according to the Fedora
guidance for packaging utilities in /bin
- dbufstat.py -> dbufstat
- arcstat.py -> arcstat
- arc_summary.py -> arc_summary
- arc_summary3.py -> arc_summary3
* Updated python-cffi package name. On CentOS 6, CentOS 7, and
Amazon Linux it's called python-cffi, not python2-cffi. For
Python3 it's called python3-cffi or python3x-cffi.
* Install one version of arc_summary. Depending on the version
of Python available install either arc_summary2 or arc_summary3
as arc_summary. The user output is only slightly different.
Reviewed-by: John Ramsden <johnramsden@riseup.net>
Reviewed-by: Neal Gompa <ngompa@datto.com>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8096
2018-10-31 16:22:59 +00:00
|
|
|
arc_summary
|
|
|
|
|
arcstat
|
|
|
|
|
dbufstat
|
2017-04-11 21:56:54 +00:00
|
|
|
zed
|
2017-07-19 01:11:08 +00:00
|
|
|
zgenhostid
|
Add `zstream redup` command to convert deduplicated send streams
Deduplicated send and receive is deprecated. To ease migration to the
new dedup-send-less world, the commit adds a `zstream redup` utility to
convert deduplicated send streams to normal streams, so that they can
continue to be received indefinitely.
The new `zstream` command also replaces the functionality of
`zstreamdump`, by way of the `zstream dump` subcommand. The
`zstreamdump` command is replaced by a shell script which invokes
`zstream dump`.
The way that `zstream redup` works under the hood is that as we read the
send stream, we build up a hash table which maps from `<GUID, object,
offset> -> <file_offset>`.
Whenever we see a WRITE record, we add a new entry to the hash table,
which indicates where in the stream file to find the WRITE record for
this block. (The key is `drr_toguid, drr_object, drr_offset`.)
For entries other than WRITE_BYREF, we pass them through unchanged
(except for the running checksum, which is recalculated).
For WRITE_BYREF records, we change them to WRITE records. We find the
referenced WRITE record by looking in the hash table (for the record
with key `drr_refguid, drr_refobject, drr_refoffset`), and then reading
the record header and payload from the specified offset in the stream
file. This is why the stream can not be a pipe. The found WRITE record
replaces the WRITE_BYREF record, with its `drr_toguid`, `drr_object`,
and `drr_offset` fields changed to be the same as the WRITE_BYREF's
(i.e. we are writing the same logical block, but with the data supplied
by the previous WRITE record).
This algorithm requires memory proportional to the number of WRITE
records (same as `zfs send -D`), but the size per WRITE record is
relatively low (40 bytes, vs. 72 for `zfs send -D`). A 1TB send stream
with 8KB blocks (`recordsize=8k`) would use around 5GB of RAM to
"redup".
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #10124
Closes #10156
2020-04-10 17:39:55 +00:00
|
|
|
zstream
|
2020-05-20 17:05:33 +00:00
|
|
|
zstreamdump
|
|
|
|
|
zfs_ids_to_path'
|
2017-04-06 00:18:22 +00:00
|
|
|
|
2019-12-19 19:53:55 +00:00
|
|
|
export ZFSTEST_FILES='btree_test
|
|
|
|
|
chg_usr_exec
|
2017-04-06 00:18:22 +00:00
|
|
|
devname2devid
|
|
|
|
|
dir_rd_update
|
|
|
|
|
file_check
|
|
|
|
|
file_trunc
|
|
|
|
|
file_write
|
Implement Redacted Send/Receive
Redacted send/receive allows users to send subsets of their data to
a target system. One possible use case for this feature is to not
transmit sensitive information to a data warehousing, test/dev, or
analytics environment. Another is to save space by not replicating
unimportant data within a given dataset, for example in backup tools
like zrepl.
Redacted send/receive is a three-stage process. First, a clone (or
clones) is made of the snapshot to be sent to the target. In this
clone (or clones), all unnecessary or unwanted data is removed or
modified. This clone is then snapshotted to create the "redaction
snapshot" (or snapshots). Second, the new zfs redact command is used
to create a redaction bookmark. The redaction bookmark stores the
list of blocks in a snapshot that were modified by the redaction
snapshot(s). Finally, the redaction bookmark is passed as a parameter
to zfs send. When sending to the snapshot that was redacted, the
redaction bookmark is used to filter out blocks that contain sensitive
or unwanted information, and those blocks are not included in the send
stream. When sending from the redaction bookmark, the blocks it
contains are considered as candidate blocks in addition to those
blocks in the destination snapshot that were modified since the
creation_txg of the redaction bookmark. This step is necessary to
allow the target to rehydrate data in the case where some blocks are
accidentally or unnecessarily modified in the redaction snapshot.
The changes to bookmarks to enable fast space estimation involve
adding deadlists to bookmarks. There is also logic to manage the
life cycles of these deadlists.
The new size estimation process operates in cases where previously
an accurate estimate could not be provided. In those cases, a send
is performed where no data blocks are read, reducing the runtime
significantly and providing a byte-accurate size estimate.
Reviewed-by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Prashanth Sreenivasa <pks@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Chris Williamson <chris.williamson@delphix.com>
Reviewed-by: Pavel Zhakarov <pavel.zakharov@delphix.com>
Reviewed-by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #7958
2019-06-19 16:48:13 +00:00
|
|
|
get_diff
|
2017-04-06 00:18:22 +00:00
|
|
|
largest_file
|
Add basic zfs ioc input nvpair validation
We want newer versions of libzfs_core to run against an existing
zfs kernel module (i.e. a deferred reboot or module reload after
an update).
Programmatically document, via a zfs_ioc_key_t, the valid arguments
for the ioc commands that rely on nvpair input arguments (i.e. non
legacy commands from libzfs_core). Automatically verify the expected
pairs before dispatching a command.
This initial phase focuses on the non-legacy ioctls. A follow-on
change can address the legacy ioctl input from the zfs_cmd_t.
The zfs_ioc_key_t for zfs_keys_channel_program looks like:
static const zfs_ioc_key_t zfs_keys_channel_program[] = {
{"program", DATA_TYPE_STRING, 0},
{"arg", DATA_TYPE_UNKNOWN, 0},
{"sync", DATA_TYPE_BOOLEAN_VALUE, ZK_OPTIONAL},
{"instrlimit", DATA_TYPE_UINT64, ZK_OPTIONAL},
{"memlimit", DATA_TYPE_UINT64, ZK_OPTIONAL},
};
Introduce four input errors to identify specific input failures
(in addition to generic argument value errors like EINVAL, ERANGE,
EBADF, and E2BIG).
ZFS_ERR_IOC_CMD_UNAVAIL the ioctl number is not supported by kernel
ZFS_ERR_IOC_ARG_UNAVAIL an input argument is not supported by kernel
ZFS_ERR_IOC_ARG_REQUIRED a required input argument is missing
ZFS_ERR_IOC_ARG_BADTYPE an input argument has an invalid type
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Don Brady <don.brady@delphix.com>
Closes #7780
2018-09-02 19:14:01 +00:00
|
|
|
libzfs_input_check
|
2017-04-06 00:18:22 +00:00
|
|
|
mkbusy
|
|
|
|
|
mkfile
|
|
|
|
|
mkfiles
|
|
|
|
|
mktree
|
|
|
|
|
mmap_exec
|
2018-03-28 17:19:22 +00:00
|
|
|
mmap_libaio
|
2017-04-06 00:18:22 +00:00
|
|
|
mmapwrite
|
2018-02-08 16:16:23 +00:00
|
|
|
nvlist_to_lua
|
2017-04-06 00:18:22 +00:00
|
|
|
randfree_file
|
2016-12-16 22:11:29 +00:00
|
|
|
randwritecomp
|
2017-04-06 00:18:22 +00:00
|
|
|
readmmap
|
|
|
|
|
rename_dir
|
|
|
|
|
rm_lnkcnt_zero_file
|
|
|
|
|
threadsappend
|
2018-03-07 23:40:42 +00:00
|
|
|
user_ns_exec
|
Implement Redacted Send/Receive
Redacted send/receive allows users to send subsets of their data to
a target system. One possible use case for this feature is to not
transmit sensitive information to a data warehousing, test/dev, or
analytics environment. Another is to save space by not replicating
unimportant data within a given dataset, for example in backup tools
like zrepl.
Redacted send/receive is a three-stage process. First, a clone (or
clones) is made of the snapshot to be sent to the target. In this
clone (or clones), all unnecessary or unwanted data is removed or
modified. This clone is then snapshotted to create the "redaction
snapshot" (or snapshots). Second, the new zfs redact command is used
to create a redaction bookmark. The redaction bookmark stores the
list of blocks in a snapshot that were modified by the redaction
snapshot(s). Finally, the redaction bookmark is passed as a parameter
to zfs send. When sending to the snapshot that was redacted, the
redaction bookmark is used to filter out blocks that contain sensitive
or unwanted information, and those blocks are not included in the send
stream. When sending from the redaction bookmark, the blocks it
contains are considered as candidate blocks in addition to those
blocks in the destination snapshot that were modified since the
creation_txg of the redaction bookmark. This step is necessary to
allow the target to rehydrate data in the case where some blocks are
accidentally or unnecessarily modified in the redaction snapshot.
The changes to bookmarks to enable fast space estimation involve
adding deadlists to bookmarks. There is also logic to manage the
life cycles of these deadlists.
The new size estimation process operates in cases where previously
an accurate estimate could not be provided. In those cases, a send
is performed where no data blocks are read, reducing the runtime
significantly and providing a byte-accurate size estimate.
Reviewed-by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed-by: Matt Ahrens <mahrens@delphix.com>
Reviewed-by: Prashanth Sreenivasa <pks@delphix.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Chris Williamson <chris.williamson@delphix.com>
Reviewed-by: Pavel Zhakarov <pavel.zakharov@delphix.com>
Reviewed-by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #7958
2019-06-19 16:48:13 +00:00
|
|
|
xattrtest
|
|
|
|
|
stride_dd'
|
