Author: John Wren Kennedy <john.kennedy@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Don Brady <don.brady@intel.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: David Quigley <david.quigley@intel.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Ported-by: Don Brady <don.brady@intel.com>
OpenZFS-issue: https://www.illumos.org/issues/6950
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/dcbf3bd6
Delphix-commit: https://github.com/delphix/delphix-os/commit/978ed49Closes#4929
ZFS Test Suite Performance Regression Tests
This was pulled into OpenZFS via the compressed arc featureand was
separated out in zfsonlinux as a separate pull request from PR-4768.
It originally came in as QA-4903 in Delphix-OS from John Kennedy.
Expected Usage:
$ DISKS="sdb sdc sdd" zfs-tests.sh -r perf-regression.run
Porting Notes:
1. Added assertions in the setup script to make sure required tools
(fio, mpstat, ...) are present.
2. For the config.json generation in perf.shlib used arcstats and
other binaries instead of dtrace to query the values.
3. For the perf data collection:
- use "zpool iostat -lpvyL" instead of the io.d dtrace script
(currently not collecting zfs_read/write latency stats)
- mpstat and iostat take different arguments
- prefetch_io.sh is a placeholder that uses arcstats instead of
dtrace
4. Build machines require fio, mdadm and sysstat pakage (YMMV).
Future Work:
- Need a way to measure zfs_read and zfs_write latencies per pool.
- Need tools to takes two sets of output and display/graph the
differences
- Bring over additional regression tests from Delphix
Both libudev and libattr are recommended build requirements. As
such their development headers should lists in the rpm spec file
so those dependencies are pulled in when building rpm packages.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#4676
Add the ZFS Test Suite and test-runner framework from illumos.
This is a continuation of the work done by Turbo Fredriksson to
port the ZFS Test Suite to Linux. While this work was originally
conceived as a stand alone project integrating it directly with
the ZoL source tree has several advantages:
* Allows the ZFS Test Suite to be packaged in zfs-test package.
* Facilitates easy integration with the CI testing.
* Users can locally run the ZFS Test Suite to validate ZFS.
This testing should ONLY be done on a dedicated test system
because the ZFS Test Suite in its current form is destructive.
* Allows the ZFS Test Suite to be run directly in the ZoL source
tree enabled developers to iterate quickly during development.
* Developers can easily add/modify tests in the framework as
features are added or functionality is changed. The tests
will then always be in sync with the implementation.
Full documentation for how to run the ZFS Test Suite is available
in the tests/README.md file.
Warning: This test suite is designed to be run on a dedicated test
system. It will make modifications to the system including, but
not limited to, the following.
* Adding new users
* Adding new groups
* Modifying the following /proc files:
* /proc/sys/kernel/core_pattern
* /proc/sys/kernel/core_uses_pid
* Creating directories under /
Notes:
* Not all of the test cases are expected to pass and by default
these test cases are disabled. The failures are primarily due
to assumption made for illumos which are invalid under Linux.
* When updating these test cases it should be done in as generic
a way as possible so the patch can be submitted back upstream.
Most existing library functions have been updated to be Linux
aware, and the following functions and variables have been added.
* Functions:
* is_linux - Used to wrap a Linux specific section.
* block_device_wait - Waits for block devices to be added to /dev/.
* Variables: Linux Illumos
* ZVOL_DEVDIR "/dev/zvol" "/dev/zvol/dsk"
* ZVOL_RDEVDIR "/dev/zvol" "/dev/zvol/rdsk"
* DEV_DSKDIR "/dev" "/dev/dsk"
* DEV_RDSKDIR "/dev" "/dev/rdsk"
* NEWFS_DEFAULT_FS "ext2" "ufs"
* Many of the disabled test cases fail because 'zfs/zpool destroy'
returns EBUSY. This is largely causes by the asynchronous nature
of device handling on Linux and is expected, the impacted test
cases will need to be updated to handle this.
* There are several test cases which have been disabled because
they can trigger a deadlock. A primary example of this is to
recursively create zpools within zpools. These tests have been
disabled until the root issue can be addressed.
* Illumos specific utilities such as (mkfile) should be added to
the tests/zfs-tests/cmd/ directory. Custom programs required by
the test scripts can also be added here.
* SELinux should be either is permissive mode or disabled when
running the tests. The test cases should be updated to conform
to a standard policy.
* Redundant test functionality has been removed (zfault.sh).
* Existing test scripts (zconfig.sh) should be migrated to use
the framework for consistency and ease of testing.
* The DISKS environment variable currently only supports loopback
devices because of how the ZFS Test Suite expects partitions to
be named (p1, p2, etc). Support must be added to generate the
correct partition name based on the device location and name.
* The ZFS Test Suite is part of the illumos code base at:
https://github.com/illumos/illumos-gate/tree/master/usr/src/test
Original-patch-by: Turbo Fredriksson <turbo@bayour.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Olaf Faaland <faaland1@llnl.gov>
Closes#6Closes#1534
ZFS on Linux is regularly tested on arm, ppc, ppc64, i686 and x86_64
architectures. Given this the artificial architecture restriction in
the packaging has been removed.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Historically libblkid support was detected as part of configure
and optionally enabled. This was done because at the time support
for detecting ZFS pool vdevs had just be added to libblkid and
those updated packages were not yet part of many distributions.
This is no longer the case and any reasonably current distribution
will ship a version of libblkid which can detect ZFS pool vdevs.
This patch makes libblkid mandatory at build time and libblkid
the preferred method of scanning for ZFS pools. For distributions
which include a modern version of libblkid there is no change in
behavior. Explicitly scanning the default search paths is still
supported and can be enabled with the '-s' command line option.
Additionally making libblkid mandatory means that the 'zpool create'
command can reliably detect if a specified device has an existing
non-ZFS filesystem (ext4, xfs) and print a warning.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#2448
The _initconfdir macro is normally provided by global rpm macros
file for use in the spec file. However, older distributions such
as CentOS 6 do not define it. To prevent a build failure in this
case the spec file has been updated to use a reasonable default
when the value is undefined.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#3617
The dracut code is analogous to the initramfs code and as such
it should be located in the contrib with initramfs for consistency.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
* Supports booting of a ZFS snapshot.
Do this by cloning the snapshot into a dataset. If this, the resulting
dataset, already exists, destroy it. Then mount it on root.
* If snapshot does not exist, use base dataset (the part before '@')
as boot filesystem instead.
* If no snapshot is specified on the 'root=' kernel command line, but there
is an '@', then get a list of snapshots below that filesystem and ask the
user which to use.
* Clone with 'mountpoint=none' and 'canmount=noauto' - we mount manually
and explicitly.
* For sub-filesystems, that doesn't have a mountpoint property set, we use
the 'org.zol:mountpoint' to keep track of it's mountpoint.
* Allow rollback of snapshots instead of clone it and boot from the clone.
* Allow mounting a root- and subfs with mountpoint=legacy set
* Allow mounting a filesystem which is using nativ encryption.
* Support all currently used kernel command line arguments
All the different distributions have their own standard on what to specify
on the kernel command line to boot of a ZFS filesystem.
* Extra options:
* zfsdebug=(on,yes,1) Show extra debugging information
* zfsforce=(on,yes,1) Force import the pool
* rollback=(on,yes,1) Rollback (instead of clone) the snapshot
* Only try to import pool if it haven't already been imported
* This will negate the need to force import a pool that have not been exported cleanly.
* Support exclusion of pools to import by setting ZFS_POOL_EXCEPTIONS in /etc/default/zfs.
* Support additional configuration variable ZFS_INITRD_ADDITIONAL_DATASETS
to mount additional filesystems not located under your root dataset.
* Include /etc/modprobe.d/{zfs,spl}.conf in the initrd if it/they exist.
* Include the udev rule to use by-vdev for pool imports.
* Include the /etc/default/zfs file to the initrd.
* Only try /dev/disk/by-* in the initrd if USE_DISK_BY_ID is set.
* Use /dev/disk/by-vdev before anything.
* Add /dev as a last ditch attempt.
* Fallback to using the cache file if that exist if nothing else worked.
* Use /sbin/modprobe instead of built-in (BusyBox) modprobe.
This gets rid of the message "modprobe: can't load module zcommon".
Thanx to pcoultha for finding this.
Signed-off-by: Turbo Fredriksson <turbo@bayour.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#2116Closes#2114
* Based on the init scripts included with Debian GNU/Linux, then take code
from the already existing ones, trying to merge them into one set of
scripts that will work for 'everyone' for better maintainability.
* Add configurable variables to control the workings of the init scripts:
* ZFS_INITRD_PRE_MOUNTROOT_SLEEP
Set a sleep time before we load the module (used primarily by initrd
scripts to allow for slower media (such as USB devices etc) to be
availible before we load the zfs module).
* ZFS_INITRD_POST_MODPROBE_SLEEP
Set a timed sleep in the initrd to after the load of the zfs module.
* ZFS_INITRD_ADDITIONAL_DATASETS
To allow for mounting additional datasets in the initrd. Primarily used
in initrd scripts to allow for when filesystem needed to boot (such as
/usr, /opt, /var etc) isn't directly under the root dataset.
* ZFS_POOL_EXCEPTIONS
Exclude pools from being imported (in the initrd and/or init scripts).
* ZFS_DKMS_ENABLE_DEBUG, ZFS_DKMS_ENABLE_DEBUG_DMU_TX, ZFS_DKMS_DISABLE_STRIP
Set to control how dkms should build the dkms packages.
* ZPOOL_IMPORT_PATH
Set path(s) where "zpool import" should import pools from.
This was previously the job of "USE_DISK_BY_ID" (which is still used
for backwards compatibility) but was renamed to allow for better
control of import path(s).
* If old USE_DISK_BY_ID is set, but not new ZPOOL_IMPORT_PATH, then we
set ZPOOL_IMPORT_PATH to sane defaults just to be on the safe side.
* ZED_ARGS
To allow for local options to zed without having to change the init script.
* The import function, do_import(), imports pools by name instead of '-a'
for better control of pools to import and from where.
* If USE_DISK_BY_ID is set (for backwards compatibility), but isn't 'yes'
then ignore it.
* If pool(s) isn't found with a simple "zpool import" (seen it happen),
try looking for them in /dev/disk/by-id (if it exists). Any duplicates
(pools found with both commands) is filtered out.
* IF we have found extra pool(s) this way, we must force USE_DISK_BY_ID
so that the first, simple "zpool import $pool" is able to find it.
* Fallback on importing the pool using the cache file (if it exists) only
if 'simple' import (either with ZPOOL_IMPORT_PATH or the 'built in'
defaults) didn't work.
* The export function, do_export(), will export all pools imported, EXCEPT
the root pool (if there is one).
* ZED script from the Debian GNU/Linux packages added.
* Refreshed ZED init script from behlendorf@5e7a660 to be portable so it
may be used on both LSB and Redhat style systems.
* If there is no pool(s) imported and zed successfully shut down, we will
unload the zfs modules.
* The function library file for the ZoL init script is installed as
/etc/init.d/zfs-functions.
* The four init scripts, the /etc/{defaults,sysconfig,conf.d}/zfs config file
as well as the common function library is tagged as '%config(noreplace)' in
the rpm rules file to make sure they are not replaced automatically if locally
modifed.
* Pitfals and workarounds:
* If we're running from init, remove stale /etc/dfs/sharetab before importing
pools in the zfs-import init script.
* On Debian GNU/Linux, there's a 'sendsigs' script that will kill basically
everything quite early in the shutdown phase and zed is/should be stopped
much later than that. We don't want zed to be among the ones killed, so add
the zed pid to list of pids for 'sendsigs' to ignore.
* CentOS uses echo_success() and echo_failure() to print out status of
command. These in turn uses "echo -n \0xx[etc]" to move cursor and choose
colour etc. This doesn't work with the modified IFS variable we need to
use in zfs-import for some reason, so work around that when we define
zfs_log_{end,failure}_msg() for RedHat and derivative distributions.
* All scripts passes ShellCheck (with one false positive in do_mount()).
Signed-off-by: Turbo Fredriksson turbo@bayour.com
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Richard Yao <ryao@gentoo.org>
Reviewed by: Chris Dunlap <cdunlap@llnl.gov>
Closes#2974Closes#2107
Missing space made the %post directive be part of the package
%description and not have a %post scriptlet defined.
Signed-off-by: Andy Bakun <github@thwartedefforts.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#2961
Providing a pkg-config file makes is easy for 3rd party applications
to link against the libzfs libraries. It also allows the libzfs
developers to modify the list of required libraries and cflags
without breaking existing applications.
The following example illustrates how pkg-config can be used:
cc `pkg-config --cflags --libs libzfs` -o myapp myapp.c
/*
* myapp.c
*/
void main()
{
libzfs_handle_t *hdl;
hdl = libzfs_init();
if (hdl)
libzfs_fini(hdl);
}
Signed-off-by: Turbo Fredriksson <turbo@bayour.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes: #585
Commit 2ee4e7da accidentally introduced two issues which only occur
when rebuilding the ZFS source rpm outside the ZFS build system.
1) The _dracutdir, _udevdir, and _udevruledir macros must be checked
using the 'undefined' keyword. This was just overlooked in the
patch review and does not cause a failure when using 'make pkg'
because the values are provided by the make target.
2) The default _udevruledir path included a typo.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #2310
There are two common locations where udev and dracut components are
commonly installed. When building packages using the 'make rpm|deb'
targets check those common locations and pass them to rpmbuild. For
non-standard configurations these values can be provided by the
the following configure options:
--with-udevdir=DIR install udev helpers [default=check]
--with-udevruledir=DIR install udev rules [[UDEVDIR/rules.d]]
--with-dracutdir=DIR install dracut helpers [default=check]
When rebuilding using the source packages the per-distribution
default values specified in the spec file will be used. This is
the preferred way to build packages for a distribution but the
ability to override the defaults is provided as a convenience.
Signed-off-by: Turbo Fredriksson <turbo@bayour.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#2310Closes#1680
From day one the various ZFS libraries should have been placed in their
own sub-packages. Primarily this allows for multiple major versions of
the libraries to be concurrently installed. It also facilitates a
smaller build environment by minimizing the required dependencies.
The specific changes required to split the libraries from the utilities
are as follows:
* libzpool2, libnvpair1, libuutil1, and libzfs2 packages were added
and contain the versioned shared libraries. The Fedora packaging
guidelines discourage providing static libraries so they are not
included in the packages.
http://fedoraproject.org/wiki/Packaging:Guidelines#Packaging_Static_Libraries
* The zfs-devel package was renamed libzfs2-devel and the new package
obsoletes the old zfs-devel package. This package includes all
the required headers for the libzpool2, libnvpair1, libuutil1, and
libzfs2 libraries and their respective unversioned shared libraries.
This package should eventually be split in to individual lib*-devel
packages but it will still take some work to cleanly separate them.
Therefore the libzfs2-devel package provides the expected lib*-devel
packages so the all proper dependencies can still be created.
http://fedoraproject.org/wiki/Packaging:Guidelines#Devel_Packages
* Moved '/sbin/ldconfig' execution from the zfs packge to each of the
new library packages as described by the packaging guidelines.
http://fedoraproject.org/wiki/Packaging:Guidelines#Shared_Libraries
* The /usr/share/doc/ files were moved in to the libzfs2-devel package.
* Updated config/deb.am to be aware of the packaging changes. This
ensures that 'deb-utils' make target converts all the resulting
packages generated by the 'rpm-utils' target.
Signed-off-by: Turbo Fredriksson <turbo@bayour.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes: #2329Closes: #2341
Issue: #2145
zed monitors ZFS events. When a zevent is posted, zed will run any
scripts that have been enabled for the corresponding zevent class.
Multiple scripts may be invoked for a given zevent. The zevent
nvpairs are passed to the scripts as environment variables.
Events are processed synchronously by the single thread, and there is
no maximum timeout for script execution. Consequently, a misbehaving
script can delay (or forever block) the processing of subsequent
zevents. Plans are to address this in future commits.
Initial scripts have been developed to log events to syslog
and send email in response to checksum/data/io errors and
resilver.finish/scrub.finish events. By default, email will only
be sent if the ZED_EMAIL variable is configured in zed.rc (which is
serving as a config file of sorts until a proper configuration file
is implemented).
Signed-off-by: Chris Dunlap <cdunlap@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #2
zfs-fuse provides the same commands and man page names as ZoL.
Changing the names on either side would make each incompatible with
all existing documentation about ZFS. Providing bit identical files
is not possible due to differing codebases.
Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Turbo Fredriksson <turbo@bayour.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1866
This adds systemd unit files replacing the functionality offered by
the SysV init script found in etc/init.d.
It has been developed and tested on Fedora 19, Fedora 20
and openSuSE 13.1.
Four unit files and one target are offered.
zfs-import-cache.service:
Import pools from /etc/zfs/zpool.cache. This unit will wait for
udev to settle.
zfs-import-scan.service:
Import pools by scanning /dev/disk/by-id for zvols. This unit will
only run if /etc/zfs/zpool.cache is not present. This unit will wait
for udev to settle
zfs-mount.service:
Mount ZFS native filesystems. It contains a dependency to be loaded
before local-fs.target.
zfs-share.service:
Share NFS/SMB filesystems. This unit contains a dependency that
will cause it to be restarted whenever the smb or nfs-server unit
is restarted, restoring the shares added.
zfs.target:
This target pulls in the other units in order to start ZFS. It's
the only unit that can be enabled/disabled, all other services
are static and pulled in by dependencies. It will honour zfs=off
and zfs=no options on the kernel command line.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#2108
Four new dataset properties have been added to support SELinux. They
are 'context', 'fscontext', 'defcontext' and 'rootcontext' which map
directly to the context options described in mount(8). When one of
these properties is set to something other than 'none'. That string
will be passed verbatim as a mount option for the given context when
the filesystem is mounted.
For example, if you wanted the rootcontext for a filesystem to be set
to 'system_u:object_r:fs_t' you would set the property as follows:
$ zfs set rootcontext="system_u:object_r:fs_t" storage-pool/media
This will ensure the filesystem is automatically mounted with that
rootcontext. It is equivalent to manually specifying the rootcontext
with the -o option like this:
$ zfs mount -o rootcontext=system_u:object_r:fs_t storage-pool/media
By default all four contexts are set to 'none'. Further information
on SELinux contexts is detailed in mount(8) and selinux(8) man pages.
Signed-off-by: Matthew Thode <prometheanfire@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Richard Yao <ryao@gentoo.org>
Closes#1504
This is an extension of commit ffb2111. As the fedora conditional
has been added, this allows centos/rhel-6 to fall back to the
proper directory (/usr/share/dracut)
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1984
Make zfs depend on the same version of zfs-kmod, rather than on same or
better. When yum repository contains a number of versions the dependency
resolution breaks on trying to install non-latest version.
Signed-off-by: Cyril Plisko <cyril.plisko@mountall.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1677
The GNU libtool documentation states to start with a version of 0:0:0,
rather than 1:1:0. Illumos uses the name libzfs_core.so.1, so to be
consistent, we should go with 1:0:0.
http://www.gnu.org/software/libtool/manual/libtool.html#Updating-version-info
The GNU libtool documentation also provides guidence on how the version
information should be incremented. Doing this does a SONAME bump of the
libzfs and libzpool libraries. This is particularly important on Gentoo
because a SONAME bump enables portage to retain the older libraries
until any packages that link to them are rebuilt. The main example of
this is GRUB2's grub2-mkconfig, which will break unless it is rebuilt
against the new libraries.
Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #1751
Unconditionally exit with zero to avoid returning failures
from the scriptlets. This should have been part of the
previous ba661a6 commit.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1376
Prior to adopting the kmod style packaging the zfs packages
would conditionally invoke /sbin/chkconfig to create the
proper links for the init script. This is done conditionally
because many distributions are moving away from SysV style
init scripts and we don't want to cause errors on those.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1376
Remove from the zfs package the depenencies on the zfs-dracut and
zfs-test subpackages. Neither of these packages are required for
normal operation and they bring in many unnecessary dependencies
during installation.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1395
* Modified kstat_update() to read arcstats from proc.
* Fix shebang.
* Added Makefile.am entries for arcstat.py
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1506
The distribution specific init scripts where excluded from the
packaging when it was reworked. The intention is to replace
them with systemd equivilants. However, that work has not yet
been done and the init scripts are still useful so they have
been added back in to the packaging.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
The spl, zfs-test, and zfs-dracut packages should be pulled in
by the core zfs package. This allows all the required zfs packages
to be installed from a yum repository by running:
yum install zfs
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Refresh the existing RPM packaging to conform to the 'Fedora
Packaging Guidelines'. This includes adopting the kmods2
packaging standard which is used fod kmods distributed by
rpmfusion for Fedora/RHEL.
http://fedoraproject.org/wiki/Packaging:Guidelineshttp://rpmfusion.org/Packaging/KernelModules/Kmods2
While the spec files have been entirely rewritten from a
user perspective the only major changes are:
* The Fedora packages now have a build dependency on the
rpmfusion repositories. The generic kmod packages also
have a new dependency on kmodtool-1.22 but it is bundled
with the source rpm so no additional packages are needed.
* The kernel binary module packages have been renamed from
zfs-modules-* to kmod-zfs-* as specificed by kmods2.
* The is now a common kmod-zfs-devel-* package in addition
to the per-kernel devel packages. The common package
contains the development headers while the per-kernel
package contains kernel specific build products.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1341
John Wren Kennedy
Brian Behlendorf
Turbo Fredriksson
Andy Bakun
Ben Allen
Chris Dunlap
Ralf Ertzinger
Matthew Thode
Aaron Fineman
Cyril Plisko
Richard Yao
Christ Schlacta