Commit Graph

72 Commits

Author SHA1 Message Date
Gregor Kopka 8bd2a2866c Removed suggestion to use root dataset as bootfs
The dracut howto proposed to boot from the root dataset of a pool.
Apart from this giving problems when booting (as the code seems to
expect a child dataset and creates an illegal dataset name when using
the root dataset) the technical limitations of the root dataset
(among others the inability to rename or destroy through the `zfs`
command) resulted in the general consensus to only use it as a
container for the datasets in the pool - not as a filesystem itself.

Removed the idea to boot from the root dataset.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: bunder2015 <omfgbunder@gmail.com>
Signed-off-by: Gregor Kopka <gregor@kopka.net>
Closes #8247
2019-01-08 16:15:30 -08:00
Neal Gompa (ニール・ゴンパ) 9ef798b771 Use ZFS version for pyzfs & drop unused reqs file
Now that 'pyzfs' is part of the ZFS codebase, it should be
versioned the same as the rest of the source tree. This eliminates
confusion on what version of the bindings are being used, especially
for dependent Python projects that may use the Python dist metadata
to identify compatible versions of pyzfs to work from.

In addition, a trivial change to drop the unused requirements.txt
file is included, simply because it's unused and a leftover from
before it was imported into the ZFS codebase and wired into the
autotools build scripts.

Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Neal Gompa <ngompa@datto.com>
Closes #8243
2019-01-08 15:56:42 -08:00
Brian Behlendorf 6e72a5b9b6 pyzfs: python3 support (build system)
Almost all of the Python code in the respository has been updated
to be compatibile with Python 2.6, Python 3.4, or newer.  The only
exceptions are arc_summery3.py which requires Python 3, and pyzfs
which requires at least Python 2.7.  This allows us to maintain a
single version of the code and support most default versions of
python.  This change does the following:

* Sets the default shebang for all Python scripts to python3.  If
  only Python 2 is available, then at install time scripts which
  are compatible with Python 2 will have their shebangs replaced
  with /usr/bin/python.  This is done for compatibility until
  Python 2 goes end of life.  Since only the installed versions
  are changed this means Python 3 must be installed on the system
  for test-runner when testing in-tree.

* Added --with-python=<2|3|3.4,etc> configure option which sets
  the PYTHON environment variable to target a specific python
  version.  By default the newest installed version of Python
  will be used or the preferred distribution version when
  creating pacakges.

* Fixed --enable-pyzfs configure checks so they are run when
  --enable-pyzfs=check and --enable-pyzfs=yes.

* Enabled pyzfs for Python 3.4 and newer, which is now supported.

* Renamed pyzfs package to python<VERSION>-pyzfs and updated to
  install in the appropriate site location.  For example, when
  building with --with-python=3.4 a python34-pyzfs will be
  created which installs in /usr/lib/python3.4/site-packages/.

* Renamed the following python scripts according to the Fedora
  guidance for packaging utilities in /bin

  - dbufstat.py     -> dbufstat
  - arcstat.py      -> arcstat
  - arc_summary.py  -> arc_summary
  - arc_summary3.py -> arc_summary3

* Updated python-cffi package name.  On CentOS 6, CentOS 7, and
  Amazon Linux it's called python-cffi, not python2-cffi.  For
  Python3 it's called python3-cffi or python3x-cffi.

* Install one version of arc_summary.  Depending on the version
  of Python available install either arc_summary2 or arc_summary3
  as arc_summary.  The user output is only slightly different.

Reviewed-by: John Ramsden <johnramsden@riseup.net>
Reviewed-by: Neal Gompa <ngompa@datto.com>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8096
2019-01-06 10:39:41 -08:00
Brian Behlendorf 4b1c4062d0 pyzfs: python3 support (unit tests)
* Updated unit tests to be compatbile with python 2 or 3.  In most
  cases all that was required was to add the 'b' prefix to existing
  strings to convert them to type bytes for python 3 compatibility.

* There were several places where the python version need to be
  checked to remain compatible with pythong 2 and 3.  Some one
  more seasoned with Python may be able to find a way to rewrite
  these statements in a compatible fashion.

Reviewed-by: John Ramsden <johnramsden@riseup.net>
Reviewed-by: Neal Gompa <ngompa@datto.com>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Signed-off-by: John Wren Kennedy <john.kennedy@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8096
2019-01-06 10:39:41 -08:00
Brian Behlendorf e5fb1dc586 pyzfs: python3 support (library 2/2)
* All pool, dataset, and nvlist keys must be of type bytes.

Reviewed-by: John Ramsden <johnramsden@riseup.net>
Reviewed-by: Neal Gompa <ngompa@datto.com>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Signed-off-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8096
2019-01-06 10:39:41 -08:00
Antonio Russo 9de8c0cd7f pyzfs: python3 support (library 1/2)
These changes are efficient and valid in python 2 and 3. For the
most part, they are also pythonic.

* 2to3 conversion
* add __future__ imports
* iterator changes
* integer division
* relative import fixes

Reviewed-by: John Ramsden <johnramsden@riseup.net>
Reviewed-by: Neal Gompa <ngompa@datto.com>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Antonio Russo <antonio.e.russo@gmail.com>
Closes #8096
2019-01-06 10:39:41 -08:00
Andriy Gapon dc1c630b8a OpenZFS 9630 - add lzc_rename and lzc_destroy to libzfs_core
Porting Notes:
* Additional changes to recv_rename_impl() were required due to
  encryption code not being merged in OpenZFS yet.
* libzfs_core python bindings (pyzfs) were updated to fully support
  both lzc_rename() and lzc_destroy()

Authored by: Andriy Gapon <avg@FreeBSD.org>
Reviewed by: Andy Stormont <astormont@racktopsystems.com>
Reviewed by: Matt Ahrens <matt@delphix.com>
Reviewed by: Serapheim Dimitropoulos <serapheim.dimitro@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Approved by: Dan McDonald <danmcd@joyent.com>
Ported-by: loli10K <ezomori.nozomu@gmail.com>

OpenZFS-issue: https://www.illumos.org/issues/9630
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/049ba63
Closes #8207
2018-12-14 09:49:45 -08:00
Ben Cordero eff7d78f8a Add `cut` binary to the initramfs
Since the `cut -b` command is used by `parse-zfs.sh`,
ensure that it is copied to the initramfs.

Fix spl_hostid when set by cmdline. This follows a
similar logic from the `zgenhostid` script, using `echo`
instead of `printf`.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ben Cordero <bencord0@condi.me>
Closes #8197
2018-12-13 15:48:46 -08:00
kpande eb1a0b6174 Allow spaces in pool names for cmdline argument
PR #8114 quoted the ${ENCRYPTIONROOT} parameter to ensure we don't
lose spaces when unlocking root filesystem in the off chance that 
it has a space in its name.

Unfortunately, dracut and initramfs-tools do not actually get the 
quotes from the cmdline. If we use root=ZFS="root pool/filesystem 
name" the script still only sees root=ZFS=root and no quotation 
marks.

Because + is a reserved character in ZFS, it's used as a 
placeholder for spaces in the kernel cmdline.  In this way,
root=ZFS=root+pool/filesystem+name will properly expand by 
replacing the character with sed (POSIX compliant method).

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: bunder2015 <omfgbunder@gmail.com>
Signed-off-by: Kash Pande <kash@tripleback.net>
Issue #8114 
Closes #8117
2018-11-11 18:23:11 -08:00
Garrett Fields 0500bfd0b9 Make initramfs-tools script encryption aware
Changed decrypt_fs zfs command to "load-key"
Plymouth case code based on "contrib/dracut/90zfs/zfs-lib.sh.in"
Systemd case based on "contrib/dracut/90zfs/zfs-load-key.sh.in"
Cleaned up misspelling of "available" throughout

Code style fixes
Single quote for ${ENCRYPTIONROOT}
Changed "${DECRYPT_CMD}"  to "eval ${DECRYPT_CMD}"

Reviewed-by: Kash Pande <kash@tripleback.net>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Signed-off-by: Garrett Fields <ghfields@gmail.com>
Closes #8093
2018-11-09 11:30:09 -08:00
kpande 13c59bb76b Add quotations for ${ENCRYPTIONROOT}
Add quotations for ${ENCRYPTIONROOT} to avoid breaking systems
with a space in the name.

Reviewed-by: bunder2015 <omfgbunder@gmail.com>
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Kash Pande <kash@tripleback.net>
Related-to: #8093 
Closes #8114
2018-11-09 09:32:01 -08:00
George Melikov e871a8f058 Allow use of pool GUID as root pool
It's helpful if there are pools with same names,
but you need to use only one of them.

Main case is twin servers, meanwhile some software
requires the same name of pools (e.g. Proxmox).

Reviewed-by: Kash Pande <kash@tripleback.net>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Igor ‘guardian’ Lidin of Moscow, Russia
Closes #8052
2018-10-23 20:06:40 -07:00
Garrett Fields 5b3bfd86a4 Check scheduler for "noop" before setting "noop"
Originally code only checked for presence of "/sys/block/$i/queue/
scheduler".  "sh: write error: Invalid argument" was produced when
trying to set "noop" on certain devices (eg. virtio) when it isn't
a listed option. This modification continues to check for the presence
of "/sys/block/$i/queue/scheduler" and also checks that it contains
"noop" as an option before setting "noop".

Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Signed-off-by: Garrett Fields <ghfields@gmail.com>
Closes #8004
2018-10-10 08:46:22 -07:00
LOLi c962fd6c4e pyzfs: add missing libzfs_core functions
This change adds the following libzfs_core functions to pyzfs:
lzc_remap, lzc_pool_checkpoint, lzc_pool_checkpoint_discard

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #7793 
Closes #7800
2018-08-20 10:11:52 -07:00
Tom Caputi d9c460a0b6 Added encryption support for zfs recv -o / -x
One small integration that was absent from b52563 was
support for zfs recv -o / -x with regards to encryption
parameters. The main use cases of this are as follows:

* Receiving an unencrypted stream as encrypted without
  needing to create a "dummy" encrypted parent so that
  encryption can be inheritted.

* Allowing users to change their keylocation on receive,
  so long as the receiving dataset is an encryption root.

* Allowing users to explicitly exclude or override the
  encryption property from an unencrypted properties stream,
  allowing it to be received as encrypted.

* Receiving a recursive heirarchy of unencrypted datasets,
  encrypting the top-level one and forcing all children to
  inherit the encryption.

Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #7650
2018-08-15 09:48:49 -07:00
Antonio Russo 9b9d1adc38 Use zfs-import.target in contrib/dracut
The new zfs-import.target should be used in place of the
zfs-import-*.service units.

Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Manuel Amador (Rudd-O) <rudd-o@rudd-o.com>
Signed-off-by: Antonio Russo <antonio.e.russo@gmail.com>
Closes #6964
2018-07-31 10:15:41 -07:00
George Diamantopoulos fb7307b892 Fix initramfs missing systemd binaries
Systemd binaries necessary for mounting an encrypted root dataset
weren't copied to initramfs generated by dracut. This patch fixes
this and copies these binaries unconditionally, that is
regardless of whether native ZFS encryption is used for the
root dataset.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Diamantopoulos <georgediam@gmail.com>
Closes #7607
Closes #7719
2018-07-27 09:29:43 -07:00
Tom Caputi c634808ebb Add pyzfs build directories to gitignore
The recent addition of pyzfs does not include the generated 'build'
and 'pyzfs.egg-info' directories in the pyzfs .gitignore or the
'make clean' target. This patch simply corrects this problem.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #7612
2018-06-11 18:42:12 -07:00
Steffen Müthing 3c28c63642 Install basename utility into dracut initramfs
vdev_id requires the program `basename` when handling short aliases
defined in `vdev_id.conf` (those defined without a leading path), but
`basename` is not always available in the dracut environment. This
causes the pool device names to change when using `by-vdev/` devices
or (in extreme cases) can make the pool import fail in dracut.

This commit fixes the problem by explicitly installing `basename`.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Steffen Müthing <steffen.muething@iwr.uni-heidelberg.de>
Closes #7562
2018-05-29 17:32:05 -07:00
Tomohiro Kusumi e1245d83e9 Prevent `make distclean` removing 0 sized file
__init__.py used by Python packages typically has nothing in it
including contrib/pyzfs/libzfs_core/test/__init__.py, however this
causes `make distclean` to delete the file.

This is the only file with size 0, and it seems reasonable to have
a comment to avoid being deleted, rather than trying to modify
distclean behavior.

 # find . -size 0
 ./contrib/pyzfs/libzfs_core/test/__init__.py
 # ./autogen.sh ; ./configure ; make -j8
 # make distclean
 # ls contrib/pyzfs/libzfs_core/test/__init__.py
 ls: cannot access 'contrib/pyzfs/libzfs_core/test/__init__.py':
     No such file or directory
 # git diff
 diff --git a/contrib/pyzfs/libzfs_core/test/__init__.py
     b/contrib/pyzfs/libzfs_core/test/__init__.py
 deleted file mode 100644

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@osnexus.com>
Closes #7505
2018-05-06 20:46:13 -07:00
loli10K 85ce3f4fd1 Adopt pyzfs from ClusterHQ
This commit introduces several changes:

 * Update LICENSE and project information

 * Give a good PEP8 talk to existing Python source code

 * Add RPM/DEB packaging for pyzfs

 * Fix some outstanding issues with the existing pyzfs code caused by
   changes in the ABI since the last time the code was updated

 * Integrate pyzfs Python unittest with the ZFS Test Suite

 * Add missing libzfs_core functions: lzc_change_key,
   lzc_channel_program, lzc_channel_program_nosync, lzc_load_key,
   lzc_receive_one, lzc_receive_resumable, lzc_receive_with_cmdprops,
   lzc_receive_with_header, lzc_reopen, lzc_send_resume, lzc_sync,
   lzc_unload_key, lzc_remap

Note: this commit slightly changes zfs_ioc_unload_key() ABI. This allow
to differentiate the case where we tried to unload a key on a
non-existing dataset (ENOENT) from the situation where a dataset has
no key loaded: this is consistent with the "change" case where trying
to zfs_ioc_change_key() from a dataset with no key results in EACCES.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #7230
2018-05-01 10:33:35 -07:00
Andriy Gapon 6abf922574 Import pyzfs source code from ClusterHQ
libzfs_core is intended to be a stable interface for programmatic
administration of ZFS.

This wrapper provides one-to-one wrappers for libzfs_core API functions,
but the signatures and types are more natural to Python.
nvlists are wrapped as dictionaries or lists depending on their usage.
Some parameters have default values depending on typical use for
increased convenience.
Enumerations and bit flags become strings and lists of strings in
Python.
Errors are reported as exceptions rather than integer errno-style
error codes.  The wrapper takes care to provide one-to-many mapping
of the error codes to the exceptions by interpreting a context
in which the error code is produced.

Unit tests and automated test for the libzfs_core API are provided
with this package.

Please note that the API tests perform lots of ZFS dataset level
operations and ZFS tries hard to ensure that any modifications
do reach stable storage. That means that the operations are done
synchronously and that, for example, disk caches are flushed.
Thus, the tests can be very slow on real hardware.
It is recommended to place the default temporary directory or
a temporary directory specified by, for instance, TMP environment
variable on a memory backed filesystem.

Original-patch-by: Andriy Gapon <avg@FreeBSD.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: loli10K <ezomori.nozomu@gmail.com>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #7230
2018-05-01 10:31:11 -07:00
Tony Hutter 6c9af9e8f4 Fix "file is executable, but no shebang" warnings
Fedora 28's RPM build checks warn when executable files don't have a
shebang line.  These warnings are caused when we (incorrectly)
include data & config files in the_SCRIPTS automake lines. Files in
_SCRIPTS are marked executable by automake. This patch fixes the
issue by including non-executable scripts in a _DATA line instead.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Tony Hutter <hutter2@llnl.gov>
Closes #7359 
Closes #7395
2018-04-06 16:34:21 -07:00
kpande 05747eca5b modprobe zfs during dracut mount
Resolves importing root pool during boot in dracut.  This case was
inadvertently broken with the module autoloading change in #7287.

Reviewed-by: Matthew Thode <prometheanfire@gentoo.org>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: Kash Pande <kash@tripleback.net>
Closes #7322
2018-03-22 10:14:29 -07:00
John Eismeier d699aaef09 Fix some typos
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: George Melikov <mail@gmelikov.ru>
Signed-off-by: John Eismeier <john.eismeier@gmail.com>
Closes #7237
2018-02-28 08:57:10 -08:00
Kash Pande 41532e5a29 Shellcheck cleanup for initrd scripts
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: Kash Pande <kash@tripleback.net>
Co-authored-by: Matthew Thode <mthode@mthode.org>
Signed-off-by: Kash Pande <kash@tripleback.net>
Signed-off-by: Matthew Thode <mthode@mthode.org>
Closes #7214
2018-02-23 12:57:41 -08:00
Kash Pande 7280d58197 Enable booting from nested encrypted datasets
- enable booting from nested encrypted datasets
- fix plymouth boot splash passphrase entry
- optimize unlock process

Co-authored-by: Kash Pande <kash@tripleback.net>
Co-authored-by: Matthew Thode <mthode@mthode.org>
Signed-off-by: Kash Pande <kash@tripleback.net>
Signed-off-by: Matthew Thode <mthode@mthode.org>
Closes #7214
2018-02-23 12:57:28 -08:00
LOLi a893627fac contrib/initramfs: add missing conf.d/zfs
When upgrading from the distribution-provided zfs-initramfs package on
root-on-zfs Ubuntu and Debian the system may fail to boot: this change
adds the missing initramfs configuration file.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #7158
2018-02-12 11:40:00 -08:00
Chris Williamson d99a015343 OpenZFS 7431 - ZFS Channel Programs
Authored by: Chris Williamson <chris.williamson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Approved by: Garrett D'Amore <garrett@damore.org>
Ported-by: Don Brady <don.brady@delphix.com>
Ported-by: John Kennedy <john.kennedy@delphix.com>

OpenZFS-issue: https://www.illumos.org/issues/7431
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/dfc11533

Porting Notes:
* The CLI long option arguments for '-t' and '-m' don't parse on linux
* Switched from kmem_alloc to vmem_alloc in zcp_lua_alloc
* Lua implementation is built as its own module (zlua.ko)
* Lua headers consumed directly by zfs code moved to 'include/sys/lua/'
* There is no native setjmp/longjump available in stock Linux kernel.
  Brought over implementations from illumos and FreeBSD
* The get_temporary_prop() was adapted due to VFS platform differences
* Use of inline functions in lua parser to reduce stack usage per C call
* Skip some ZFS Test Suite ZCP tests on sparc64 to avoid stack overflow
2018-02-08 15:28:18 -08:00
Matthew Thode 6f259b59cf Only run pre-mount hook zfs-load-key on systemd
Reviewed-by: Kash Pande <kash@tripleback.net>
Reviewed-by: bunder2015 <omfgbunder@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Matthew Thode <mthode@mthode.org>
Closes #7136 
Closes #7140
2018-02-07 18:31:54 -08:00
Brian Behlendorf 6d82b79699
Add zfs-load-key.sh to .gitignore
The generated zfs-load-key.sh file should have been added to
the .gitignore file as part of commit 7da8f8d8.  And the
generated file should not be included in the repo.

Reviewed-by: Matthew Thode <prometheanfire@gentoo.org>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #7134
2018-02-06 16:39:18 -08:00
Tom Caputi ae76f45cda Encryption Stability and On-Disk Format Fixes
The on-disk format for encrypted datasets protects not only
the encrypted and authenticated blocks themselves, but also
the order and interpretation of these blocks. In order to
make this work while maintaining the ability to do raw
sends, the indirect bps maintain a secure checksum of all
the MACs in the block below it along with a few other
fields that determine how the data is interpreted.

Unfortunately, the current on-disk format erroneously
includes some fields which are not portable and thus cannot
support raw sends. It is not possible to easily work around
this issue due to a separate and much smaller bug which
causes indirect blocks for encrypted dnodes to not be
compressed, which conflicts with the previous bug. In
addition, the current code generates incompatible on-disk
formats on big endian and little endian systems due to an
issue with how block pointers are authenticated. Finally,
raw send streams do not currently include dn_maxblkid when
sending both the metadnode and normal dnodes which are
needed in order to ensure that we are correctly maintaining
the portable objset MAC.

This patch zero's out the offending fields when computing
the bp MAC and ensures that these MACs are always
calculated in little endian order (regardless of the host
system's byte order). This patch also registers an errata
for the old on-disk format, which we detect by adding a
"version" field to newly created DSL Crypto Keys. We allow
datasets without a version (version 0) to only be mounted
for read so that they can easily be migrated. We also now
include dn_maxblkid in raw send streams to ensure the MAC
can be maintained correctly.

This patch also contains minor bug fixes and cleanups.

Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Closes #6845
Closes #6864
Closes #7052
2018-02-02 11:37:16 -08:00
Brian Behlendorf f55a8757a6
Update README.initramfs.markdown
Fix several typos and grammar.

Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Reviewed by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Arno van Wyk <avw1987@users.noreply.github.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #7080
2018-01-26 09:55:16 -08:00
Matthew Thode 7da8f8d81b Run zfs load-key if needed in dracut
'zfs load-key -a' will only be called if needed.  If a dataset not
needed for boot does not have its key loaded (home directories for
example) boot can still continue.

zfs:AUTO was not working via dracut, so we still need the generator
script to do its thing.

Reviewed-by: Richard Yao <ryao@gentoo.org>
Reviewed-by: Manuel Amador (Rudd-O) <rudd-o@rudd-o.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Signed-off-by: Matthew Thode <mthode@mthode.org>
Closes #6982 
Closes #7004
2018-01-18 10:20:34 -08:00
LOLi e2d936e0f8 Honor --with-mounthelperdir where applicable
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #6962
2017-12-17 14:14:07 -08:00
JKDingwall 9717fe052b Add /usr/bin/env to COPY_EXEC_LIST initramfs hook
5dc1ff29 changed the user space program to mount a zfs snapshot
from /bin/sh to /usr/bin/env.  If the executable is not present
in the initramfs then snapshots cannot be automounted.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Signed-off-by: James Dingwall <james.dingwall@zynstra.com>
Closes #5360 
Closes #6913
2017-12-04 11:53:57 -08:00
Richard Laager 48ac22d855 initramfs: Honor canmount=off
The initramfs script was not honoring canmount=off.  With this change,
it does.  If the administrator has asked that a filesystem not be
mounted, that should be honored.

As an exception, the initramfs script ignores canmount=off on the
rootfs.  The rootfs should not have canmount=off set either.  However,
mounting it anyway seems harmless because it is being asked for
explicitly.  The point of this exception is to avoid the risk of
breaking existing systems, just in case someone has canmount=off set on
their rootfs.

The initramfs still mounts filesystems with canmount=noauto.  This is
necessary because it is typical to set that on the rootfs so that it can
be cloned.  Without canmount=noauto, the clones' duplicate mountpoints
would conflict.

This is the remainder of the fix for:
https://github.com/zfsonlinux/pkg-zfs/issues/221

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #6897
2017-11-28 09:38:13 -08:00
Richard Laager bd2958dea0 initramfs: Honor mountpoint=none/legacy
For filesystems that are children of the rootfs, when mountpoint=none or
mountpoint=legacy, the initrafms script would assume a mountpoint based
on the dataset path.  Given that the rootfs should have mountpoint=/ and
mountpoint inheritance is is the default behavior of ZFS, this behavior
seems unnecessary.  In any event, it turns mountpoint=none into a no-op.
That removes this option from the administrator, and if someone uses it,
it does not work as expected.  Worse yet, if the mountpoint directory
does not exist (which is the typical case for mountpoint=none), the
mounting and thus the boot process will fail.  For the case of
mountpoint=legacy, the assumed mountpoint may not be the correct value
set in /etc/fstab.

This change makes the initramfs script not mount the filesystem in
either case.  For mountpoint=none, this means we are correctly honoring
the setting.  For mountpoint=legacy, there are two scenarios:  If
canmount=on, the filesystem will be mounted by the normal mechanisms
later in the boot process.  If canmount=noauto, the filesystem will not
be mounted at all, unless the administrator has done something special.
If they're not doing something special and they want it mounted by the
initramfs, they can simply not set mountpoint=legacy.

This is part of the fix for:
https://github.com/zfsonlinux/pkg-zfs/issues/221

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #6897
2017-11-28 09:38:00 -08:00
LOLi cb3b0419ba contrib/initramfs: switch to automake
Use automake to build initramfs scripts and hooks.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu@gmail.com>
Closes #6761
2017-11-07 14:53:57 -08:00
Richard Laager 4fc411f7a3 initramfs: Set elevator=noop on the rpool's disks
ZFS already sets elevator=noop for wholedisk vdevs (for all pools), but
typical root-on-ZFS installations use partitions.  This sets
elevator=noop on the disks in the root pool.

Ubuntu 16.04 and 16.10 had this.  It was lost in 17.04 due to Debian
switching to this upstream initramfs script.

Signed-off-by: Richard Laager <rlaager@wiktel.com>
2017-11-01 21:54:56 -05:00
Richard Laager 11b9dcfb2d initramfs: Fix a spelling error
This fixes a typo in a comment.

Signed-off-by: Richard Laager <rlaager@wiktel.com>
2017-11-01 21:54:28 -05:00
Richard Laager 4767c7a14e initramfs: Fix inconsistent whitespace
This fixes one instance of inconsistent whitespace.

Signed-off-by: Richard Laager <rlaager@wiktel.com>
2017-11-01 21:53:22 -05:00
Brian Behlendorf 21a932b83c Post-Encryption Followup
This PR includes fixes for bugs and documentation issues found 
after the encryption patch was merged and general code improvements 
for long-term maintainability.

Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tom Caputi <tcaputi@datto.com>
Issue #6526
Closes #6639
Closes #6703
Cloese #6706
Closes #6714
Closes #6595
2017-10-13 10:02:39 -07:00
Antonio Russo 085b501fb8 Explicitly depend on icp module in initramfs hook
Automatic dependency resolution is unreliable on many systems.
Follow suit with existing code, and explicitly include icp
in module dependencies.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Antonio Russo <antonio.e.russo@gmail.com>
Closes #6751
2017-10-12 09:39:45 -07:00
Tom Caputi 4807c0badb Encryption patch follow-up
* PBKDF2 implementation changed to OpenSSL implementation.

* HKDF implementation moved to its own file and tests
  added to ensure correctness.

* Removed libzfs's now unnecessary dependency on libzpool
  and libicp.

* Ztest can now create and test encrypted datasets. This is
  currently disabled until issue #6526 is resolved, but
  otherwise functions as advertised.

* Several small bug fixes discovered after enabling ztest
  to run on encrypted datasets.

* Fixed coverity defects added by the encryption patch.

* Updated man pages for encrypted send / receive behavior.

* Fixed a bug where encrypted datasets could receive
  DRR_WRITE_EMBEDDED records.

* Minor code cleanups / consolidation.

Signed-off-by: Tom Caputi <tcaputi@datto.com>
2017-10-11 16:54:48 -04:00
aun d4404c3fdb Fix boot from ZFS issues
* Correct ZFS snapshot listing
* Disable "lvm is not available" message on quiet boot

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Alar Aun <spamtoaun@gmail.com>
Closes #6700 
Closes #6747
2017-10-11 10:06:20 -07:00
privb0x23 4f23c5d0c4 Fix inclusion of libgcc_s.so on Void
On Void Linux (x86_64 musl) libgcc_s.so is located in "/usr/lib"
so it is not found by dracut and it produces an error.

Add a simple additional path check for "/usr/lib/libgcc_s.so*"
and install it in the initramfs.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: privb0x23 <privb0x23@users.noreply.github.com>
Closes #6715
2017-10-09 14:34:26 -07:00
Fabian-Gruenbichler 42a76fc8d7 dracut: make module-setup.sh shebang explicit
while these are source by dracut (which is a bash script)
the practical difference is small, but it is more correct:

/bin/sh is not bash on all systems (e.g. Debian and its
derivatives use /bin/dash as /bin/sh by default).

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Closes #6491
2017-08-14 10:56:04 -07:00
Karsten Kretschmer d19a6d5c80 dracut: Install commands required for vdev_id
The vdev_id script requires awk, grep, and head.  Use dracut_install to
ensure that these commands are available in the initrd environment.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Karsten Kretschmer <kkretschmer@gmail.com>
Closes #6443
Closes #6452
2017-08-04 11:14:48 -07:00
Matt Kemp 912e2ba92f Fix initramfs hook for merged /usr/lib and /lib
Under a merged `/lib` -> `/usr/lib` which renders `/lib` as a symlink,
`find /lib -type f -name libgcc_s.so.1` will not return a result as
`find` will not traverse the symlink. Modifying it to `find /lib/ -type
f -name libgcc_s.so.1` should work for both symlinked and non-symlinked
`/lib` directories.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Matt Kemp <matt@mattikus.com>
Closes #5834
2017-02-27 12:03:23 -08:00