Commit Graph

6837 Commits

Author SHA1 Message Date
Paul Dagnelie 99b281f1ae
Fix double mutex_init bug in send code
It was possible to cause a kernel panic in the send code by 
initializing an already-initialized mutex, if a record was created 
with type DATA, destroyed with a different type (bypassing the 
mutex_destroy call) and then re-allocated as a DATA record again.

We tweak the logic to not change the type of a record once it has 
been created, avoiding the issue.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Jorgen Lundman <lundman@lundman.net>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #10374
2020-06-03 19:53:21 -07:00
George Melikov 52998c7f36
Update wiki links with new address
Use direct links to new documentation resource.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Melikov <mail@gmelikov.ru>
Closes #10395
2020-06-03 19:46:31 -07:00
Allan Jude 7da304bbce zfsvfs_setup(): zap_stats_t may have undefined content when accessed
Signed-off-by: Allan Jude <allanjude@klarasystems.com>
2020-06-03 22:20:27 +00:00
Ryan Moeller c0eb5c35ef
FreeBSD: Simplify zvol and fix locking
zvol_geom_bio_strategy should handle its own use of the zvol
suspend reader lock and ensure the zilog exists when needed.

A few other places using the zvol zilog should use the suspend
reader lock as well.

Simplify consumers of zvol_geom_bio_strategy, fix the locking, and
while in here, use the boolean_t constants with doread.

Reviewed-by: Matt Macy <mmacy@FreeBSD.org>
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #10381
2020-06-03 10:45:12 -07:00
Ryan Moeller a9dcfac51c
Periodically update ARC kstats
FreeBSD needs arc_adjust_zthr to run periodically for kstats to be
updated.  A comment in the code suggests this may have been the
original intent in illumos as well:

c946d5a913/module/zfs/arc.c (L4697-L4700)

Create the thread with a 1 second timer.

Reviewed-by: Matt Macy <mmacy@FreeBSD.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #10371
2020-06-03 09:52:38 -07:00
Jorgen Lundman 081de9a86d
Restore avl_update() calls and related functions
The macOS kmem implementation uses avl_update() and related
functions.  These same function exist in the Solaris AVL code but
were removed because they were unused.  Restore them.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Jorgen Lundman <lundman@lundman.net>
Closes #10390
2020-06-03 09:49:32 -07:00
Matthew Macy 3bf3b164ee
Fix crypto build on FreeBSD HEAD
Update API usage to reflect recent change.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
Closes #10384
2020-05-30 12:54:57 -07:00
gregory-lee-bartholomew 9052e3d70b
Add bootfs.snapshot and bootfs.rollback kernel parameters
Unlike other filesystems, snapshots and rollbacks of bootfs need to be
done from a rescue environment. This patch makes it possible to snap-
shot or rollback the bootfs simply by specifying bootfs.snapshot or
bootfs.rollback on the kernel command line. The operation will be
performed by dracut just before bootfs is mounted.

Reviewed-by: Antonio Russo <antonio.e.russo@gmail.com> 
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Gregory Bartholomew <gregory.lee.bartholomew@gmail.com>
Closes #10198
2020-05-29 21:16:08 -07:00
Brian Behlendorf 3d93161b01
ztest: Fix ztest_run_zdb() failure
It's possible for ztest to be killed while the pool is exported
which results in an empty cache file.  This is a valid state to
test, but the validation check performed by ztest_run_zdb()
depends on the pool being in the cache file.  If it's not the
following error is printed.

    zdb -bccsv -G -d -Y -U /tmp/zloop-run/zpool.cache ztest
    zdb: can't open '/tmp/zloop-run': No such file or directory

Resolve these failures by removing the dependency on the cache
file.  Functionally, we only care that the pool can be imported
and that the zdb verification passes.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10385
2020-05-29 21:14:10 -07:00
allen-4 4d829ad9c1
Update zfs-functions.in
The init.d zfs-share script does not perform the intended 
action without having a variable set for ZFS_SHARE and 
ZFS_UNSHARE

Assign default values to ZFS_SHARE and ZFS_UNSHARE. Export 
the environment variables after sourcing the configuration 
file.

Reviewed-by: Richard Yao <ryao@gentoo.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Georgy Yakovlev <gyakovlev@gentoo.org>
Signed-off-by: Allen Holl <allen.m.holl@gmail.com>
Closes #10341 
Closes #10382
2020-05-29 12:01:57 -07:00
наб a1ba120927 Always use "%lld" for formatting time_ts
Given the following test program:
  #include <time.h>
  #include <stdio.h>
  #include <stdint.h>
  int main() {
    printf("time_t:    %d\n", sizeof(time_t));
    printf("long:      %d\n", sizeof(long));
    printf("long long: %d\n", sizeof(long long));
  }

These are output on various x86 architectures:
  x32$   time_t:    8
  x32$   long:      4
  x32$   long long: 8

  amd64$ time_t:    8
  amd64$ long:      8
  amd64$ long long: 8

  i386$  time_t:    4
  i386$  long:      4
  i386$  long long: 8

Therefore code using "%l[du]" to format time_ts produced warnings on x32

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@gmail.com>
Closes #10357
Closes #844
2020-05-28 10:29:58 -07:00
наб 6059f3a1f6 Correctly handle the x32 ABI
__x86_64__ && _ILP32 => don't forcibly define _LP64

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@gmail.com>
Closes #10357
Closes #844
2020-05-28 10:28:20 -07:00
John Gallagher 50ff632787
Rework error handling in zpool_trim()
When a manual trim is run against an entire pool, errors about
particular devices which don't support trim are suppressed. This changes
zpool_trim() in libzfs so that it doesn't return an error when the only
errors are suppressed ones. An exception is made when none of the
devices support trim, in which case an error is reported and a non-zero
status is returned.

This also fixes how the --wait flag works in the presence of suppressed
errors. In particular, suppressed errors no longer cause zpool_trim()
to skip the wait.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: John Gallagher <john.gallagher@delphix.com>
Closes #10263 
Closes #10372
2020-05-27 17:27:28 -07:00
Georgy Yakovlev 40f96f4643
etc/zfs/Makefile.am: set initconfdir
The initconfdir variable is not defined in etc/zfs/Makefile,
so the sed code does not perform the correct replacement.

Reviewed-by: Richard Yao <ryao@gentoo.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Georgy Yakovlev <gyakovlev@gentoo.org>
Issue #10375 
Closes #10376
2020-05-27 17:22:19 -07:00
Ryan Moeller 4f8b2356cc
ZTS: Retry export/destroy when busy in zpool_import_012
It can take a moment for the NFS server to give up the mountpoint
after unsharing a filesystem.

Use log_must_busy to retry export/destroy a few times after switching
off sharenfs.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #10380
2020-05-27 17:18:06 -07:00
Jorgen Lundman 70a5fc0530
Memory leak in dsl_destroy_snapshots_nvl error case
The dsl_destroy_snapshots_nvl() function has an early error out, 
and temporary nvlists were not freed.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Jorgen Lundman <lundman@lundman.net>
Closes #10366
2020-05-26 16:13:41 -07:00
Brian Behlendorf d1b84da8c1
Revert "Let zfs mount all tolerate in-progress mounts"
This reverts commit a9cd8bf which introduced a segfault when running
`zfs mount -a` multiple times when there are mountpoints which are
not empty.  This segfault is now seen frequently by the CI after
the mount code was updated to directly call mount(2).

The original reason this logic was added is described in #8881.
Since then the systemd `zfs-share.target` has been updated to run
"After" the `zfs-mount.server` which should avoid this issue.

Reviewed-by: Don Brady <don.brady@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #9560
Closes #10364
2020-05-26 16:07:50 -07:00
Marcel Schilling ce98ed25de
Fix dead links http://list.zfsonlinux.org
Originally, I wanted to point to directly to
https://zfsonlinux.topicbox.com/groups/zfs-discuss
as the text refers to that specific mailing list, but George Melikov
requested to change it to the general to give users the overview.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Marcel Schilling <marcel.schilling@uni-luebeck.de>
Closes #10367
Closes #10369
2020-05-26 15:09:25 -07:00
Brian Behlendorf c946d5a913
ZTS: Fix zfs_mount.kshlib cleanup
Update cleanup_filesystem to use destroy_dataset when performing
cleanup.  This ensures the destroy is retried if the pool is busy
preventing occasional failures.

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <guss80@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10358
2020-05-23 17:13:42 -07:00
Brian Atkinson fb822260b1
Gang ABD Type
Adding the gang ABD type, which allows for linear and scatter ABDs to
be chained together into a single ABD.

This can be used to avoid doing memory copies to/from ABDs. An example
of this can be found in vdev_queue.c in the vdev_queue_aggregate()
function.

Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: Brian <bwa@clemson.edu>
Co-authored-by: Mark Maybee <mmaybee@cray.com>
Signed-off-by: Brian Atkinson <batkinson@lanl.gov>
Closes #10069
2020-05-20 18:06:09 -07:00
felixdoerre 501a1511ae
mount: use the mount syscall directly
Allow zfs datasets to be mounted on Linux without relying on the
invocation of an external processes.  This is the same behavior
which is implemented for FreeBSD.

Use of the libmount library was originally considered because it 
provides functionality to properly lock and update the /etc/mtab 
file.  However, these days /etc/mtab is typically a symlink to 
/proc/self/mounts so there's nothing to updated.  Therefore, we
call mount(2) directly and avoid any additional dependencies. 

If required the legacy behavior can be enabled by setting the 
ZFS_MOUNT_HELPER environment variable.  This may be needed in
environments where SELinux in enabled and the zfs binary does  
not have mount permission.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Felix Dörre <felix@dogcraft.de>
#10294
2020-05-20 18:02:41 -07:00
DeHackEd 57434abae6
Use boot_ncpus in place of max_ncpus in taskq_create
Due to hotplug support or BIOS bugs sometimes max_ncpus can be
an absurdly high value. I have a system with 32 cores/threads
but reports max_ncpus == 440. This many threads potentially
cripples the system during arc_prune floods for example.

boot_ncpus is the number of working CPUs when called so use
that instead.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: DHE <git@dehacked.net>
Closes #10282
2020-05-20 10:07:21 -07:00
Paul Dagnelie de4f06c275
Small program that converts a dataset id and an object id to a path
Small program that converts a dataset id and an object id to a path

Reviewed-by: Prakash Surya <prakash.surya@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #10204
2020-05-20 10:05:33 -07:00
Ryan Moeller 7b0e39030c
freebsd: Correct the order of arguments to copyin() for Q_SETQUOTA
Sponsored by: DARPA
External-issue: https://reviews.freebsd.org/D24656
FreeBSD-commit: freebsd/freebsd@a431c095d3

Authored by: jhb <jhb@FreeBSD.org>
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Ported-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #10344
2020-05-19 16:45:25 -07:00
George Amanakis 7cd723e685
Fix gcc 10.1 stringop-truncation error
As we do not expect the destination of these strncpy calls to be NULL
terminated, substitute them with memcpy.

Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Amanakis <gamanakis@gmail.com>
Closes #10346
2020-05-19 14:24:10 -07:00
Kyle Evans 5b090d57d4
freebsd: return EISDIR for read(2) on directories
This is arguably a change for internal consistency within OpenZFS, as the
Linux implementation will reject read(2) on directories with EISDIR. It's
not unreasonable for read(2) to do something here on FreeBSD, but we don't
currently copy out anything useful anyways so start rejecting it with the
appropriate error.

Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Kyle Evans <kevans@FreeBSD.org>
Closes #10338
2020-05-16 10:12:01 -07:00
Ryan Moeller d2782af461
Fix ZVOL_DIR
We only use ZVOL_DIR on FreeBSD, and on FreeBSD it isn't correct.

Move the definition to the file where it is needed, and define it as
/dev/zvol/.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #10337
2020-05-16 10:10:38 -07:00
ColMelvin 4d6043f2b7
RPM: Remove old versions of DKMS on upgrade
Due to a mismatch between the text and a regex looking for that text,
the `%preuninstall` script would never run the `dkms remove` command
necessary to avoid corrupting the DKMS data configuration.  Increase
regex specificity to avoid this issue.

Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Chris Lindee <chris.lindee+github@gmail.com>
Closes: #9891
Closes #10327
2020-05-14 20:51:33 -07:00
Matthew Ahrens 1b9cd1a9d9
Fix error handling in receive_writer_thread()
If `receive_writer_thread()` gets an error from `receive_process_record()`,
it should be saved in `rwa->err` so that we will stop processing records,
and the main thread will notice that the receive has failed.

When an error is first encountered, this happens correctly.  However, if
there are more records to dequeue, the next time through the loop we
will reset `rwa->err` to zero, allowing us to try to process the
following record (2 after the failed record).  Depending on what types
of records remain, we may incorrectly complete the receive
"successfully", but without actually having processed all the records.

The fix is to only set `rwa->err` if we got a *non-zero* error.

This bug was introduced by #10099 "Improve zfs receive performance by
batching writes".

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #10320
2020-05-14 20:48:29 -07:00
yparitcher cdcce2f019
Fix VN_OPEN_INVFS typo
The VN_OPEN_INVFS literal is in the wrong field.

Reviewed-by: Matt Macy <mmacy@FreeBSD.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Signed-off-by: yparitcher <y@paritcher.com>
Closes #10322
2020-05-14 20:47:14 -07:00
Brian Behlendorf 2ade659eb4
Fix abd_enter/exit_critical wrappers
Commit fc551d7 introduced the wrappers abd_enter_critical() and
abd_exit_critical() to mark critical sections.  On Linux these are
implemented with the local_irq_save() and local_irq_restore() macros
which set the 'flags' argument when saving.  By wrapping them with
a function the local variable is no longer set by the macro and is
no longer properly restored.

Convert abd_enter_critical() and abd_exit_critical() to macros to
resolve this issue and ensure the flags are properly restored.

Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Atkinson <batkinson@lanl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10332
2020-05-14 20:45:16 -07:00
Jorgen Lundman eeb8fae9c7
Upstream: add missing thread_exit()
Undo FreeBSD wrapper for thread_create() added to call thread_exit.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: Jorgen Lundman <lundman@lundman.net>
Closes #10314
2020-05-14 15:58:09 -07:00
Matthew Ahrens 8b240f14f9
remove unneeded member drc_err of dmu_recv_cookie_t
The member drc_err of dmu_recv_cookie_t is used only locally in
receive_read, so we can replace it with a local variable.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #10319
2020-05-14 12:10:29 -07:00
Brian Behlendorf c87f958668
flake8 E741 variable name warning
Update the zts-report.py script to conform to the flake8 E741 rule.

    "Variables named I, O, and l can be very hard to read. This is
    because the letter I and the letter l are easily confused, and
    the letter O and the number 0 can be easily confused."

- https://www.flake8rules.com/rules/E741.html

Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10323
2020-05-14 09:41:29 -07:00
John Wren Kennedy e1fcd940e7
ZTS: zpool_split_indirect deletes zfstest log file
The cleanup routine for this test attempts to remove some temporary
files with `rm -f $VDEV_*`, but VDEV_ is undefined. As a result, all
files in the current working directory (/var/tmp/test_results/current)
get removed instead. This includes the complete log file of all tests.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: George Amanakis <gamanakis@gmail.com>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Signed-off-by: John Kennedy <john.kennedy@delphix.com>
Closes #10324
2020-05-14 09:39:47 -07:00
John Poduska 41035a0496
Resilver restarts unnecessarily when it encounters errors
When a resilver finishes, vdev_dtl_reassess is called to hopefully
excise DTL_MISSING (amongst other things). If there are errors during
the resilver, they are tracked in DTL_SCRUB, as spelled out in the
block comment in vdev.c. DTL_SCRUB is in-core only, so it can only
be used if the pool was online for the whole resilver. This state is
tracked with the spa_scrub_started flag, which only gets set when
the scan is initialized. Unfortunately, this flag gets cleared right
before vdev_dtl_reassess gets called, so if there are any errors
during the scan, DTL_MISSING will never get excised and the resilver
will just continually restart. This fix simply moves clearing that
flag until after the call to vdev_dtl_reasses.

In addition, if a pool is imported and already has scn_errors > 0,
this change will restart the resilver immediately instead of doing
the rest of the scan and then restarting it from the beginning. On
the other hand, if scn_errors == 0 at import, then no errors have
been encountered so far, so the spa_scrub_started flag can be safely
set.

A test has been added to verify that resilver does not restart when
relevant DTL's are available.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Paul Zuchowski <pzuchowski@datto.com>
Signed-off-by: John Poduska <jpoduska@datto.com>
Closes #10291
2020-05-13 10:54:27 -07:00
AJ Jordan b29e31d80d Fix outdated comment header
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: AJ Jordan <alex@strugee.net>
Closes #10288
2020-05-11 16:23:16 -07:00
AJ Jordan 5a04b17717 Fix up arcstat(1) to match our version
Turns out the illumos manpage, which is what this originates from, was
written for the original Perl version of the utility which is not the
version in the OpenZFS tree. *That* version originates from a Python
rewrite that was done for FreeNAS. So fix up the manpage to match what
we actually ship (and fix a few typos in the process).

Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: AJ Jordan <alex@strugee.net>
Closes #10288
2020-05-11 16:22:48 -07:00
AJ Jordan ac806a2559 Import the arcstat(1m) manpage from illumos
And move it from section 1m to section 1 for consistency.

Imported from illumos commit f34d737f.

Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: AJ Jordan <alex@strugee.net>
Closes #10288
2020-05-11 16:22:18 -07:00
AJ Jordan 2b21da4f76 Fix inconsistent capitalization in `arcstat -v`
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: AJ Jordan <alex@strugee.net>
Closes #10288
2020-05-11 16:21:08 -07:00
Richard Laager 7fcf82451c
Change zfsunlock for better busybox compatibility
It turns out that there are two versions of Busybox, at least on Ubuntu
18.04.  If you have the busybox-static package installed, you get a
busybox that supports `ps a` and `head`.  If you only have
busybox-initramfs, you don't.  Either way, you have `awk`.

This change should also make this compatible with GNU ps, if you somehow
end up with that in the initramfs environment.

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Andrey Prokopenko <job@terem.fr>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #10307
2020-05-10 12:26:08 -07:00
Brian Atkinson fc551d7efb
Combine OS-independent ABD Code into Common Source File
Reorganizing ABD code base so OS-independent ABD code has been placed
into a common abd.c file. OS-dependent ABD code has been left in each
OS's ABD source files, and these source files have been renamed to
abd_os.

The OS-independent ABD code is now under:
module/zfs/abd.c
With the OS-dependent code in:
module/os/linux/zfs/abd_os.c
module/os/freebsd/zfs/abd_os.c

Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Brian Atkinson <batkinson@lanl.gov>
Closes #10293
2020-05-10 12:23:52 -07:00
Petros Koutoupis bd95f00d4b
Fixed LDADD library links in Makefiles for cross compilation builds
When building on native dev system, there are no issues but when
cross-compiling for target system, some linker errors are observed.
The only way to avoid these errors is by adjusting the Makefile.am
of those various components to add the library dependencies.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Petros Koutoupis <petros@petroskoutoupis.com>
Closes #10304
2020-05-09 10:17:08 -07:00
Brian Behlendorf d775c86dd4
ZTS: refreserv_005_pos.ksh
When recursively destroying the dataset it's possible for the
dataset volume to be open by an unrelated process, like blkid.
Use the destroy_dataset() which will retry when this occurs.

Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #10305
2020-05-08 13:50:02 -07:00
Andrey Prokopenko 1cc635a2dd Unlock encrypted root partition over SSH
This commit add a new feature for Debian-based distributions to unlock
encrypted root partition over SSH.  This feature is very handy on
headless NAS or VPS cloud servers.  To use this feature, you will need
to install the dropbear-initramfs package.

Reviewed-By: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-By: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Andrey Prokopenko <job@terem.fr>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #10027
2020-05-07 16:41:16 -07:00
Richard Laager 746d22ee02 Rework README.initramfs.markdown
This file is listed as being in Markdown format, but it didn't really
use much Markdown.  I have added a fair amount of formatting.

I have reordered and reworded things to improve the flow of the text.

Reviewed-By: Andrey Prokopenko <job@terem.fr>
Reviewed-By: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-By: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #10027
2020-05-07 16:40:00 -07:00
Richard Laager 4cfd339ce4 Cleanup contrib/initramfs automake
The initramfs hook scripts depend on Makefile.  This way, if the
substitution code is changed, they should update.  This brings it in
line with etc/init.d (which was modified to match the example in the
automake docs).

The initramfs hook script cleaning now matches etc/init.d.

There was a mix of SUBDIRS recursion and custom install rules for files
in subdirectories.  This was duplicated for the "hooks" and "scripts"
subdirectories.  Now everything uses SUBDIRS.

I fixed the substitution of DEFAULT_INITCONF_DIR for hooks/zfs.

Reviewed-By: Andrey Prokopenko <job@terem.fr>
Reviewed-By: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-By: Tom Caputi <tcaputi@datto.com>
Signed-off-by: Richard Laager <rlaager@wiktel.com>
Closes #10027
2020-05-07 16:39:08 -07:00
George Amanakis 657fd33bcf
Improvements on persistent L2ARC
Functional changes:

We implement refcounts of log blocks and their aligned size on the
cache device along with two corresponding arcstats. The refcounts are
reflected in the header of the device and provide valuable information
as to whether log blocks are accounted for correctly. These are
dynamically adjusted as log blocks are committed/evicted. zdb also uses
this information in the device header and compares it to the
corresponding values as reported by dump_l2arc_log_blocks() which
emulates l2arc_rebuild(). If the refcounts saved in the device header
report higher values, zdb exits with an error. For this feature to work
correctly there should be no active writes on the device. This is also
employed in the tests of persistent L2ARC. We extend the structure of
the cache device header by adding the two new variables mirroring the
refcounts after the existing variables to preserve backward
compatibility in terms of persistent L2ARC.

1) a new arcstat "l2_log_blk_asize" and refcount "l2ad_lb_asize" which
   reflect the total aligned size of log blocks on the device. This is
   also reflected in the header of the cache device as "dh_lb_asize".
2) a new arcstat "l2arc_log_blk_count" and refcount "l2ad_lb_count"
   which reflect the total number of L2ARC log blocks present on cache
   devices.  It is also reflected in the header of the cache device as
   "dh_lb_count".

In l2arc_rebuild_vdev() if the amount of committed log entries in a log
block is 0 and the device header is valid we update the device header.
This will facilitate trimming of the whole device in this case when
TRIM for L2ARC is implemented.

Improve loop protection in l2arc_rebuild() by using the starting offset
of the payload of each log block instead of the starting offset of the
log block.

If the zio in l2arc_write_buffers() fails, restore the lbps array in the
header of the device to its previous state in l2arc_write_done().

If l2arc_rebuild() ends the rebuild process without restoring any L2ARC
log blocks in ARC and without any other error, this means that the lbps
array in the header is pointing to non-existent or invalid log blocks.
Reset the device header in this case.

In l2arc_rebuild() change the zfs_dbgmsg messages to
spa_history_log_internal() making them user visible with zpool history
command.

Non-functional changes:

Make the first test in persistent L2ARC use `zdb -lll` to increase
coverage in `zdb.c`.

Rename psize with asize when referring to log blocks, since
L2ARC_SET_PSIZE stores the vdev aligned size for log blocks. Also
rename dh_log_blk_entries to dh_log_entries to make it clear that
it is a mirror of l2ad_log_entries. Added comments for both changes.

Fix inaccurate comments for example in l2arc_log_blk_restore().

Add asserts at the end in l2arc_evict() and l2arc_write_buffers().

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Amanakis <gamanakis@gmail.com>
Closes #10228
2020-05-07 16:34:03 -07:00
Paul Dagnelie 108a454a46
Add support for boot environment data to be stored in the label
Modern bootloaders leverage data stored in the root filesystem to 
enable some of their powerful features. GRUB specifically has a grubenv 
file which can store large amounts of configuration data that can be 
read and written at boot time and during normal operation. This allows 
sysadmins to configure useful features like automated failover after 
failed boot attempts. Unfortunately, due to the Copy-on-Write nature 
of ZFS, the standard behavior of these tools cannot handle writing to
ZFS files safely at boot time. We need an alternative way to store 
data that allows the bootloader to make changes to the data.

This work is very similar to work that was done on Illumos to enable 
similar functionality in the FreeBSD bootloader. This patch is different 
in that the data being stored is a raw grubenv file; this file can store 
arbitrary variables and values, and the scripting provided by grub is 
powerful enough that special structures are not required to implement 
advanced behavior.

We repurpose the second padding area in each label to store the grubenv 
file, protected by an embedded checksum. We add two ioctls to get and 
set this data, and libzfs_core and libzfs functions to access them more 
easily. There are no direct command line interfaces to these functions; 
these will be added directly to the bootloader utilities.

Reviewed-by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #10009
2020-05-07 09:36:33 -07:00
Philip Pokorny a36bad1759
Fix column width calculation issue with certain terminal widths
If the reported terminal width is 0 or less than 42, the signed variable
width was set to a negative number that was then assigned to the
unsigned column width becoming a huge number.

Add comments and change logic to better explain what's happening.

Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Philip Pokorny <ppokorny@mindspring.com>
Closes #10247
2020-05-06 17:17:38 -07:00