3581 spa_zio_taskq[ZIO_TYPE_FREE][ZIO_TASKQ_ISSUE]->tq_lock is piping hot
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Gordon Ross <gordon.ross@nexenta.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
References:
illumos/illumos-gate@ec94d32https://illumos.org/issues/3581
Notes for Linux port:
Earlier commit 08d08eb reduced contention on this taskq lock by simply
reducing the number of z_fr_iss threads from 100 to one-per-CPU. We
also optimized the taskq implementation in zfsonlinux/spl@3c6ed54.
These changes significantly improved unlink performance to acceptable
levels.
This patch further reduces time spent spinning on this lock by
randomly dispatching the work items over multiple independent task
queues. The Illumos ZFS developers stated that this lock contention
only arose after "3329 spa_sync() spends 10-20% of its time in
spa_free_sync_cb()" was landed. It's not clear if 3329 affects the
Linux port or not. I didn't see spa_free_sync_cb() show up in
oprofile sessions while unlinking large files, but I may just not
have used the right test case.
I tested unlinking a 1 TB of data with and without the patch and
didn't observe a meaningful difference in elapsed time. However,
oprofile showed that the percent time spent in taskq_thread() was
reduced from about 16% to about 5%. Aside from a possible slight
performance benefit this may be worth landing if only for the sake of
maintaining consistency with upstream.
Ported-by: Ned Bass <bass6@llnl.gov>
Closes#1327
3329 spa_sync() spends 10-20% of its time in spa_free_sync_cb()
3330 space_seg_t should have its own kmem_cache
3331 deferred frees should happen after sync_pass 1
3335 make SYNC_PASS_* constants tunable
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Matt Ahrens <matthew.ahrens@delphix.com>
Reviewed by: Christopher Siden <chris.siden@delphix.com>
Reviewed by: Eric Schrock <eric.schrock@delphix.com>
Reviewed by: Richard Lowe <richlowe@richlowe.net>
Reviewed by: Dan McDonald <danmcd@nexenta.com>
Approved by: Eric Schrock <eric.schrock@delphix.com>
References:
illumos/illumos-gate@01f55e48fbhttps://www.illumos.org/issues/3329https://www.illumos.org/issues/3330https://www.illumos.org/issues/3331https://www.illumos.org/issues/3335
Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
3306 zdb should be able to issue reads in parallel
3321 'zpool reopen' command should be documented in the man
page and help
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Matt Ahrens <matthew.ahrens@delphix.com>
Reviewed by: Christopher Siden <chris.siden@delphix.com>
Approved by: Garrett D'Amore <garrett@damore.org>
References:
illumos/illumos-gate@31d7e8fa33https://www.illumos.org/issues/3306https://www.illumos.org/issues/3321
The vdev_file.c implementation in this patch diverges significantly
from the upstream version. For consistenty with the vdev_disk.c
code the upstream version leverages the Illumos bio interfaces.
This makes sense for Illumos but not for ZoL for two reasons.
1) The vdev_disk.c code in ZoL has been rewritten to use the
Linux block device interfaces which differ significantly
from those in Illumos. Therefore, updating the vdev_file.c
to use the Illumos interfaces doesn't get you consistency
with vdev_disk.c.
2) Using the upstream patch as is would requiring implementing
compatibility code for those Solaris block device interfaces
in user and kernel space. That additional complexity could
lead to confusion and doesn't buy us anything.
For these reasons I've opted to simply move the existing vn_rdwr()
as is in to the taskq function. This has the advantage of being
low risk and easy to understand. Moving the vn_rdwr() function
in to its own taskq thread also neatly avoids the possibility of
a stack overflow.
Finally, because of the additional work which is being handled by
the free taskq the number of threads has been increased. The
thread count under Illumos defaults to 100 but was decreased to 2
in commit 08d08e due to contention. We increase it to 8 until
the contention can be address by porting Illumos #3581.
Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1354
The previous code was only waiting for the symver file. But the
postinst target of the DKMS script for SPL will not only create
the symvers file, but also the header spl_config.h.
If we are waiting in the configure script of ZFS for the SPL
symvers file, then we also need to wait for spl_config.h.
Otherwise the configure script will abort because the spl_config.h
is not yet available.
On top of that, the function ZFS_AC_SPL_MODULE_SYMVERS is moved
to the end of the function ZFS_AC_SPL to allow both checks share
the with-spl-timeout parameter.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1431
Recent changes have caused older versions of gcc to mistakenly
flag 'old_umask' in vn_open() as an unitialized variable. To
silence the warning initialize it.
kernel.c: In function 'vn_open':
kernel.c:525:6: error: 'old_umask' may be used uninitialized
in this function
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
The zfs_fd must be opened before calling print_all_handlers() or
the ioctl() cannot be used to the zfs control device. This brings
the zinject code back in sync with the Illumos implementation.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Matt Ahrens <matthew.ahrens@delphix.com>
Reviewed by: Eric Schrock <eric.schrock@delphix.com>
Reviewed by: Christopher Siden <chris.siden@delphix.com>
Approved by: Garrett D'Amore <garrett@damore.org>
NOTES: This patch has been reworked from the original in the
following ways to accomidate Linux ZFS implementation
*) Usage of the cyclic interface was replaced by the delayed taskq
interface. This avoids the need to implement new compatibility
code and allows us to rely on the existing taskq implementation.
*) An extern for zfs_txg_synctime_ms was added to sys/dsl_pool.h
because declaring externs in source files as was done in the
original patch is just plain wrong.
*) Instead of panicing the system when the deadman triggers a
zevent describing the blocked vdev and the first pending I/O
is posted. If the panic behavior is desired Linux provides
other generic methods to panic the system when threads are
observed to hang.
*) For reference, to delay zios by 30 seconds for testing you can
use zinject as follows: 'zinject -d <vdev> -D30 <pool>'
References:
illumos/illumos-gate@283b84606bhttps://www.illumos.org/issues/3246
Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1396
A deadlock was accidentally introduced by commit e95853a which
can occur when the system is under memory pressure. What happens
is that while the txg_quiesce thread is holding the tx->tx_cpu
locks it enters memory reclaim. In the context of this memory
reclaim it then issues synchronous I/O to a ZVOL swap device.
Because the txg_quiesce thread is holding the tx->tx_cpu locks
a new txg cannot be opened to handle the I/O. Deadlock.
The fix is straight forward. Move the memory allocation outside
the critical region where the tx->tx_cpu locks are held. And for
good measure change the offending allocation to KM_PUSHPAGE to
ensure it never attempts to issue I/O during reclaim.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #1274
These are build products and should be ignored.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Turbo Fredriksson <turbo@bayour.com>
Issue #1402
When the kmod packaging was introduced the ability to pass the
--enable-debug and --enable-dmu-tx options from configure all
the way through to `make rpm|deb` was accidenally lost. Update
ZFS_AC_RPM to explicitlu set RPM_DEFINE_COMMON with these
rpmbuild defines.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #1402
Preserve the release field when creating Debian packages. The
--keep-version option was not used because it results in a failure
when the git '<commit>_<hash>' syntax is used for the release.
The '_' is a valid character for RPM packages but not for DEBs.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Turbo Fredriksson <turbo@bayour.com>
Issue #1402
Issue #928
When building a custom release in a git tree provide the ability
to prevent the release field from being overwritten by the
`git describe` output.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #1402
There are a number of issues with the generic kmod RPM spec in its
current state:
- The "%{__id_u}" macro seems to not be available on some systems (e.g.
Debian squeeze). It appears it has been deprecated. Use "${__id} -u"
instead.
- The way the "--with-linux=" configure option is generated in the
non-RHEL/Fedora case is completely wrong with various newline and
escaping issues (also, $kernel_version is not available in the
generator context).
The second issue made the generator shell snippet (almost) silently
fail, which under specific circumstances can result in broken builds
against the wrong kernel sources.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1416
According to the getxattr(2) man page the ERANGE errno should be
returned when the size of the value buffer is to small to hold the
result. Prior to this patch the implementation would just truncate
the value to size bytes.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1408
The zpl_readdir() function shouldn't be registered as part of
the zpl_file_operations table, it must only be part of the
zpl_dir_file_operations table. By removing this callback
the VFS will now correctly return ENOTDIR when calling
getdents() on a file.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1404
Previous patches have allowed you to set an increased ashift to
avoid doing 512b IO with 4k sector devices. However, it was not
possible to set the ashift lower than the reported physical sector
size even when a smaller logical size was supported. In practice,
there are several cases where settong a lower ashift is useful:
* Most modern drives now correctly report their physical sector
size as 4k. This causes zfs to correctly default to using a 4k
sector size (ashift=12). However, for some usage models this
new default ashift value causes an unacceptable increase in
space usage. Filesystems with many small files may see the
total available space reduced to 30-40% which is unacceptable.
* When replacing a drive in an existing pool which was created
with ashift=9 a modern 4k sector drive cannot be used. The
'zpool replace' command will issue an error that the new drive
has an 'incompatible sector alignment'. However, by allowing
the ashift to be manual specified as smaller, non-optimal,
value the device may still be safely used.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1381Closes#1328
Issue #967
Issue #548
3422 zpool create/syseventd race yield non-importable pool
3425 first write to a new zvol can fail with EFBIG
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Garrett D'Amore <garrett@damore.org>
References:
illumos/illumos-gate@bda8819455https://www.illumos.org/issues/3422https://www.illumos.org/issues/3425
Ported-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1390
For the DKMS package to successfully build the kernel-devel
headers must be included along gcc, make, and perl. The ZFS
code never directly invokes perl but the kernel build system
depends on it.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1380
The only remaining perl dependency is part of the ZFS_AC_META macro.
By eliminating this and replacing it with awk we can avoid the need
to pull in perl to rebuild the packages.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #1380
Commit f6fb7651a0 introduced the idea
of working builds which work correctly. However, because the zfs-kmod
depends on a specific 'spl-devel-kmod = {version}-%{release}' package
and the release component is unique the dependency is never satisfied.
This requires line was introduced to ensure the correct version of the
spl is always used. In this context only the version number is required
so the release component has been dropped to satisfy the dependency.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Part of the automated testing involves building the source on Debian Lenny
which ships an ancient version of automake (1.10.1). Historically, this
has caused a non-fatal warning about AM_SILENT_RULES not being defined.
But when the autogen.sh script was updated to use autoreconf the warning
became fatal.
configure.ac:31: warning: macro `AM_SILENT_RULES' not found in library
autoreconf: running: /usr/bin/autoconf --force
configure.ac:34: error: possibly undefined macro: AM_SILENT_RULES
If this token and others are legitimate, please use m4_pattern_allow.
To resolve this build issue the call to AM_SILENT_RULES has been wrapped
by m4_ifdef(). This prevents the macro from being expanded on platforms
where it's undefined.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
-D and -I are preprocessor flags, so should preferably be in the
appropriate variable.
Signed-off-by: Jan Engelhardt <jengelh@inai.de>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
* Update manpage with more information about the ACL, guest access
and that samba needs to be able to authenticate user(s).
* Add information that 'net' can be used to modify the share after
ZFS sharing and that it will be undone with a 'zfs unshare'.
* Give an example on how to mount a SMB filesystem shared via ZFS.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1181
Issue #1170
Do basic sanity checks in smb_available() to verify that the 'net'
command and the usershare directory exists.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1124
The distribution specific init scripts where excluded from the
packaging when it was reworked. The intention is to replace
them with systemd equivilants. However, that work has not yet
been done and the init scripts are still useful so they have
been added back in to the packaging.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
In order to ensure that yum-builddep pulls in all the build
requirements a generic ${kmodname}-devel-kmod provides line is
added. This allows a version of the development headers to be
included without requiring knowledge of the kernel version.
This is important because unlike rpmbuild which does correctly
expand the source rpm spec file, yum-builddep does not. Without
this generic provides line mock which relies on yum-builddep is
unable to automatically satisfy the dependency.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
When building from an arbitrary commit in the git tree it's useful
for the resulting packages to be uniquely identifiable. Therefore,
the build system has been updated to detect if your compiling in
git tree.
If you are building in a git tree, and there are commits after the
last annotated tag. Then the <id>-<hash> component of 'git describe'
will be used to overwrite the 'Release:' field in the META file.
The only tricky part is that to ensure the 'make dist' tarball is
built using the correct release. A dist-hook was added to the top
level make file to rewrite the META file using the correct release.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Correct spelling mistakes in the AUTHORS and DISCLAIMER files, and
update the README.markdown file to credit Illumos and mention that
the ZPL is finished.
The README.markdown file is also the first impression for a handful
of new users that discover ZoL through a web search because it
doubles as the splash page for the Github repository. The build blurbs
are therefore removed because these people should be encouraged to
visit the regular home page before installing the product.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1366
The --with-linux and --with-linux-obj options must be specified
as part of the dkms build otherwise the package will be built
against the running kernel.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Support was added to dkms so build dependencies can be specified.
This allows us to ensure that the spl package will always be built
before the zfs package. Those patches have not yet been merged
upstream but they are available in the zfsonlinux/dkms repository.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Because the zfs-dkms package also provides zfs-kmod for the
zfs user package yum flags this as a conflict. To avoid the
problem remove the Conflicts tag from zfs-dkms and just rely
on the one in zfs-kmod.
zfs-dkms-0.6.0-rc14.fc18.noarch has installed conflicts
zfs-kmod: zfs-dkms-0.6.0-rc14.fc18.noarch
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
60-zpool.rules was retired some time ago in favor of 69-vdev.rules.
Additionally, there is no guarentee a zpool.cache file exists so
only install it conditionally upon its existance.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
The spl, zfs-test, and zfs-dracut packages should be pulled in
by the core zfs package. This allows all the required zfs packages
to be installed from a yum repository by running:
yum install zfs
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Adding the 'spl-dkms = VERSION' dependency to the zfs-dkms
package ensures the spl will be installed before zfs. This
cleanly handles the initial 'yum localinstall' case.
However, this does not address the dkms rebuilds caused by
a new kernel being installed. For that we still rely on the
clunky --with-spl-timeout=<timeout> configure option which
allows the zfs compilation to be briefly delayed while the
spl components are built.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
The automake templates have been updated to install them,
and the existing packaging will automatically include them.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Supplements #518
The assumption in zio_ddt_free() is that ddt_phys_select() must
always find a match. However, if that fails due to a damaged
DDT or some other reason the code will NULL dereference in
ddt_phys_decref().
While this should never happen it has been observed on various
platforms. The result is that unless your willing to patch the
ZFS code the pool is inaccessible. Therefore, we're choosing
to more gracefully handle this case rather than leave it fatal.
http://mail.opensolaris.org/pipermail/zfs-discuss/2012-February/050972.html
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1308
Enabling metaslab debugging will prevent space maps from being
automatically unloaded. This can significantly increase the
memory footprint but being able to dynamically control this is
helpful for debugging and certain performance testing.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Refresh the existing RPM packaging to conform to the 'Fedora
Packaging Guidelines'. This includes adopting the kmods2
packaging standard which is used fod kmods distributed by
rpmfusion for Fedora/RHEL.
http://fedoraproject.org/wiki/Packaging:Guidelineshttp://rpmfusion.org/Packaging/KernelModules/Kmods2
While the spec files have been entirely rewritten from a
user perspective the only major changes are:
* The Fedora packages now have a build dependency on the
rpmfusion repositories. The generic kmod packages also
have a new dependency on kmodtool-1.22 but it is bundled
with the source rpm so no additional packages are needed.
* The kernel binary module packages have been renamed from
zfs-modules-* to kmod-zfs-* as specificed by kmods2.
* The is now a common kmod-zfs-devel-* package in addition
to the per-kernel devel packages. The common package
contains the development headers while the per-kernel
package contains kernel specific build products.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes#1341
Because the install location for the spl/zfs-devel headers was
changed we need to refresh the auto-detect code. Note that
for packaging which already explicitly calls --with-spl{-obj}
nothing has changed.
The updated code is now structured like that in ZFS_AC_KERNEL
and should be cleaner and easier to maintain. In addition,
it's stricter about detecting a valid source and object
directory. It requires:
* The source directory contains the file 'spl.release'
* The object directory contains the file 'spl_config.h'
* The following paths will be checked. Notice the /var/lib/
and /usr/src paths require that the spl and zfs version be
matched. This is done to prevent accidentally mixing releases.
dnl # 1) /var/lib/dkms/spl/<version>/build
dnl # 2) /usr/src/spl-<version>/<kernel-version>
dnl # 3) /usr/src/spl-<version>
dnl # 4) ../spl
dnl # 5) /usr/src/kernels/<kernel-version>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Install the common zfs kernel development headers under
/usr/src/zfs-<version>/ rather than in a kernel specific
directory. The kernel specific build products such as
zfs_config.h and Modules.symvers are left installed under
/usr/src/zfs-<version>/<kernel>.
This was done to be consistent with where dkms expects
kernel module source to be packaged. It also allows for
a common zfs-kmod-devel package which includes the headers,
and per-kernel zfs-kmod-devel-<kernel> packages.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
There's no reason to keep this file around it's redundant with
the COPYRIGHT and OPENSOLARIS.LICENSE files and causes lintian
to emit an extra-license-file warning.
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Adam Leventhal
George Wilson
Carlos Alberto Lopez Perez
Brian Behlendorf
Turbo Fredriksson
Etienne Dechamps
Chris Dunlop
Caleb James DeLisle
Martin Matuska
Jan Engelhardt
Nick Garvey
Darik Horn
Michael Gebetsroither