2008-12-11 19:14:49 +00:00
|
|
|
/*
|
|
|
|
|
* CDDL HEADER START
|
|
|
|
|
*
|
|
|
|
|
* The contents of this file are subject to the terms of the
|
|
|
|
|
* Common Development and Distribution License (the "License").
|
|
|
|
|
* You may not use this file except in compliance with the License.
|
|
|
|
|
*
|
|
|
|
|
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
|
|
|
|
|
* or http://www.opensolaris.org/os/licensing.
|
|
|
|
|
* See the License for the specific language governing permissions
|
|
|
|
|
* and limitations under the License.
|
|
|
|
|
*
|
|
|
|
|
* When distributing Covered Code, include this CDDL HEADER in each
|
|
|
|
|
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
|
|
|
|
|
* If applicable, add the following below this CDDL HEADER, with the
|
|
|
|
|
* fields enclosed by brackets "[]" replaced with your own identifying
|
|
|
|
|
* information: Portions Copyright [yyyy] [name of copyright owner]
|
|
|
|
|
*
|
|
|
|
|
* CDDL HEADER END
|
|
|
|
|
*/
|
|
|
|
|
/*
|
2010-09-04 20:26:23 +00:00
|
|
|
* Copyright 2009 Sun Microsystems, Inc. All rights reserved.
|
|
|
|
|
* Use is subject to license terms.
|
2008-12-11 19:14:49 +00:00
|
|
|
*/
|
2011-11-08 00:26:52 +00:00
|
|
|
/*
|
|
|
|
|
* Copyright 2011 Nexenta Systems, Inc. All rights reserved.
|
2012-05-09 22:05:14 +00:00
|
|
|
* Copyright (c) 2012, Joyent, Inc. All rights reserved.
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
* Copyright (c) 2013 by Delphix. All rights reserved.
|
2011-11-08 00:26:52 +00:00
|
|
|
*/
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
#ifndef _SYS_ZFS_CONTEXT_H
|
|
|
|
|
#define _SYS_ZFS_CONTEXT_H
|
|
|
|
|
|
2010-09-10 16:36:39 +00:00
|
|
|
#ifdef __KERNEL__
|
2010-09-04 20:26:23 +00:00
|
|
|
|
|
|
|
|
#include <sys/note.h>
|
|
|
|
|
#include <sys/types.h>
|
|
|
|
|
#include <sys/t_lock.h>
|
|
|
|
|
#include <sys/atomic.h>
|
|
|
|
|
#include <sys/sysmacros.h>
|
|
|
|
|
#include <sys/bitmap.h>
|
|
|
|
|
#include <sys/cmn_err.h>
|
|
|
|
|
#include <sys/kmem.h>
|
|
|
|
|
#include <sys/taskq.h>
|
|
|
|
|
#include <sys/buf.h>
|
|
|
|
|
#include <sys/param.h>
|
|
|
|
|
#include <sys/systm.h>
|
|
|
|
|
#include <sys/cpuvar.h>
|
|
|
|
|
#include <sys/kobj.h>
|
|
|
|
|
#include <sys/conf.h>
|
|
|
|
|
#include <sys/disp.h>
|
|
|
|
|
#include <sys/debug.h>
|
|
|
|
|
#include <sys/random.h>
|
|
|
|
|
#include <sys/byteorder.h>
|
|
|
|
|
#include <sys/systm.h>
|
|
|
|
|
#include <sys/list.h>
|
2010-12-22 20:13:57 +00:00
|
|
|
#include <sys/uio_impl.h>
|
2010-09-04 20:26:23 +00:00
|
|
|
#include <sys/dirent.h>
|
|
|
|
|
#include <sys/time.h>
|
|
|
|
|
#include <vm/seg_kmem.h>
|
|
|
|
|
#include <sys/zone.h>
|
2013-03-08 18:41:28 +00:00
|
|
|
#include <sys/sdt.h>
|
2010-09-04 20:26:23 +00:00
|
|
|
#include <sys/zfs_debug.h>
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
#include <sys/zfs_delay.h>
|
2010-09-04 20:26:23 +00:00
|
|
|
#include <sys/fm/fs/zfs.h>
|
|
|
|
|
#include <sys/sunddi.h>
|
2011-11-15 19:01:27 +00:00
|
|
|
#include <sys/ctype.h>
|
2012-08-24 00:45:31 +00:00
|
|
|
#include <sys/disp.h>
|
2011-01-18 19:42:25 +00:00
|
|
|
#include <linux/dcache_compat.h>
|
2010-09-04 20:26:23 +00:00
|
|
|
|
|
|
|
|
#else /* _KERNEL */
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
#define _SYS_MUTEX_H
|
|
|
|
|
#define _SYS_RWLOCK_H
|
|
|
|
|
#define _SYS_CONDVAR_H
|
|
|
|
|
#define _SYS_SYSTM_H
|
|
|
|
|
#define _SYS_T_LOCK_H
|
|
|
|
|
#define _SYS_VNODE_H
|
|
|
|
|
#define _SYS_VFS_H
|
|
|
|
|
#define _SYS_SUNDDI_H
|
|
|
|
|
#define _SYS_CALLB_H
|
|
|
|
|
|
|
|
|
|
#include <stdio.h>
|
|
|
|
|
#include <stdlib.h>
|
|
|
|
|
#include <stddef.h>
|
|
|
|
|
#include <stdarg.h>
|
|
|
|
|
#include <fcntl.h>
|
|
|
|
|
#include <unistd.h>
|
|
|
|
|
#include <errno.h>
|
|
|
|
|
#include <string.h>
|
|
|
|
|
#include <strings.h>
|
2010-08-26 17:43:27 +00:00
|
|
|
#include <pthread.h>
|
2008-12-11 19:14:49 +00:00
|
|
|
#include <assert.h>
|
|
|
|
|
#include <alloca.h>
|
|
|
|
|
#include <umem.h>
|
|
|
|
|
#include <limits.h>
|
|
|
|
|
#include <atomic.h>
|
|
|
|
|
#include <dirent.h>
|
|
|
|
|
#include <time.h>
|
2011-11-15 19:01:27 +00:00
|
|
|
#include <ctype.h>
|
2013-05-16 21:18:06 +00:00
|
|
|
#include <signal.h>
|
|
|
|
|
#include <sys/mman.h>
|
2008-12-11 19:14:49 +00:00
|
|
|
#include <sys/note.h>
|
|
|
|
|
#include <sys/types.h>
|
|
|
|
|
#include <sys/cred.h>
|
|
|
|
|
#include <sys/sysmacros.h>
|
|
|
|
|
#include <sys/bitmap.h>
|
|
|
|
|
#include <sys/resource.h>
|
|
|
|
|
#include <sys/byteorder.h>
|
|
|
|
|
#include <sys/list.h>
|
|
|
|
|
#include <sys/uio.h>
|
|
|
|
|
#include <sys/zfs_debug.h>
|
|
|
|
|
#include <sys/sdt.h>
|
|
|
|
|
#include <sys/kstat.h>
|
|
|
|
|
#include <sys/u8_textprep.h>
|
2010-08-26 18:42:43 +00:00
|
|
|
#include <sys/fm/fs/zfs.h>
|
2010-05-28 20:45:14 +00:00
|
|
|
#include <sys/sunddi.h>
|
2013-05-10 21:17:03 +00:00
|
|
|
#include <sys/debug.h>
|
2008-12-11 19:14:49 +00:00
|
|
|
|
2010-08-26 17:58:36 +00:00
|
|
|
/*
|
|
|
|
|
* Stack
|
|
|
|
|
*/
|
|
|
|
|
|
2013-03-08 18:41:28 +00:00
|
|
|
#define noinline __attribute__((noinline))
|
2010-08-26 17:58:36 +00:00
|
|
|
|
2008-12-11 19:14:49 +00:00
|
|
|
/*
|
|
|
|
|
* Debugging
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Note that we are not using the debugging levels.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#define CE_CONT 0 /* continuation */
|
|
|
|
|
#define CE_NOTE 1 /* notice */
|
|
|
|
|
#define CE_WARN 2 /* warning */
|
|
|
|
|
#define CE_PANIC 3 /* panic */
|
|
|
|
|
#define CE_IGNORE 4 /* print nothing */
|
|
|
|
|
|
2010-08-26 17:43:27 +00:00
|
|
|
extern int aok;
|
|
|
|
|
|
2008-12-11 19:14:49 +00:00
|
|
|
/*
|
|
|
|
|
* ZFS debugging
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
extern void dprintf_setup(int *argc, char **argv);
|
2012-01-20 23:02:57 +00:00
|
|
|
extern void __dprintf(const char *file, const char *func,
|
|
|
|
|
int line, const char *fmt, ...);
|
2008-12-11 19:14:49 +00:00
|
|
|
extern void cmn_err(int, const char *, ...);
|
|
|
|
|
extern void vcmn_err(int, const char *, __va_list);
|
|
|
|
|
extern void panic(const char *, ...);
|
|
|
|
|
extern void vpanic(const char *, __va_list);
|
|
|
|
|
|
|
|
|
|
#define fm_panic panic
|
|
|
|
|
|
2013-03-08 18:41:28 +00:00
|
|
|
#ifdef __sun
|
2008-12-11 19:14:49 +00:00
|
|
|
/*
|
|
|
|
|
* DTrace SDT probes have different signatures in userland than they do in
|
|
|
|
|
* kernel. If they're being used in kernel code, re-define them out of
|
|
|
|
|
* existence for their counterparts in libzpool.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#ifdef DTRACE_PROBE
|
|
|
|
|
#undef DTRACE_PROBE
|
|
|
|
|
#endif /* DTRACE_PROBE */
|
2013-03-08 18:41:28 +00:00
|
|
|
#define DTRACE_PROBE(a) \
|
|
|
|
|
ZFS_PROBE0(#a)
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
#ifdef DTRACE_PROBE1
|
|
|
|
|
#undef DTRACE_PROBE1
|
|
|
|
|
#endif /* DTRACE_PROBE1 */
|
2013-03-08 18:41:28 +00:00
|
|
|
#define DTRACE_PROBE1(a, b, c) \
|
|
|
|
|
ZFS_PROBE1(#a, (unsigned long)c)
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
#ifdef DTRACE_PROBE2
|
|
|
|
|
#undef DTRACE_PROBE2
|
|
|
|
|
#endif /* DTRACE_PROBE2 */
|
2013-03-08 18:41:28 +00:00
|
|
|
#define DTRACE_PROBE2(a, b, c, d, e) \
|
|
|
|
|
ZFS_PROBE2(#a, (unsigned long)c, (unsigned long)e)
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
#ifdef DTRACE_PROBE3
|
|
|
|
|
#undef DTRACE_PROBE3
|
|
|
|
|
#endif /* DTRACE_PROBE3 */
|
2013-03-08 18:41:28 +00:00
|
|
|
#define DTRACE_PROBE3(a, b, c, d, e, f, g) \
|
|
|
|
|
ZFS_PROBE3(#a, (unsigned long)c, (unsigned long)e, (unsigned long)g)
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
#ifdef DTRACE_PROBE4
|
|
|
|
|
#undef DTRACE_PROBE4
|
|
|
|
|
#endif /* DTRACE_PROBE4 */
|
2013-03-08 18:41:28 +00:00
|
|
|
#define DTRACE_PROBE4(a, b, c, d, e, f, g, h, i) \
|
|
|
|
|
ZFS_PROBE4(#a, (unsigned long)c, (unsigned long)e, (unsigned long)g, \
|
|
|
|
|
(unsigned long)i)
|
2008-12-11 19:14:49 +00:00
|
|
|
|
2013-03-08 18:41:28 +00:00
|
|
|
/*
|
|
|
|
|
* We use the comma operator so that this macro can be used without much
|
|
|
|
|
* additional code. For example, "return (EINVAL);" becomes
|
|
|
|
|
* "return (SET_ERROR(EINVAL));". Note that the argument will be evaluated
|
|
|
|
|
* twice, so it should not have side effects (e.g. something like:
|
|
|
|
|
* "return (SET_ERROR(log_error(EINVAL, info)));" would log the error twice).
|
|
|
|
|
*/
|
|
|
|
|
#define SET_ERROR(err) (ZFS_SET_ERROR(err), err)
|
|
|
|
|
#else
|
|
|
|
|
#define SET_ERROR(err) (err)
|
|
|
|
|
#endif
|
2008-12-11 19:14:49 +00:00
|
|
|
/*
|
|
|
|
|
* Threads
|
|
|
|
|
*/
|
2010-08-26 17:43:27 +00:00
|
|
|
#define TS_MAGIC 0x72f158ab4261e538ull
|
|
|
|
|
#define TS_RUN 0x00000002
|
|
|
|
|
#ifdef __linux__
|
|
|
|
|
#define STACK_SIZE 8192 /* Linux x86 and amd64 */
|
|
|
|
|
#else
|
|
|
|
|
#define STACK_SIZE 24576 /* Solaris */
|
|
|
|
|
#endif
|
2010-05-28 20:45:14 +00:00
|
|
|
|
|
|
|
|
/* in libzpool, p0 exists only to have its address taken */
|
2010-08-26 17:43:27 +00:00
|
|
|
typedef struct proc {
|
2010-05-28 20:45:14 +00:00
|
|
|
uintptr_t this_is_never_used_dont_dereference_it;
|
2010-08-26 17:43:27 +00:00
|
|
|
} proc_t;
|
2010-05-28 20:45:14 +00:00
|
|
|
|
|
|
|
|
extern struct proc p0;
|
2012-05-09 22:05:14 +00:00
|
|
|
#define curproc (&p0)
|
2010-05-28 20:45:14 +00:00
|
|
|
|
2010-08-26 17:43:27 +00:00
|
|
|
typedef void (*thread_func_t)(void *);
|
|
|
|
|
typedef void (*thread_func_arg_t)(void *);
|
|
|
|
|
typedef pthread_t kt_did_t;
|
|
|
|
|
|
2013-11-01 19:26:11 +00:00
|
|
|
#define kpreempt(x) ((void)0)
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
|
2010-08-26 17:43:27 +00:00
|
|
|
typedef struct kthread {
|
|
|
|
|
kt_did_t t_tid;
|
|
|
|
|
thread_func_t t_func;
|
|
|
|
|
void * t_arg;
|
|
|
|
|
} kthread_t;
|
2008-12-11 19:14:49 +00:00
|
|
|
|
2010-08-26 17:43:27 +00:00
|
|
|
#define curthread zk_thread_current()
|
Add visibility in to arc_read
This change is an attempt to add visibility into the arc_read calls
occurring on a system, in real time. To do this, a list was added to the
in memory SPA data structure for a pool, with each element on the list
corresponding to a call to arc_read. These entries are then exported
through the kstat interface, which can then be interpreted in userspace.
For each arc_read call, the following information is exported:
* A unique identifier (uint64_t)
* The time the entry was added to the list (hrtime_t)
(*not* wall clock time; relative to the other entries on the list)
* The objset ID (uint64_t)
* The object number (uint64_t)
* The indirection level (uint64_t)
* The block ID (uint64_t)
* The name of the function originating the arc_read call (char[24])
* The arc_flags from the arc_read call (uint32_t)
* The PID of the reading thread (pid_t)
* The command or name of thread originating read (char[16])
From this exported information one can see, in real time, exactly what
is being read, what function is generating the read, and whether or not
the read was found to be already cached.
There is still some work to be done, but this should serve as a good
starting point.
Specifically, dbuf_read's are not accounted for in the currently
exported information. Thus, a follow up patch should probably be added
to export these calls that never call into arc_read (they only hit the
dbuf hash table). In addition, it might be nice to create a utility
similar to "arcstat.py" to digest the exported information and display
it in a more readable format. Or perhaps, log the information and allow
for it to be "replayed" at a later time.
Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
2013-09-06 23:09:05 +00:00
|
|
|
#define getcomm() "unknown"
|
2010-08-26 17:43:27 +00:00
|
|
|
#define thread_exit zk_thread_exit
|
|
|
|
|
#define thread_create(stk, stksize, func, arg, len, pp, state, pri) \
|
|
|
|
|
zk_thread_create(stk, stksize, (thread_func_t)func, arg, \
|
2013-03-08 18:41:28 +00:00
|
|
|
len, NULL, state, pri, PTHREAD_CREATE_DETACHED)
|
2010-08-26 17:43:27 +00:00
|
|
|
#define thread_join(t) zk_thread_join(t)
|
2013-03-08 18:41:28 +00:00
|
|
|
#define newproc(f, a, cid, pri, ctp, pid) (ENOSYS)
|
2010-08-26 17:43:27 +00:00
|
|
|
|
|
|
|
|
extern kthread_t *zk_thread_current(void);
|
|
|
|
|
extern void zk_thread_exit(void);
|
|
|
|
|
extern kthread_t *zk_thread_create(caddr_t stk, size_t stksize,
|
|
|
|
|
thread_func_t func, void *arg, size_t len,
|
2012-09-27 11:31:46 +00:00
|
|
|
proc_t *pp, int state, pri_t pri, int detachstate);
|
2010-08-26 17:43:27 +00:00
|
|
|
extern void zk_thread_join(kt_did_t tid);
|
|
|
|
|
|
2012-08-24 00:45:31 +00:00
|
|
|
#define kpreempt_disable() ((void)0)
|
|
|
|
|
#define kpreempt_enable() ((void)0)
|
|
|
|
|
|
2010-08-26 17:43:27 +00:00
|
|
|
#define PS_NONE -1
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
#define issig(why) (FALSE)
|
|
|
|
|
#define ISSIG(thr, why) (FALSE)
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Mutexes
|
|
|
|
|
*/
|
2010-08-26 17:43:27 +00:00
|
|
|
#define MTX_MAGIC 0x9522f51362a6e326ull
|
|
|
|
|
#define MTX_INIT ((void *)NULL)
|
|
|
|
|
#define MTX_DEST ((void *)-1UL)
|
|
|
|
|
|
2008-12-11 19:14:49 +00:00
|
|
|
typedef struct kmutex {
|
|
|
|
|
void *m_owner;
|
2010-08-26 17:43:27 +00:00
|
|
|
uint64_t m_magic;
|
|
|
|
|
pthread_mutex_t m_lock;
|
2008-12-11 19:14:49 +00:00
|
|
|
} kmutex_t;
|
|
|
|
|
|
2010-08-26 17:43:27 +00:00
|
|
|
#define MUTEX_DEFAULT 0
|
2013-03-08 18:41:28 +00:00
|
|
|
#define MUTEX_HELD(m) ((m)->m_owner == curthread)
|
2010-08-26 21:24:34 +00:00
|
|
|
#define MUTEX_NOT_HELD(m) (!MUTEX_HELD(m))
|
2008-12-11 19:14:49 +00:00
|
|
|
|
2010-08-26 17:43:27 +00:00
|
|
|
extern void mutex_init(kmutex_t *mp, char *name, int type, void *cookie);
|
|
|
|
|
extern void mutex_destroy(kmutex_t *mp);
|
2008-12-11 19:14:49 +00:00
|
|
|
extern void mutex_enter(kmutex_t *mp);
|
|
|
|
|
extern void mutex_exit(kmutex_t *mp);
|
|
|
|
|
extern int mutex_tryenter(kmutex_t *mp);
|
|
|
|
|
extern void *mutex_owner(kmutex_t *mp);
|
2010-08-26 17:43:27 +00:00
|
|
|
extern int mutex_held(kmutex_t *mp);
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* RW locks
|
|
|
|
|
*/
|
2010-08-26 17:43:27 +00:00
|
|
|
#define RW_MAGIC 0x4d31fb123648e78aull
|
|
|
|
|
#define RW_INIT ((void *)NULL)
|
|
|
|
|
#define RW_DEST ((void *)-1UL)
|
|
|
|
|
|
2008-12-11 19:14:49 +00:00
|
|
|
typedef struct krwlock {
|
2010-08-26 17:43:27 +00:00
|
|
|
void *rw_owner;
|
|
|
|
|
void *rw_wr_owner;
|
|
|
|
|
uint64_t rw_magic;
|
|
|
|
|
pthread_rwlock_t rw_lock;
|
|
|
|
|
uint_t rw_readers;
|
2008-12-11 19:14:49 +00:00
|
|
|
} krwlock_t;
|
|
|
|
|
|
|
|
|
|
typedef int krw_t;
|
|
|
|
|
|
|
|
|
|
#define RW_READER 0
|
|
|
|
|
#define RW_WRITER 1
|
2013-03-08 18:41:28 +00:00
|
|
|
#define RW_DEFAULT RW_READER
|
2008-12-11 19:14:49 +00:00
|
|
|
|
2010-08-26 17:43:27 +00:00
|
|
|
#define RW_READ_HELD(x) ((x)->rw_readers > 0)
|
|
|
|
|
#define RW_WRITE_HELD(x) ((x)->rw_wr_owner == curthread)
|
|
|
|
|
#define RW_LOCK_HELD(x) (RW_READ_HELD(x) || RW_WRITE_HELD(x))
|
2008-12-11 19:14:49 +00:00
|
|
|
|
2013-09-04 12:00:57 +00:00
|
|
|
#undef RW_LOCK_HELD
|
|
|
|
|
#define RW_LOCK_HELD(x) (RW_READ_HELD(x) || RW_WRITE_HELD(x))
|
|
|
|
|
|
|
|
|
|
#undef RW_LOCK_HELD
|
|
|
|
|
#define RW_LOCK_HELD(x) (RW_READ_HELD(x) || RW_WRITE_HELD(x))
|
|
|
|
|
|
2008-12-11 19:14:49 +00:00
|
|
|
extern void rw_init(krwlock_t *rwlp, char *name, int type, void *arg);
|
|
|
|
|
extern void rw_destroy(krwlock_t *rwlp);
|
|
|
|
|
extern void rw_enter(krwlock_t *rwlp, krw_t rw);
|
|
|
|
|
extern int rw_tryenter(krwlock_t *rwlp, krw_t rw);
|
|
|
|
|
extern int rw_tryupgrade(krwlock_t *rwlp);
|
|
|
|
|
extern void rw_exit(krwlock_t *rwlp);
|
|
|
|
|
#define rw_downgrade(rwlp) do { } while (0)
|
|
|
|
|
|
|
|
|
|
extern uid_t crgetuid(cred_t *cr);
|
2013-08-28 11:45:09 +00:00
|
|
|
extern uid_t crgetruid(cred_t *cr);
|
2008-12-11 19:14:49 +00:00
|
|
|
extern gid_t crgetgid(cred_t *cr);
|
|
|
|
|
extern int crgetngroups(cred_t *cr);
|
|
|
|
|
extern gid_t *crgetgroups(cred_t *cr);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Condition variables
|
|
|
|
|
*/
|
2013-03-08 18:41:28 +00:00
|
|
|
#define CV_MAGIC 0xd31ea9a83b1b30c4ull
|
2010-08-26 17:43:27 +00:00
|
|
|
|
|
|
|
|
typedef struct kcondvar {
|
|
|
|
|
uint64_t cv_magic;
|
|
|
|
|
pthread_cond_t cv;
|
|
|
|
|
} kcondvar_t;
|
2008-12-11 19:14:49 +00:00
|
|
|
|
2010-08-26 17:43:27 +00:00
|
|
|
#define CV_DEFAULT 0
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
extern void cv_init(kcondvar_t *cv, char *name, int type, void *arg);
|
|
|
|
|
extern void cv_destroy(kcondvar_t *cv);
|
|
|
|
|
extern void cv_wait(kcondvar_t *cv, kmutex_t *mp);
|
|
|
|
|
extern clock_t cv_timedwait(kcondvar_t *cv, kmutex_t *mp, clock_t abstime);
|
2013-08-28 23:05:48 +00:00
|
|
|
extern clock_t cv_timedwait_hires(kcondvar_t *cvp, kmutex_t *mp, hrtime_t tim,
|
|
|
|
|
hrtime_t res, int flag);
|
2008-12-11 19:14:49 +00:00
|
|
|
extern void cv_signal(kcondvar_t *cv);
|
|
|
|
|
extern void cv_broadcast(kcondvar_t *cv);
|
2013-03-08 18:41:28 +00:00
|
|
|
#define cv_timedwait_interruptible(cv, mp, at) cv_timedwait(cv, mp, at)
|
|
|
|
|
#define cv_wait_interruptible(cv, mp) cv_wait(cv, mp)
|
|
|
|
|
#define cv_wait_io(cv, mp) cv_wait(cv, mp)
|
2008-12-11 19:14:49 +00:00
|
|
|
|
2013-09-04 12:00:57 +00:00
|
|
|
/*
|
|
|
|
|
* Thread-specific data
|
|
|
|
|
*/
|
|
|
|
|
#define tsd_get(k) pthread_getspecific(k)
|
|
|
|
|
#define tsd_set(k, v) pthread_setspecific(k, v)
|
|
|
|
|
#define tsd_create(kp, d) pthread_key_create(kp, d)
|
|
|
|
|
#define tsd_destroy(kp) /* nothing */
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Thread-specific data
|
|
|
|
|
*/
|
|
|
|
|
#define tsd_get(k) pthread_getspecific(k)
|
|
|
|
|
#define tsd_set(k, v) pthread_setspecific(k, v)
|
|
|
|
|
#define tsd_create(kp, d) pthread_key_create(kp, d)
|
|
|
|
|
#define tsd_destroy(kp) /* nothing */
|
|
|
|
|
|
2008-12-11 19:14:49 +00:00
|
|
|
/*
|
|
|
|
|
* kstat creation, installation and deletion
|
|
|
|
|
*/
|
2013-08-27 00:09:29 +00:00
|
|
|
extern kstat_t *kstat_create(const char *, int,
|
|
|
|
|
const char *, const char *, uchar_t, ulong_t, uchar_t);
|
2008-12-11 19:14:49 +00:00
|
|
|
extern void kstat_install(kstat_t *);
|
|
|
|
|
extern void kstat_delete(kstat_t *);
|
2013-08-27 00:09:29 +00:00
|
|
|
extern void kstat_waitq_enter(kstat_io_t *);
|
|
|
|
|
extern void kstat_waitq_exit(kstat_io_t *);
|
|
|
|
|
extern void kstat_runq_enter(kstat_io_t *);
|
|
|
|
|
extern void kstat_runq_exit(kstat_io_t *);
|
|
|
|
|
extern void kstat_waitq_to_runq(kstat_io_t *);
|
|
|
|
|
extern void kstat_runq_back_to_waitq(kstat_io_t *);
|
Add visibility in to arc_read
This change is an attempt to add visibility into the arc_read calls
occurring on a system, in real time. To do this, a list was added to the
in memory SPA data structure for a pool, with each element on the list
corresponding to a call to arc_read. These entries are then exported
through the kstat interface, which can then be interpreted in userspace.
For each arc_read call, the following information is exported:
* A unique identifier (uint64_t)
* The time the entry was added to the list (hrtime_t)
(*not* wall clock time; relative to the other entries on the list)
* The objset ID (uint64_t)
* The object number (uint64_t)
* The indirection level (uint64_t)
* The block ID (uint64_t)
* The name of the function originating the arc_read call (char[24])
* The arc_flags from the arc_read call (uint32_t)
* The PID of the reading thread (pid_t)
* The command or name of thread originating read (char[16])
From this exported information one can see, in real time, exactly what
is being read, what function is generating the read, and whether or not
the read was found to be already cached.
There is still some work to be done, but this should serve as a good
starting point.
Specifically, dbuf_read's are not accounted for in the currently
exported information. Thus, a follow up patch should probably be added
to export these calls that never call into arc_read (they only hit the
dbuf hash table). In addition, it might be nice to create a utility
similar to "arcstat.py" to digest the exported information and display
it in a more readable format. Or perhaps, log the information and allow
for it to be "replayed" at a later time.
Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
2013-09-06 23:09:05 +00:00
|
|
|
extern void kstat_set_raw_ops(kstat_t *ksp,
|
|
|
|
|
int (*headers)(char *buf, size_t size),
|
|
|
|
|
int (*data)(char *buf, size_t size, void *data),
|
|
|
|
|
void *(*addr)(kstat_t *ksp, loff_t index));
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Kernel memory
|
|
|
|
|
*/
|
|
|
|
|
#define KM_SLEEP UMEM_NOFAIL
|
|
|
|
|
#define KM_PUSHPAGE KM_SLEEP
|
|
|
|
|
#define KM_NOSLEEP UMEM_DEFAULT
|
2010-08-26 18:46:09 +00:00
|
|
|
#define KM_NODEBUG 0x0
|
2008-12-11 19:14:49 +00:00
|
|
|
#define KMC_NODEBUG UMC_NODEBUG
|
2011-11-01 23:56:48 +00:00
|
|
|
#define KMC_KMEM 0x0
|
|
|
|
|
#define KMC_VMEM 0x0
|
2008-12-11 19:14:49 +00:00
|
|
|
#define kmem_alloc(_s, _f) umem_alloc(_s, _f)
|
|
|
|
|
#define kmem_zalloc(_s, _f) umem_zalloc(_s, _f)
|
|
|
|
|
#define kmem_free(_b, _s) umem_free(_b, _s)
|
2010-08-26 18:46:09 +00:00
|
|
|
#define vmem_alloc(_s, _f) kmem_alloc(_s, _f)
|
|
|
|
|
#define vmem_zalloc(_s, _f) kmem_zalloc(_s, _f)
|
|
|
|
|
#define vmem_free(_b, _s) kmem_free(_b, _s)
|
2008-12-11 19:14:49 +00:00
|
|
|
#define kmem_cache_create(_a, _b, _c, _d, _e, _f, _g, _h, _i) \
|
|
|
|
|
umem_cache_create(_a, _b, _c, _d, _e, _f, _g, _h, _i)
|
|
|
|
|
#define kmem_cache_destroy(_c) umem_cache_destroy(_c)
|
|
|
|
|
#define kmem_cache_alloc(_c, _f) umem_cache_alloc(_c, _f)
|
|
|
|
|
#define kmem_cache_free(_c, _b) umem_cache_free(_c, _b)
|
|
|
|
|
#define kmem_debugging() 0
|
2010-08-26 21:24:34 +00:00
|
|
|
#define kmem_cache_reap_now(_c) /* nothing */
|
|
|
|
|
#define kmem_cache_set_move(_c, _cb) /* nothing */
|
|
|
|
|
#define POINTER_INVALIDATE(_pp) /* nothing */
|
|
|
|
|
#define POINTER_IS_VALID(_p) 0
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
typedef umem_cache_t kmem_cache_t;
|
|
|
|
|
|
2010-08-26 21:24:34 +00:00
|
|
|
typedef enum kmem_cbrc {
|
|
|
|
|
KMEM_CBRC_YES,
|
|
|
|
|
KMEM_CBRC_NO,
|
|
|
|
|
KMEM_CBRC_LATER,
|
|
|
|
|
KMEM_CBRC_DONT_NEED,
|
|
|
|
|
KMEM_CBRC_DONT_KNOW
|
|
|
|
|
} kmem_cbrc_t;
|
|
|
|
|
|
2008-12-11 19:14:49 +00:00
|
|
|
/*
|
|
|
|
|
* Task queues
|
|
|
|
|
*/
|
|
|
|
|
typedef struct taskq taskq_t;
|
|
|
|
|
typedef uintptr_t taskqid_t;
|
|
|
|
|
typedef void (task_func_t)(void *);
|
|
|
|
|
|
2011-11-08 00:26:52 +00:00
|
|
|
typedef struct taskq_ent {
|
|
|
|
|
struct taskq_ent *tqent_next;
|
|
|
|
|
struct taskq_ent *tqent_prev;
|
|
|
|
|
task_func_t *tqent_func;
|
|
|
|
|
void *tqent_arg;
|
|
|
|
|
uintptr_t tqent_flags;
|
|
|
|
|
} taskq_ent_t;
|
|
|
|
|
|
|
|
|
|
#define TQENT_FLAG_PREALLOC 0x1 /* taskq_dispatch_ent used */
|
|
|
|
|
|
2008-12-11 19:14:49 +00:00
|
|
|
#define TASKQ_PREPOPULATE 0x0001
|
|
|
|
|
#define TASKQ_CPR_SAFE 0x0002 /* Use CPR safe protocol */
|
|
|
|
|
#define TASKQ_DYNAMIC 0x0004 /* Use dynamic thread scheduling */
|
2010-05-28 20:45:14 +00:00
|
|
|
#define TASKQ_THREADS_CPU_PCT 0x0008 /* Scale # threads by # cpus */
|
|
|
|
|
#define TASKQ_DC_BATCH 0x0010 /* Mark threads as batch */
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
#define TQ_SLEEP KM_SLEEP /* Can block for memory */
|
|
|
|
|
#define TQ_NOSLEEP KM_NOSLEEP /* cannot block for memory; may fail */
|
2012-09-04 22:52:36 +00:00
|
|
|
#define TQ_PUSHPAGE KM_PUSHPAGE /* Cannot perform I/O */
|
2010-05-28 20:45:14 +00:00
|
|
|
#define TQ_NOQUEUE 0x02 /* Do not enqueue if can't dispatch */
|
|
|
|
|
#define TQ_FRONT 0x08 /* Queue in front */
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
extern taskq_t *system_taskq;
|
|
|
|
|
|
|
|
|
|
extern taskq_t *taskq_create(const char *, int, pri_t, int, int, uint_t);
|
2010-05-28 20:45:14 +00:00
|
|
|
#define taskq_create_proc(a, b, c, d, e, p, f) \
|
|
|
|
|
(taskq_create(a, b, c, d, e, f))
|
|
|
|
|
#define taskq_create_sysdc(a, b, d, e, p, dc, f) \
|
|
|
|
|
(taskq_create(a, b, maxclsyspri, d, e, f))
|
2008-12-11 19:14:49 +00:00
|
|
|
extern taskqid_t taskq_dispatch(taskq_t *, task_func_t, void *, uint_t);
|
2013-04-29 22:49:23 +00:00
|
|
|
extern taskqid_t taskq_dispatch_delay(taskq_t *, task_func_t, void *, uint_t,
|
|
|
|
|
clock_t);
|
2011-11-08 00:26:52 +00:00
|
|
|
extern void taskq_dispatch_ent(taskq_t *, task_func_t, void *, uint_t,
|
|
|
|
|
taskq_ent_t *);
|
|
|
|
|
extern int taskq_empty_ent(taskq_ent_t *);
|
|
|
|
|
extern void taskq_init_ent(taskq_ent_t *);
|
2008-12-11 19:14:49 +00:00
|
|
|
extern void taskq_destroy(taskq_t *);
|
|
|
|
|
extern void taskq_wait(taskq_t *);
|
2013-05-03 21:17:21 +00:00
|
|
|
extern void taskq_wait_id(taskq_t *, taskqid_t);
|
2010-08-26 17:43:27 +00:00
|
|
|
extern int taskq_member(taskq_t *, kthread_t *);
|
2013-04-29 22:49:23 +00:00
|
|
|
extern int taskq_cancel_id(taskq_t *, taskqid_t);
|
2008-12-11 19:14:49 +00:00
|
|
|
extern void system_taskq_init(void);
|
2010-05-28 20:45:14 +00:00
|
|
|
extern void system_taskq_fini(void);
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
#define XVA_MAPSIZE 3
|
|
|
|
|
#define XVA_MAGIC 0x78766174
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* vnodes
|
|
|
|
|
*/
|
|
|
|
|
typedef struct vnode {
|
|
|
|
|
uint64_t v_size;
|
|
|
|
|
int v_fd;
|
|
|
|
|
char *v_path;
|
|
|
|
|
} vnode_t;
|
|
|
|
|
|
2010-05-28 20:45:14 +00:00
|
|
|
#define AV_SCANSTAMP_SZ 32 /* length of anti-virus scanstamp */
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
typedef struct xoptattr {
|
|
|
|
|
timestruc_t xoa_createtime; /* Create time of file */
|
|
|
|
|
uint8_t xoa_archive;
|
|
|
|
|
uint8_t xoa_system;
|
|
|
|
|
uint8_t xoa_readonly;
|
|
|
|
|
uint8_t xoa_hidden;
|
|
|
|
|
uint8_t xoa_nounlink;
|
|
|
|
|
uint8_t xoa_immutable;
|
|
|
|
|
uint8_t xoa_appendonly;
|
|
|
|
|
uint8_t xoa_nodump;
|
|
|
|
|
uint8_t xoa_settable;
|
|
|
|
|
uint8_t xoa_opaque;
|
|
|
|
|
uint8_t xoa_av_quarantined;
|
|
|
|
|
uint8_t xoa_av_modified;
|
2010-05-28 20:45:14 +00:00
|
|
|
uint8_t xoa_av_scanstamp[AV_SCANSTAMP_SZ];
|
|
|
|
|
uint8_t xoa_reparse;
|
2010-08-26 21:24:34 +00:00
|
|
|
uint8_t xoa_offline;
|
|
|
|
|
uint8_t xoa_sparse;
|
2008-12-11 19:14:49 +00:00
|
|
|
} xoptattr_t;
|
|
|
|
|
|
|
|
|
|
typedef struct vattr {
|
|
|
|
|
uint_t va_mask; /* bit-mask of attributes */
|
|
|
|
|
u_offset_t va_size; /* file size in bytes */
|
|
|
|
|
} vattr_t;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
typedef struct xvattr {
|
|
|
|
|
vattr_t xva_vattr; /* Embedded vattr structure */
|
|
|
|
|
uint32_t xva_magic; /* Magic Number */
|
|
|
|
|
uint32_t xva_mapsize; /* Size of attr bitmap (32-bit words) */
|
|
|
|
|
uint32_t *xva_rtnattrmapp; /* Ptr to xva_rtnattrmap[] */
|
|
|
|
|
uint32_t xva_reqattrmap[XVA_MAPSIZE]; /* Requested attrs */
|
|
|
|
|
uint32_t xva_rtnattrmap[XVA_MAPSIZE]; /* Returned attrs */
|
|
|
|
|
xoptattr_t xva_xoptattrs; /* Optional attributes */
|
|
|
|
|
} xvattr_t;
|
|
|
|
|
|
|
|
|
|
typedef struct vsecattr {
|
|
|
|
|
uint_t vsa_mask; /* See below */
|
|
|
|
|
int vsa_aclcnt; /* ACL entry count */
|
|
|
|
|
void *vsa_aclentp; /* pointer to ACL entries */
|
|
|
|
|
int vsa_dfaclcnt; /* default ACL entry count */
|
|
|
|
|
void *vsa_dfaclentp; /* pointer to default ACL entries */
|
|
|
|
|
size_t vsa_aclentsz; /* ACE size in bytes of vsa_aclentp */
|
|
|
|
|
} vsecattr_t;
|
|
|
|
|
|
|
|
|
|
#define AT_TYPE 0x00001
|
|
|
|
|
#define AT_MODE 0x00002
|
|
|
|
|
#define AT_UID 0x00004
|
|
|
|
|
#define AT_GID 0x00008
|
|
|
|
|
#define AT_FSID 0x00010
|
|
|
|
|
#define AT_NODEID 0x00020
|
|
|
|
|
#define AT_NLINK 0x00040
|
|
|
|
|
#define AT_SIZE 0x00080
|
|
|
|
|
#define AT_ATIME 0x00100
|
|
|
|
|
#define AT_MTIME 0x00200
|
|
|
|
|
#define AT_CTIME 0x00400
|
|
|
|
|
#define AT_RDEV 0x00800
|
|
|
|
|
#define AT_BLKSIZE 0x01000
|
|
|
|
|
#define AT_NBLOCKS 0x02000
|
|
|
|
|
#define AT_SEQ 0x08000
|
|
|
|
|
#define AT_XVATTR 0x10000
|
|
|
|
|
|
|
|
|
|
#define CRCREAT 0
|
|
|
|
|
|
2010-05-28 20:45:14 +00:00
|
|
|
extern int fop_getattr(vnode_t *vp, vattr_t *vap);
|
|
|
|
|
|
2012-09-27 14:05:15 +00:00
|
|
|
#define VOP_CLOSE(vp, f, c, o, cr, ct) vn_close(vp)
|
2008-12-11 19:14:49 +00:00
|
|
|
#define VOP_PUTPAGE(vp, of, sz, fl, cr, ct) 0
|
2010-05-28 20:45:14 +00:00
|
|
|
#define VOP_GETATTR(vp, vap, fl, cr, ct) fop_getattr((vp), (vap));
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
#define VOP_FSYNC(vp, f, cr, ct) fsync((vp)->v_fd)
|
|
|
|
|
|
|
|
|
|
#define VN_RELE(vp) vn_close(vp)
|
|
|
|
|
|
|
|
|
|
extern int vn_open(char *path, int x1, int oflags, int mode, vnode_t **vpp,
|
|
|
|
|
int x2, int x3);
|
|
|
|
|
extern int vn_openat(char *path, int x1, int oflags, int mode, vnode_t **vpp,
|
|
|
|
|
int x2, int x3, vnode_t *vp, int fd);
|
|
|
|
|
extern int vn_rdwr(int uio, vnode_t *vp, void *addr, ssize_t len,
|
|
|
|
|
offset_t offset, int x1, int x2, rlim64_t x3, void *x4, ssize_t *residp);
|
|
|
|
|
extern void vn_close(vnode_t *vp);
|
|
|
|
|
|
|
|
|
|
#define vn_remove(path, x1, x2) remove(path)
|
|
|
|
|
#define vn_rename(from, to, seg) rename((from), (to))
|
|
|
|
|
#define vn_is_readonly(vp) B_FALSE
|
|
|
|
|
|
|
|
|
|
extern vnode_t *rootdir;
|
|
|
|
|
|
|
|
|
|
#include <sys/file.h> /* for FREAD, FWRITE, etc */
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Random stuff
|
|
|
|
|
*/
|
2010-05-28 20:45:14 +00:00
|
|
|
#define ddi_get_lbolt() (gethrtime() >> 23)
|
|
|
|
|
#define ddi_get_lbolt64() (gethrtime() >> 23)
|
2008-12-11 19:14:49 +00:00
|
|
|
#define hz 119 /* frequency when using gethrtime() >> 23 for lbolt */
|
|
|
|
|
|
|
|
|
|
extern void delay(clock_t ticks);
|
|
|
|
|
|
2013-04-29 22:49:23 +00:00
|
|
|
#define SEC_TO_TICK(sec) ((sec) * hz)
|
|
|
|
|
#define MSEC_TO_TICK(msec) ((msec) / (MILLISEC / hz))
|
|
|
|
|
#define USEC_TO_TICK(usec) ((usec) / (MICROSEC / hz))
|
|
|
|
|
#define NSEC_TO_TICK(usec) ((usec) / (NANOSEC / hz))
|
|
|
|
|
|
2008-12-11 19:14:49 +00:00
|
|
|
#define gethrestime_sec() time(NULL)
|
2010-05-28 20:45:14 +00:00
|
|
|
#define gethrestime(t) \
|
|
|
|
|
do {\
|
|
|
|
|
(t)->tv_sec = gethrestime_sec();\
|
|
|
|
|
(t)->tv_nsec = 0;\
|
|
|
|
|
} while (0);
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
#define max_ncpus 64
|
|
|
|
|
|
|
|
|
|
#define minclsyspri 60
|
|
|
|
|
#define maxclsyspri 99
|
|
|
|
|
|
2010-08-26 17:43:27 +00:00
|
|
|
#define CPU_SEQID (pthread_self() & (max_ncpus - 1))
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
#define kcred NULL
|
|
|
|
|
#define CRED() NULL
|
|
|
|
|
|
|
|
|
|
#define ptob(x) ((x) * PAGESIZE)
|
|
|
|
|
|
|
|
|
|
extern uint64_t physmem;
|
|
|
|
|
|
|
|
|
|
extern int highbit(ulong_t i);
|
|
|
|
|
extern int random_get_bytes(uint8_t *ptr, size_t len);
|
|
|
|
|
extern int random_get_pseudo_bytes(uint8_t *ptr, size_t len);
|
|
|
|
|
|
|
|
|
|
extern void kernel_init(int);
|
|
|
|
|
extern void kernel_fini(void);
|
|
|
|
|
|
|
|
|
|
struct spa;
|
|
|
|
|
extern void nicenum(uint64_t num, char *buf);
|
|
|
|
|
extern void show_pool_stats(struct spa *);
|
|
|
|
|
|
|
|
|
|
typedef struct callb_cpr {
|
|
|
|
|
kmutex_t *cc_lockp;
|
|
|
|
|
} callb_cpr_t;
|
|
|
|
|
|
|
|
|
|
#define CALLB_CPR_INIT(cp, lockp, func, name) { \
|
|
|
|
|
(cp)->cc_lockp = lockp; \
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#define CALLB_CPR_SAFE_BEGIN(cp) { \
|
|
|
|
|
ASSERT(MUTEX_HELD((cp)->cc_lockp)); \
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#define CALLB_CPR_SAFE_END(cp, lockp) { \
|
|
|
|
|
ASSERT(MUTEX_HELD((cp)->cc_lockp)); \
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#define CALLB_CPR_EXIT(cp) { \
|
|
|
|
|
ASSERT(MUTEX_HELD((cp)->cc_lockp)); \
|
|
|
|
|
mutex_exit((cp)->cc_lockp); \
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#define zone_dataset_visible(x, y) (1)
|
|
|
|
|
#define INGLOBALZONE(z) (1)
|
|
|
|
|
|
2010-08-26 18:46:09 +00:00
|
|
|
extern char *kmem_vasprintf(const char *fmt, va_list adx);
|
2010-05-28 20:45:14 +00:00
|
|
|
extern char *kmem_asprintf(const char *fmt, ...);
|
2013-09-04 12:00:57 +00:00
|
|
|
#define strfree(str) kmem_free((str), strlen(str) + 1)
|
2010-05-28 20:45:14 +00:00
|
|
|
|
2008-12-11 19:14:49 +00:00
|
|
|
/*
|
|
|
|
|
* Hostname information
|
|
|
|
|
*/
|
2009-02-18 20:51:31 +00:00
|
|
|
extern char hw_serial[]; /* for userland-emulated hostid access */
|
2008-12-11 19:14:49 +00:00
|
|
|
extern int ddi_strtoul(const char *str, char **nptr, int base,
|
|
|
|
|
unsigned long *result);
|
|
|
|
|
|
2010-05-28 20:45:14 +00:00
|
|
|
extern int ddi_strtoull(const char *str, char **nptr, int base,
|
|
|
|
|
u_longlong_t *result);
|
|
|
|
|
|
2008-12-11 19:14:49 +00:00
|
|
|
/* ZFS Boot Related stuff. */
|
|
|
|
|
|
|
|
|
|
struct _buf {
|
|
|
|
|
intptr_t _fd;
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
struct bootstat {
|
|
|
|
|
uint64_t st_size;
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
typedef struct ace_object {
|
|
|
|
|
uid_t a_who;
|
|
|
|
|
uint32_t a_access_mask;
|
|
|
|
|
uint16_t a_flags;
|
|
|
|
|
uint16_t a_type;
|
|
|
|
|
uint8_t a_obj_type[16];
|
|
|
|
|
uint8_t a_inherit_obj_type[16];
|
|
|
|
|
} ace_object_t;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#define ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE 0x05
|
|
|
|
|
#define ACE_ACCESS_DENIED_OBJECT_ACE_TYPE 0x06
|
|
|
|
|
#define ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE 0x07
|
|
|
|
|
#define ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE 0x08
|
|
|
|
|
|
|
|
|
|
extern struct _buf *kobj_open_file(char *name);
|
|
|
|
|
extern int kobj_read_file(struct _buf *file, char *buf, unsigned size,
|
|
|
|
|
unsigned off);
|
|
|
|
|
extern void kobj_close_file(struct _buf *file);
|
|
|
|
|
extern int kobj_get_filesize(struct _buf *file, uint64_t *size);
|
|
|
|
|
extern int zfs_secpolicy_snapshot_perms(const char *name, cred_t *cr);
|
|
|
|
|
extern int zfs_secpolicy_rename_perms(const char *from, const char *to,
|
|
|
|
|
cred_t *cr);
|
|
|
|
|
extern int zfs_secpolicy_destroy_perms(const char *name, cred_t *cr);
|
|
|
|
|
extern zoneid_t getzoneid(void);
|
|
|
|
|
|
|
|
|
|
/* SID stuff */
|
|
|
|
|
typedef struct ksiddomain {
|
|
|
|
|
uint_t kd_ref;
|
|
|
|
|
uint_t kd_len;
|
|
|
|
|
char *kd_name;
|
|
|
|
|
} ksiddomain_t;
|
|
|
|
|
|
|
|
|
|
ksiddomain_t *ksid_lookupdomain(const char *);
|
|
|
|
|
void ksiddomain_rele(ksiddomain_t *);
|
|
|
|
|
|
2009-07-02 22:44:48 +00:00
|
|
|
#define DDI_SLEEP KM_SLEEP
|
|
|
|
|
#define ddi_log_sysevent(_a, _b, _c, _d, _e, _f, _g) \
|
|
|
|
|
sysevent_post_event(_c, _d, _b, "libzpool", _e, _f)
|
|
|
|
|
|
2013-11-01 19:26:11 +00:00
|
|
|
#define zfs_sleep_until(wakeup) \
|
Illumos #4045 write throttle & i/o scheduler performance work
4045 zfs write throttle & i/o scheduler performance work
1. The ZFS i/o scheduler (vdev_queue.c) now divides i/os into 5 classes: sync
read, sync write, async read, async write, and scrub/resilver. The scheduler
issues a number of concurrent i/os from each class to the device. Once a class
has been selected, an i/o is selected from this class using either an elevator
algorithem (async, scrub classes) or FIFO (sync classes). The number of
concurrent async write i/os is tuned dynamically based on i/o load, to achieve
good sync i/o latency when there is not a high load of writes, and good write
throughput when there is. See the block comment in vdev_queue.c (reproduced
below) for more details.
2. The write throttle (dsl_pool_tempreserve_space() and
txg_constrain_throughput()) is rewritten to produce much more consistent delays
when under constant load. The new write throttle is based on the amount of
dirty data, rather than guesses about future performance of the system. When
there is a lot of dirty data, each transaction (e.g. write() syscall) will be
delayed by the same small amount. This eliminates the "brick wall of wait"
that the old write throttle could hit, causing all transactions to wait several
seconds until the next txg opens. One of the keys to the new write throttle is
decrementing the amount of dirty data as i/o completes, rather than at the end
of spa_sync(). Note that the write throttle is only applied once the i/o
scheduler is issuing the maximum number of outstanding async writes. See the
block comments in dsl_pool.c and above dmu_tx_delay() (reproduced below) for
more details.
This diff has several other effects, including:
* the commonly-tuned global variable zfs_vdev_max_pending has been removed;
use per-class zfs_vdev_*_max_active values or zfs_vdev_max_active instead.
* the size of each txg (meaning the amount of dirty data written, and thus the
time it takes to write out) is now controlled differently. There is no longer
an explicit time goal; the primary determinant is amount of dirty data.
Systems that are under light or medium load will now often see that a txg is
always syncing, but the impact to performance (e.g. read latency) is minimal.
Tune zfs_dirty_data_max and zfs_dirty_data_sync to control this.
* zio_taskq_batch_pct = 75 -- Only use 75% of all CPUs for compression,
checksum, etc. This improves latency by not allowing these CPU-intensive tasks
to consume all CPU (on machines with at least 4 CPU's; the percentage is
rounded up).
--matt
APPENDIX: problems with the current i/o scheduler
The current ZFS i/o scheduler (vdev_queue.c) is deadline based. The problem
with this is that if there are always i/os pending, then certain classes of
i/os can see very long delays.
For example, if there are always synchronous reads outstanding, then no async
writes will be serviced until they become "past due". One symptom of this
situation is that each pass of the txg sync takes at least several seconds
(typically 3 seconds).
If many i/os become "past due" (their deadline is in the past), then we must
service all of these overdue i/os before any new i/os. This happens when we
enqueue a batch of async writes for the txg sync, with deadlines 2.5 seconds in
the future. If we can't complete all the i/os in 2.5 seconds (e.g. because
there were always reads pending), then these i/os will become past due. Now we
must service all the "async" writes (which could be hundreds of megabytes)
before we service any reads, introducing considerable latency to synchronous
i/os (reads or ZIL writes).
Notes on porting to ZFS on Linux:
- zio_t gained new members io_physdone and io_phys_children. Because
object caches in the Linux port call the constructor only once at
allocation time, objects may contain residual data when retrieved
from the cache. Therefore zio_create() was updated to zero out the two
new fields.
- vdev_mirror_pending() relied on the depth of the per-vdev pending queue
(vq->vq_pending_tree) to select the least-busy leaf vdev to read from.
This tree has been replaced by vq->vq_active_tree which is now used
for the same purpose.
- vdev_queue_init() used the value of zfs_vdev_max_pending to determine
the number of vdev I/O buffers to pre-allocate. That global no longer
exists, so we instead use the sum of the *_max_active values for each of
the five I/O classes described above.
- The Illumos implementation of dmu_tx_delay() delays a transaction by
sleeping in condition variable embedded in the thread
(curthread->t_delay_cv). We do not have an equivalent CV to use in
Linux, so this change replaced the delay logic with a wrapper called
zfs_sleep_until(). This wrapper could be adopted upstream and in other
downstream ports to abstract away operating system-specific delay logic.
- These tunables are added as module parameters, and descriptions added
to the zfs-module-parameters.5 man page.
spa_asize_inflation
zfs_deadman_synctime_ms
zfs_vdev_max_active
zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_async_write_max_active
zfs_vdev_async_write_min_active
zfs_vdev_scrub_max_active
zfs_vdev_scrub_min_active
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
zfs_vdev_sync_write_max_active
zfs_vdev_sync_write_min_active
zfs_dirty_data_max_percent
zfs_delay_min_dirty_percent
zfs_dirty_data_max_max_percent
zfs_dirty_data_max
zfs_dirty_data_max_max
zfs_dirty_data_sync
zfs_delay_scale
The latter four have type unsigned long, whereas they are uint64_t in
Illumos. This accommodates Linux's module_param() supported types, but
means they may overflow on 32-bit architectures.
The values zfs_dirty_data_max and zfs_dirty_data_max_max are the most
likely to overflow on 32-bit systems, since they express physical RAM
sizes in bytes. In fact, Illumos initializes zfs_dirty_data_max_max to
2^32 which does overflow. To resolve that, this port instead initializes
it in arc_init() to 25% of physical RAM, and adds the tunable
zfs_dirty_data_max_max_percent to override that percentage. While this
solution doesn't completely avoid the overflow issue, it should be a
reasonable default for most systems, and the minority of affected
systems can work around the issue by overriding the defaults.
- Fixed reversed logic in comment above zfs_delay_scale declaration.
- Clarified comments in vdev_queue.c regarding when per-queue minimums take
effect.
- Replaced dmu_tx_write_limit in the dmu_tx kstat file
with dmu_tx_dirty_delay and dmu_tx_dirty_over_max. The first counts
how many times a transaction has been delayed because the pool dirty
data has exceeded zfs_delay_min_dirty_percent. The latter counts how
many times the pool dirty data has exceeded zfs_dirty_data_max (which
we expect to never happen).
- The original patch would have regressed the bug fixed in
zfsonlinux/zfs@c418410, which prevented users from setting the
zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE.
A similar fix is added to vdev_queue_aggregate().
- In vdev_queue_io_to_issue(), dynamically allocate 'zio_t search' on the
heap instead of the stack. In Linux we can't afford such large
structures on the stack.
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brendan Gregg <brendan.gregg@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
References:
http://www.illumos.org/issues/4045
illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e
Ported-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #1913
2013-08-29 03:01:20 +00:00
|
|
|
do { \
|
|
|
|
|
hrtime_t delta = wakeup - gethrtime(); \
|
|
|
|
|
struct timespec ts; \
|
|
|
|
|
ts.tv_sec = delta / NANOSEC; \
|
|
|
|
|
ts.tv_nsec = delta % NANOSEC; \
|
|
|
|
|
(void) nanosleep(&ts, NULL); \
|
|
|
|
|
} while (0)
|
|
|
|
|
|
2010-09-04 20:26:23 +00:00
|
|
|
#endif /* _KERNEL */
|
2008-12-11 19:14:49 +00:00
|
|
|
|
|
|
|
|
#endif /* _SYS_ZFS_CONTEXT_H */
|
