Archive-Team
/
zfs
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 103 Wiki Activity
zfs/include/sys
History
Don Brady 3dfb57a35e OpenZFS 7090 - zfs should throttle allocations 
OpenZFS 7090 - zfs should throttle allocations

Authored by: George Wilson <george.wilson@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <paul.dagnelie@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Sebastien Roy <sebastien.roy@delphix.com>
Approved by: Matthew Ahrens <mahrens@delphix.com>
Ported-by: Don Brady <don.brady@intel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>

When write I/Os are issued, they are issued in block order but the ZIO
pipeline will drive them asynchronously through the allocation stage
which can result in blocks being allocated out-of-order. It would be
nice to preserve as much of the logical order as possible.

In addition, the allocations are equally scattered across all top-level
VDEVs but not all top-level VDEVs are created equally. The pipeline
should be able to detect devices that are more capable of handling
allocations and should allocate more blocks to those devices. This
allows for dynamic allocation distribution when devices are imbalanced
as fuller devices will tend to be slower than empty devices.

The change includes a new pool-wide allocation queue which would
throttle and order allocations in the ZIO pipeline. The queue would be
ordered by issued time and offset and would provide an initial amount of
allocation of work to each top-level vdev. The allocation logic utilizes
a reservation system to reserve allocations that will be performed by
the allocator. Once an allocation is successfully completed it's
scheduled on a given top-level vdev. Each top-level vdev maintains a
maximum number of allocations that it can handle (mg_alloc_queue_depth).
The pool-wide reserved allocations (top-levels * mg_alloc_queue_depth)
are distributed across the top-level vdevs metaslab groups and round
robin across all eligible metaslab groups to distribute the work. As
top-levels complete their work, they receive additional work from the
pool-wide allocation queue until the allocation queue is emptied.

OpenZFS-issue: https://www.illumos.org/issues/7090
OpenZFS-commit: https://github.com/openzfs/openzfs/commit/4756c3d7
Closes #5258 

Porting Notes:
- Maintained minimal stack in zio_done
- Preserve linux-specific io sizes in zio_write_compress
- Added module params and documentation
- Updated to use optimize AVL cmp macros
 		2016-10-13 17:59:18 -07:00
..
crypto OpenZFS 4185 - add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R 2016-10-03 14:51:15 -07:00
fm Bring over illumos ZFS FMA logic -- phase 1 2016-09-01 11:39:45 -07:00
fs OpenZFS 7090 - zfs should throttle allocations 2016-10-13 17:59:18 -07:00
sysevent Bring over illumos ZFS FMA logic -- phase 1 2016-09-01 11:39:45 -07:00
Makefile.am OpenZFS 4185 - add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R 2016-10-03 14:51:15 -07:00
arc.h DLPX-44733 combine arc_buf_alloc_impl() with arc_buf_clone() 2016-09-13 09:59:13 -07:00
arc_impl.h DLPX-40252 integrate EP-476 compressed zfs send/receive 2016-09-13 09:58:58 -07:00
avl.h Performance optimization of AVL tree comparator functions 2016-08-31 14:35:34 -07:00
avl_impl.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
blkptr.h Illumos 4757, 4913 2014-08-01 14:28:05 -07:00
bplist.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
bpobj.h Illumos 5810 - zdb should print details of bpobj 2015-05-11 15:10:24 -07:00
bptree.h Illumos 4914 - zfs on-disk bookmark structure should be named *_phys_t 2014-08-06 14:48:41 -07:00
bqueue.h Illumos 5960, 5925 2016-01-08 15:08:19 -08:00
dbuf.h fix: Shift exponent too large 2016-09-29 15:55:41 -07:00
ddt.h Add ddt, ddt_entry, and l2arc_hdr caches 2014-01-07 10:33:11 -08:00
dmu.h Add support for user/group dnode accounting & quota 2016-10-07 09:45:13 -07:00
dmu_impl.h OpenZFS 7230 - add assertions to dmu_send_impl() to verify that stream includes BEGIN and END records 2016-09-22 16:01:19 -07:00
dmu_objset.h Add support for user/group dnode accounting & quota 2016-10-07 09:45:13 -07:00
dmu_send.h DLPX-40252 integrate EP-476 compressed zfs send/receive 2016-09-13 09:58:58 -07:00
dmu_traverse.h OpenZFS 2605, 6980, 6902 2016-06-28 13:47:02 -07:00
dmu_tx.h dmu_tx kstat cleanup 2014-03-04 12:22:24 -08:00
dmu_zfetch.h OpenZFS 6322 - ZFS indirect block predictive prefetch 2016-08-30 14:26:55 -07:00
dnode.h Add support for user/group dnode accounting & quota 2016-10-07 09:45:13 -07:00
dsl_bookmark.h Illumos 4368, 4369. 2014-07-29 10:55:29 -07:00
dsl_dataset.h DLPX-40252 integrate EP-476 compressed zfs send/receive 2016-09-13 09:58:58 -07:00
dsl_deadlist.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
dsl_deleg.h Add support for user/group dnode accounting & quota 2016-10-07 09:45:13 -07:00
dsl_destroy.h Illumos #3888 2013-11-04 11:18:14 -08:00
dsl_dir.h OpenZFS 6314 - buffer overflow in dsl_dataset_name 2016-06-28 13:47:03 -07:00
dsl_pool.h Illumos 5981 - Deadlock in dmu_objset_find_dp 2015-07-06 09:31:35 -07:00
dsl_prop.h Illumos 6171 - dsl_prop_unregister() slows down dataset eviction. 2016-01-12 10:53:12 -08:00
dsl_scan.h Illumos 4914 - zfs on-disk bookmark structure should be named *_phys_t 2014-08-06 14:48:41 -07:00
dsl_synctask.h Illumos 4951 - ZFS administrative commands should use reserved space 2015-05-04 09:41:10 -07:00
dsl_userhold.h Illumos #3740 2013-11-04 11:17:48 -08:00
edonr.h OpenZFS 4185 - add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R 2016-10-03 14:51:15 -07:00
efi_partition.h Ext4's typical GPT partition type not recognized 2015-12-04 09:27:00 -08:00
metaslab.h OpenZFS 7090 - zfs should throttle allocations 2016-10-13 17:59:18 -07:00
metaslab_impl.h OpenZFS 7090 - zfs should throttle allocations 2016-10-13 17:59:18 -07:00
mntent.h Make zfs mount according to relatime config in dataset 2016-04-05 18:55:59 -07:00
multilist.h Illumos 5497 - lock contention on arcs_mtx 2015-06-11 10:27:25 -07:00
nvpair.h Replace __va_list with va_list 2014-08-13 10:35:00 -07:00
nvpair_impl.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
pathname.h Add pn_alloc()/pn_free() functions 2016-04-21 09:49:25 -07:00
policy.h Add `zfs allow` and `zfs unallow` support 2016-06-07 09:16:52 -07:00
range_tree.h Illumos #4374 2014-07-30 09:20:35 -07:00
refcount.h OpenZFS 7090 - zfs should throttle allocations 2016-10-13 17:59:18 -07:00
rrwlock.h Illumos 5008 - lock contention (rrw_exit) while running a read only load 2015-07-06 09:34:13 -07:00
sa.h Prevent SA length overflow 2015-12-30 13:20:12 -08:00
sa_impl.h Implement large_dnode pool feature 2016-06-24 13:13:21 -07:00
sdt.h Swap DTRACE_PROBE* with Linux tracepoints 2014-11-17 11:13:55 -08:00
sha2.h OpenZFS 4185 - add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R 2016-10-03 14:51:15 -07:00
skein.h OpenZFS 4185 - add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R 2016-10-03 14:51:15 -07:00
spa.h OpenZFS 4185 - add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R 2016-10-03 14:51:15 -07:00
spa_boot.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
spa_checksum.h Implementation of AVX2 optimized Fletcher-4 2016-06-02 14:30:51 -07:00
spa_impl.h OpenZFS 7090 - zfs should throttle allocations 2016-10-13 17:59:18 -07:00
space_map.h Illumos 5164-5165 - space map fixes 2014-10-23 15:30:32 -07:00
space_reftree.h Illumos #4101, #4102, #4103, #4105, #4106 2014-07-22 09:39:16 -07:00
sysevent.h OpenZFS 5997 - FRU field not set during pool creation and never updated 2016-08-12 13:06:48 -07:00
trace.h Remove duplicate typedefs from trace.h 2015-01-06 16:53:24 -08:00
trace_acl.h Fix interaction between userns uid/gid and SA 2016-08-08 10:47:43 -07:00
trace_arc.h OpenZFS 6950 - ARC should cache compressed data 2016-09-13 09:58:33 -07:00
trace_common.h OpenZFS 6531 - Provide mechanism to artificially limit disk performance 2016-05-26 10:11:51 -07:00
trace_dbgmsg.h SET_ERROR should print strings 2016-01-15 15:38:35 -08:00
trace_dbuf.h OpenZFS 6950 - ARC should cache compressed data 2016-09-13 09:58:33 -07:00
trace_dmu.h Fix build failure with Linux 4.1 and FTRACE 2015-07-29 07:35:06 -07:00
trace_dnode.h Fix build failure with Linux 4.1 and FTRACE 2015-07-29 07:35:06 -07:00
trace_multilist.h Fix build failure with Linux 4.1 and FTRACE 2015-08-18 16:47:21 -07:00
trace_txg.h Fix build failure with Linux 4.1 and FTRACE 2015-07-29 07:35:06 -07:00
trace_zil.h Fix build failure with Linux 4.1 and FTRACE 2015-07-29 07:35:06 -07:00
trace_zio.h OpenZFS 6531 - Provide mechanism to artificially limit disk performance 2016-05-26 10:11:51 -07:00
trace_zrlock.h SET_ERROR should print strings 2016-01-15 15:38:35 -08:00
txg.h Illumos 4753 - increase number of outstanding async writes when sync task is waiting 2014-09-23 13:50:55 -07:00
txg_impl.h Illumos #4045 write throttle & i/o scheduler performance work 2013-12-06 09:32:43 -08:00
u8_textprep.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
u8_textprep_data.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
uberblock.h Illumos 5347 - idle pool may run itself out of space 2015-07-14 10:35:21 -07:00
uberblock_impl.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
uio_impl.h Add basic uio support 2011-02-10 09:21:43 -08:00
unique.h Illumos #3742 2013-11-04 10:55:25 -08:00
uuid.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
vdev.h Add -lhHpw options to "zpool iostat" for avg latency, histograms, & queues 2016-05-12 12:36:32 -07:00
vdev_disk.h Remove custom root pool import code 2016-08-11 11:19:34 -07:00
vdev_file.h Update all default taskq settings 2015-06-25 08:58:16 -07:00
vdev_impl.h OpenZFS 7090 - zfs should throttle allocations 2016-10-13 17:59:18 -07:00
vdev_raidz.h Fixes and enhancements of SIMD raidz parity 2016-07-19 16:43:07 -07:00
vdev_raidz_impl.h Add parity generation/rebuild using 128-bits NEON for Aarch64 2016-10-03 09:44:00 -07:00
xvattr.h Add xvattr support 2011-03-02 11:43:50 -08:00
zap.h OpenZFS 7004 - dmu_tx_hold_zap() does dnode_hold() 7x on same object 2016-08-19 12:48:03 -07:00
zap_impl.h OpenZFS 7003 - zap_lockdir() should tag hold 2016-08-19 12:35:23 -07:00
zap_leaf.h Illumos 5056 - ZFS deadlock on db_mtx and dn_holds 2015-04-28 16:25:34 -07:00
zfeature.h Revert "zhack: Add 'feature disable' command" 2016-05-17 11:52:07 -07:00
zfs_acl.h Illumos #3742 2013-11-04 10:55:25 -08:00
zfs_context.h Performance optimization of AVL tree comparator functions 2016-08-31 14:35:34 -07:00
zfs_ctldir.h Delete unused zfsctl_snapdir_inactive declaration 2016-08-30 14:33:40 -07:00
zfs_debug.h Add dbgmsg kstat 2015-09-04 16:08:14 -07:00
zfs_delay.h cstyle: Resolve C style issues 2013-12-18 16:46:35 -08:00
zfs_dir.h Prototype/structure update for Linux 2011-02-10 09:27:21 -08:00
zfs_fuid.h Prototype/structure update for Linux 2011-02-10 09:27:21 -08:00
zfs_ioctl.h DLPX-40252 integrate EP-476 compressed zfs send/receive 2016-09-13 09:58:58 -07:00
zfs_onexit.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
zfs_rlock.h Use zfs range locks in ztest 2016-05-17 10:40:30 -07:00
zfs_sa.h Illumos 5027 - zfs large block support 2015-05-11 12:23:16 -07:00
zfs_stat.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
zfs_vfsops.h Add support for user/group dnode accounting & quota 2016-10-07 09:45:13 -07:00
zfs_vnops.h Add pn_alloc()/pn_free() functions 2016-04-21 09:49:25 -07:00
zfs_znode.h Remove znode's z_uid/z_gid member 2016-07-25 13:21:49 -07:00
zil.h Fix a typo in ZIL write handling comment 2016-08-12 10:30:16 -07:00
zil_impl.h Illumos 5027 - zfs large block support 2015-05-11 12:23:16 -07:00
zio.h OpenZFS 7090 - zfs should throttle allocations 2016-10-13 17:59:18 -07:00
zio_checksum.h OpenZFS 4185 - add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R 2016-10-03 14:51:15 -07:00
zio_compress.h DLPX-40252 integrate EP-476 compressed zfs send/receive 2016-09-13 09:58:58 -07:00
zio_impl.h OpenZFS 7090 - zfs should throttle allocations 2016-10-13 17:59:18 -07:00
zio_priority.h Add -lhHpw options to "zpool iostat" for avg latency, histograms, & queues 2016-05-12 12:36:32 -07:00
zpl.h Use file_dentry and file_inode wrappers 2016-08-11 12:06:37 -07:00
zrlock.h Support custom build directories and move includes 2010-09-08 12:38:56 -07:00
zvol.h OpenZFS 2605, 6980, 6902 2016-06-28 13:47:02 -07:00