The unlocked version of blk_end_request() should be used in the
zvol_request() error handling to avoid deadlocking the request_queue
queue_lock. Things get a little more complicated for older kernel
APIs but the compat layer has been updated as well to reflect this.
Links in /dev for volumes/snapshots were only being created
at volume/snapshot creation time. Those links are now also
created dynamically at module load time based on the spa config.
In other words all you need to do to get access to your zfs
datasets after a reboot is load the module. It's magic!
It turns out the zil allocates quite large buffers. This isn't
all the surprising but we need to suppress the warnings until
it's clear what to do about it.
These two allocations are 16k in size which trips the SPL warning
about large kmem_allocs(). For now simply shift them to a vmem_alloc().
Yes, this has it's own set of problems but this happens infrequently
enough not to be a real issue. But more importantly it means that
we've flagged this place in the code via this topic branch as a
place which still needs long term work.
This includes adding the copyright where appropriate, author
attribution, and including the Lawrence Livermore National
Security, LLC release code for new/rewritten files.
This topic branch leverages the Solaris style FMA call points
in ZFS to create a user space visible event notification system
under Linux. This new system is called zevent and it unifies
all previous Solaris style ereports and sysevent notifications.
Under this Linux specific scheme when a sysevent or ereport event
occurs an nvlist describing the event is created which looks almost
exactly like a Solaris ereport. These events are queued up in the
kernel when they occur and conditionally logged to the console.
It is then up to a user space application to consume the events
and do whatever it likes with them.
To make this possible the existing /dev/zfs ABI has been extended
with two new ioctls which behave as follows.
* ZFS_IOC_EVENTS_NEXT
Get the next pending event. The kernel will keep track of the last
event consumed by the file descriptor and provide the next one if
available. If no new events are available the ioctl() will block
waiting for the next event. This ioctl may also be called in a
non-blocking mode by setting zc.zc_guid = ZEVENT_NONBLOCK. In the
non-blocking case if no events are available ENOENT will be returned.
It is possible that ESHUTDOWN will be returned if the ioctl() is
called while module unloading is in progress. And finally ENOMEM
may occur if the provided nvlist buffer is not large enough to
contain the entire event.
* ZFS_IOC_EVENTS_CLEAR
Clear are events queued by the kernel. The kernel will keep a fairly
large number of recent events queued, use this ioctl to clear the
in kernel list. This will effect all user space processes consuming
events.
The zpool command has been extended to use this events ABI with the
'events' subcommand. You may run 'zpool events -v' to output a
verbose log of all recent events. This is very similar to the
Solaris 'fmdump -ev' command with the key difference being it also
includes what would be considered sysevents under Solaris. You
may also run in follow mode with the '-f' option. To clear the
in kernel event queue use the '-c' option.
$ sudo cmd/zpool/zpool events -fv
TIME CLASS
May 13 2010 16:31:15.777711000 ereport.fs.zfs.config.sync
class = "ereport.fs.zfs.config.sync"
ena = 0x40982b7897700001
detector = (embedded nvlist)
version = 0x0
scheme = "zfs"
pool = 0xed976600de75dfa6
(end detector)
time = 0x4bec8bc3 0x2e5aed98
pool = "zpios"
pool_guid = 0xed976600de75dfa6
pool_context = 0x0
While the 'zpool events' command is handy for interactive debugging
it is not expected to be the primary consumer of zevents. This ABI
was primarily added to facilitate the addition of a user space
monitoring daemon. This daemon would consume all events posted by
the kernel and based on the type of event perform an action. For
most events simply forwarding them on to syslog is likely enough.
But this interface also cleanly allows for more sophisticated
actions to be taken such as generating an email for a failed drive
This branch contains the majority of the changes required to cleanly
intergrate with Linux style special devices (/dev/zfs). Mainly this
means dropping all the Solaris style callbacks and replacing them
with the Linux equivilants. Additionally, this means slightly
expanding the zfs_ioc_* functions to include a 'struct file *' to
allow the tracking of private data for each open file descriptor.
This is very helpful when you need to maintain a little information
about the open file between ioctls().
All the changes required to cleanly integrate with linux style
devices are being moved to their own topic branch. It was abouti
to get to the point where too many topic branches each wanted to
tweak this code and it was getting confusing. To simplify this
all /dev/zfs device related changes are being moved to the
linux-kernel-device topic branch.
Remove the few places in the code where zfs_config.h is included.
It is now added to the gcc compile line to ensure the config results
are always available.
This was leading to a compilation error because in Linux, sizeof
(kmutex_t) can be larger than 64 in some circumstances (e.g. with
debugging and lockdep enabled).
The code was previously fixed to align the structure to 256 bytes,
but a better fix is to align it to the next multiple of 64 bytes.
The previous hack for this was to move headers 3 levels deep in to
the top level include dir and headers 4 levels deep in to sys.
Obviously this was fairly limiting and missed two important headers.
./zfs/include/sys/fm/fs/zfs.h
./zfs/include/sys/fs/zfs.h
This patch updates the code to be smarter and installs all the headers
to the correct location regardless of depth. Each header will have its
leading 3 path args stripped (i.e. ./zfs/include/) and replaced with
the correct install destination. All path information past the first
three levels will be preserved.
Recent builds against 2.6.31 flagged dmu_recv_stream() as stack heavy.
As a quick simple way to resolve this I'm preventing the inlining of
certain functions which gcc will inline here because this is the only
place they are called. Futher analysis of this function should be
performed to futher reduce its stack usage.
For 2.6.27 kernels are earlier revalidate_disk() was not available.
However, check_disk_change() has been available for far longer and
will properly inform the kernel of the volume change for both older
and newer kernels.
At last a useful user space interface for the Linux ZFS port arrives.
With the addition of the ZVOL real ZFS based block devices are available
and can be compared head to head with Linux's MD and LVM block drivers.
The Linux ZVOL has not yet had any performance work done but from a user
perspective it should be functionally complete and behave like any other
Linux block device.
The ZVOL has so far been tested using zconfig.sh on the following x86_64
based platforms: FC11, CHAOS4, RHEL5, RHEL6, and SLES11. However, more
testing is required to ensure everything is working as designed.
What follows in a somewhat detailed list of changes includes in this
commit to make ZVOL's possible. A few other issues were addressed in
the context of these changes which will also be mentioned.
* Added module/zfs/zvol.c which is based off the original Solaris ZVOL
implementation but rewritten to intergrate with the Linux block device
APIs. The basic design remains the similar in Linux with the major
change being request processing. Request processing is handled by
registering a request function which the elevator calls once all request
merges is finished and the elevator unplugs. This function is called
under a spin lock and the request structure is passed to the block driver
to be queued for IO. The elevator must be notified asyncronously once
the request completes or fails with an error. This allows us the block
driver a chance to handle many request concurrently. For the ZVOL we
maintain a taskq with a service thread per core. As requests are delivered
by the elevator each request is dispatched to the taskq. The task queue
handles each request with a write or read helper function which basically
copies the request data in to our out of the DMU object. Writes single
completion as soon as the DMU has the data unless they are marked sync.
Reads are all handled syncronously however the elevator will merge many
small reads in to a large read before it submitting the request.
* Cachine is worth specifically mentioning. Because both the Linux VFS
and the ZFS ARC both want to fully manage the cache we unfortunately
end up with two caches. This means our memory foot print is larger
than otherwise expected, and it means we have an extra copy between
the caches, but it does not impact correctness. All syncs are barrior
requests I believe are handled correctly. Longer term there is lots of
room for improvement here but it will require fairly extensive changes
to either the Linux VFS and VM layer, or additional DMU interfaces to
handle managing buffer not directly allocated by the ARC.
* Added module/zfs/include/sys/blkdev.h which contains all the Linux
compatibility foo which is required to handle changes in the Linux block
APIs from 2.6.18 thru 2.6.31 based kernels.
* The dmu_{read,write}_uio interfaces which don't make sense on Linux
have been modified to dmu_{read,write}_req functions which consume the
standard Linux IO request structure. Their function fundamentally
remains the same so this happily worked out pretty cleanly.
* The /dev/zfs character device is no longer created through the half
implemented Solaris driver DDI interfaces. It is now simply created
with it's own major number as a Linux misc device which greatly simplifies
everything. It is only capable of handling ioctls() but this fits nicely
because that's all it ever has to do. The ZVOL devices unlike in Solaris
do not leverage the same major number as /dev/zfs but instead register
their own major. Because only one major is allocated and space is reserved
for 16 partitions per-device there is a limit of 16384 concurrent ZVOL
devices. By using multiple majors like the scsi driver this limit could
be addressed if it becomes a problem.
* The {spa,zfs,zvol}_busy() functions have all be removed because they
are not required on a Linux system. Under Linux the registered module
exit function will not be called while the are still references to the
module. Once the exit function is called however it must succeed or
block, it may not fail so returning an error on module unload makes to
sense under Linux.
* With the addition of ZVOL support all the HAVE_ZVOL defines were removed
for obvious reasons. However, the HAVE_ZPL defines have been relocated
in to the linux-{kernel,user}-disk topic branches and must remain until
the ZPL is implemented.
Brian Behlendorf
Ricardo M. Correia