Archive-Team
/
zfs
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 103 Wiki Activity

Debian: Add an experimental HOWTO 

This includes native encryption support.
Richard Laager 2019-04-10 14:03:42 -05:00
parent 28ed40d37e
commit 4002ecbed0
1 changed files with 681 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

681
Debian-Buster-Encrypted-Root-on-ZFS.md Normal file

@ -0,0 +1,681 @@
### Caution
* This HOWTO uses a whole physical disk.
* Do not use these instructions for dual-booting.
* Backup your data. Any existing data will be lost.
### System Requirements
* [64-bit Debian GNU/Linux Buster Alpha1 Live CD](https://cdimage.debian.org/mirror/cdimage/buster_di_alpha1-live/amd64/iso-hybrid/)
* [A 64-bit kernel is *strongly* encouraged.](https://github.com/zfsonlinux/zfs/wiki/FAQ#32-bit-vs-64-bit-systems)
* Installing on a drive which presents 4KiB logical sectors (a “4Kn” drive) only works with UEFI booting. This not unique to ZFS. [GRUB does not and will not work on 4Kn with legacy (BIOS) booting.](http://savannah.gnu.org/bugs/?46700)
Computers that have less than 2 GiB of memory run ZFS slowly. 4 GiB of memory is recommended for normal performance in basic workloads. If you wish to use deduplication, you will need [massive amounts of RAM](http://wiki.freebsd.org/ZFSTuningGuide#Deduplication). Enabling deduplication is a permanent change that cannot be easily reverted.
## Support
If you want a supported system, see [[Debian Buster Root on ZFS]].
This *experimental* HOWTO uses 0.8.0~rc3-1 packages from Debian experimental, rebuilt from git. That version of ZFS is not a stable release. The packages are not a final release. The packages are not in the Buster repository. (They have not even landed in experimental, as they are in the NEW queue.) Do not ask for support with this version of the HOWTO unless you are able to help with development.
If you need help, reach out to the community using the [zfs-discuss mailing list](http://list.zfsonlinux.org/mailman/listinfo/zfs-discuss) or IRC at #zfsonlinux on [freenode](https://freenode.net/). If you have a bug report or feature request related to this HOWTO, please [file a new issue](https://github.com/zfsonlinux/zfs/issues/new) and mention @rlaager.
## Encryption
This guide supports two different encryption options: unencrypted and ZFS native encryption.
## Step 1: Prepare The Install Environment
1.1 Boot the Debian GNU/Linux Live CD. If prompted, login with the username `user` and password `live`.
1.2 Optional: Install and start the OpenSSH server in the Live CD environment:
If you have a second system, using SSH to access the target system can be convenient.
$ sudo apt update
$ sudo apt install --yes openssh-server
$ sudo sed -i "s/#Password/Password/" /etc/ssh/sshd_config
$ sudo systemctl restart ssh
**Hint:** You can find your IP address with `ip addr show scope global | grep inet`. Then, from your main machine, connect with `ssh user@IP`.
1.3 Become root:
$ sudo -i
1.4 Setup and update the repositories:
# echo deb http://deb.debian.org/debian buster contrib >> /etc/apt/sources.list
# apt update
1.5 Install ZFS in the Live CD environment:
**Warning:** If you do not have at least 6 GiB of RAM, this will not work, as the Live CD overlay will run out of space, mainly due to all the package updates.
# apt install --yes gcc=4:7.1.0-2 debootstrap gdisk dpkg-dev linux-headers-$(uname -r)
# apt install --yes git-buildpackage tar libattr1-dev libblkid-dev \
libselinux1-dev libssl-dev python3-cffi python3-setuptools \
python3-sphinx python3-all-dev uuid-dev zlib1g-dev
# apt install --yes --download-only build-essential
# dpkg --force-all --install /var/cache/apt/archives/build-essential_12.6_amd64.deb
# git clone https://salsa.debian.org/zfsonlinux-team/zfs.git
# cd zfs
# git checkout pristine-tar
# git checkout experimental
# gbp buildpackage --git-debian-branch=experimental -uc -us
# cd ..
# dpkg --install \
libnvpair1linux_0.8.0~rc3-1_amd64.deb \
libuutil1linux_0.8.0~rc3-1_amd64.deb \
libzfs2linux_0.8.0~rc3-1_amd64.deb \
libzpool2linux_0.8.0~rc3-1_amd64.deb \
spl_0.8.0~rc3-1_all.deb \
spl-dkms_0.8.0~rc3-1_all.deb \
zfs-dkms_0.8.0~rc3-1_all.deb \
zfsutils-linux_0.8.0~rc3-1_amd64.deb \
zfs-zed_0.8.0~rc3-1_amd64.deb
# modprobe zfs
* Until the kernel is updated in the live CD, we need to stay on the old version of gcc and force the install of build-essential so we stay on the old version of binutils. Otherwise, the resulting kernel modules will fail to load.
## Step 2: Disk Formatting
2.1 If you are re-using a disk, clear it as necessary:
If the disk was previously used in an MD array, zero the superblock:
# apt install --yes mdadm
# mdadm --zero-superblock --force /dev/disk/by-id/scsi-SATA_disk1
Clear the partition table:
# sgdisk --zap-all /dev/disk/by-id/scsi-SATA_disk1
2.2 Partition your disk(s):
Run this if you need legacy (BIOS) booting:
# sgdisk -a1 -n1:24K:+1000K -t1:EF02 /dev/disk/by-id/scsi-SATA_disk1
Run this for UEFI booting (for use now or in the future):
# sgdisk -n2:1M:+512M -t2:EF00 /dev/disk/by-id/scsi-SATA_disk1
Run this for the boot pool:
# sgdisk -n3:0:+512M -t3:BF01 /dev/disk/by-id/scsi-SATA_disk1
Run this for the root pool:
# sgdisk -n4:0:0 -t4:BF01 /dev/disk/by-id/scsi-SATA_disk1
Always use the long `/dev/disk/by-id/*` aliases with ZFS. Using the `/dev/sd*` device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool.
**Hints:**
* `ls -la /dev/disk/by-id` will list the aliases.
* Are you doing this in a virtual machine? If your virtual disk is missing from `/dev/disk/by-id`, use `/dev/vda` if you are using KVM with virtio; otherwise, read the [troubleshooting](#troubleshooting) section.
* If you are creating a mirror or raidz topology, repeat the partitioning commands for all the disks which will be part of the pool.
2.3 Create the boot pool:
# zpool create -o ashift=12 -d \
-o feature@async_destroy=enabled \
-o feature@bookmarks=enabled \
-o feature@embedded_data=enabled \
-o feature@empty_bpobj=enabled \
-o feature@enabled_txg=enabled \
-o feature@extensible_dataset=enabled \
-o feature@filesystem_limits=enabled \
-o feature@hole_birth=enabled \
-o feature@large_blocks=enabled \
-o feature@lz4_compress=enabled \
-o feature@spacemap_histogram=enabled \
-o feature@userobj_accounting=enabled \
-o feature@zpool_checkpoint=enabled \
-o feature@spacemap_v2=enabled \
-o feature@project_quota=enabled \
-o feature@resilver_defer=enabled \
-o feature@allocation_classes=enabled \
-O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off \
-O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/ -R /mnt \
bpool /dev/disk/by-id/scsi-SATA_disk1-part3
You should not need to customize any of the options for the boot pool.
GRUB does not support all of the zpool features. See `spa_feature_names` in [grub-core/fs/zfs/zfs.c](http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/zfs/zfs.c#n276). This step creates a separate boot pool for `/boot` with the features limited to only those that GRUB supports, allowing the root pool to use any/all features. Note that GRUB opens the pool read-only, so all read-only compatible features are "supported" by GRUB.
**Hints:**
* If you are creating a mirror or raidz topology, create the pool using `zpool create ... rpool mirror /dev/disk/by-id/scsi-SATA_disk1-part3 /dev/disk/by-id/scsi-SATA_disk2-part3` (or replace `mirror` with `raidz`, `raidz2`, or `raidz3` and list the partitions from additional disks).
* The pool name is arbitrary. If changed, the new name must be used consistently. The `bpool` convention originated in this HOWTO.
2.4 Create the root pool:
Choose one of the following options:
2.4a Unencrypted:
# zpool create -o ashift=12 \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \
-O mountpoint=/ -R /mnt \
rpool /dev/disk/by-id/scsi-SATA_disk1-part4
2.4b Encrypted:
# zpool create -o ashift=12 \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \
-O encryption=aes-128-gcm -O keylocation=prompt -O keyformat=passphrase \
-O mountpoint=/ -R /mnt \
rpool /dev/disk/by-id/scsi-SATA_disk1-part4
* The use of `ashift=12` is recommended here because many drives today have 4KiB (or larger) physical sectors, even though they present 512B logical sectors. Also, a future replacement drive may have 4KiB physical sectors (in which case `ashift=12` is desirable) or 4KiB logical sectors (in which case `ashift=12` is required).
* Setting `-O acltype=posixacl` enables POSIX ACLs globally. If you do not want this, remove that option, but later add `-o acltype=posixacl` (note: lowercase "o") to the `zfs create` for `/var/log`, as [journald requires ACLs](https://askubuntu.com/questions/970886/journalctl-says-failed-to-search-journal-acl-operation-not-supported)
* Setting `normalization=formD` eliminates some corner cases relating to UTF-8 filename normalization. It also implies `utf8only=on`, which means that only UTF-8 filenames are allowed. If you care to support non-UTF-8 filenames, do not use this option. For a discussion of why requiring UTF-8 filenames may be a bad idea, see [The problems with enforced UTF-8 only filenames](http://utcc.utoronto.ca/~cks/space/blog/linux/ForcedUTF8Filenames).
* Setting `relatime=on` is a middle ground between classic POSIX `atime` behavior (with its significant performance impact) and `atime=off` (which provides the best performance by completely disabling atime updates). Since Linux 2.6.30, `relatime` has been the default for other filesystems. See [RedHat's documentation](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/power_management_guide/relatime) for further information.
* Setting `xattr=sa` [vastly improves the performance of extended attributes](https://github.com/zfsonlinux/zfs/commit/82a37189aac955c81a59a5ecc3400475adb56355). Inside ZFS, extended attributes are used to implement POSIX ACLs. Extended attributes can also be used by user-space applications. [They are used by some desktop GUI applications.](https://en.wikipedia.org/wiki/Extended_file_attributes#Linux) [They can be used by Samba to store Windows ACLs and DOS attributes; they are required for a Samba Active Directory domain controller.](https://wiki.samba.org/index.php/Setting_up_a_Share_Using_Windows_ACLs) Note that [`xattr=sa` is Linux-specific.](http://open-zfs.org/wiki/Platform_code_differences) If you move your `xattr=sa` pool to another OpenZFS implementation besides ZFS-on-Linux, extended attributes will not be readable (though your data will be). If portability of extended attributes is important to you, omit the `-O xattr=sa` above. Even if you do not want `xattr=sa` for the whole pool, it is probably fine to use it for `/var/log`.
* Make sure to include the `-part4` portion of the drive path. If you forget that, you are specifying the whole disk, which ZFS will then re-partition, and you will lose the bootloader partition(s).
* ZFS uses `aes-256-ccm` by default. AES-GCM seems to be generally preferred over AES-CCM elsewhere, and is likely faster. This HOWTO uses AES-128, which matches the LUKS configuration from previous versions, which was used because it was the LUKS and Ubuntu default. You may wish to adjust this to `aes-256-gcm` or back to `aes-256-ccm`.
* Your passphrase will likely be the weakest link. Choose wisely. See [section 5 of the cryptsetup FAQ](https://gitlab.com/cryptsetup/cryptsetup/wikis/FrequentlyAskedQuestions#5-security-aspects) for guidance.
**Hints:**
* If you are creating a mirror or raidz topology, create the pool using `zpool create ... rpool mirror /dev/disk/by-id/scsi-SATA_disk1-part4 /dev/disk/by-id/scsi-SATA_disk2-part4` (or replace `mirror` with `raidz`, `raidz2`, or `raidz3` and list the partitions from additional disks).
* The pool name is arbitrary. If changed, the new name must be used consistently. On systems that can automatically install to ZFS, the root pool is named `rpool` by default.
## Step 3: System Installation
3.1 Create filesystem datasets to act as containers:
# zfs create -o canmount=off -o mountpoint=none rpool/ROOT
# zfs create -o canmount=off -o mountpoint=none bpool/BOOT
On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through `pkg image-update` or `beadm`. Similar functionality for APT is possible but currently unimplemented. Even without such a tool, it can still be used for manually created clones.
3.2 Create filesystem datasets for the root and boot filesystems:
# zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/debian
# zfs mount rpool/ROOT/debian
# zfs create -o canmount=noauto -o mountpoint=/boot bpool/BOOT/debian
# zfs mount bpool/BOOT/debian
With ZFS, it is not normally necessary to use a mount command (either `mount` or `zfs mount`). This situation is an exception because of `canmount=noauto`.
3.3 Create datasets:
# zfs create rpool/home
# zfs create -o mountpoint=/root rpool/home/root
# zfs create -o canmount=off rpool/var
# zfs create -o canmount=off rpool/var/lib
# zfs create rpool/var/log
# zfs create rpool/var/spool
The datasets below are optional, depending on your preferences and/or
software choices:
If you wish to exclude these from snapshots:
# zfs create -o com.sun:auto-snapshot=false rpool/var/cache
# zfs create -o com.sun:auto-snapshot=false rpool/var/tmp
# chmod 1777 /mnt/var/tmp
If you use /opt on this system:
# zfs create rpool/opt
If you use /srv on this system:
# zfs create rpool/srv
If you use /usr/local on this system:
# zfs create -o canmount=off rpool/usr
# zfs create rpool/usr/local
If this system will have games installed:
# zfs create rpool/var/games
If this system will store local email in /var/mail:
# zfs create rpool/var/mail
If this system will use Snap packages:
# zfs create rpool/var/snap
If you use /var/www on this system:
# zfs create rpool/var/www
If this system will use GNOME:
# zfs create rpool/var/lib/AccountsService
If this system will use Docker (which manages its own datasets & snapshots):
# zfs create -o com.sun:auto-snapshot=false rpool/var/lib/docker
If this system will use NFS (locking):
# zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs
A tmpfs is recommended later, but if you want a separate dataset for /tmp:
# zfs create -o com.sun:auto-snapshot=false rpool/tmp
# chmod 1777 /mnt/tmp
The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data such as logs (in `/var/log`). This will be especially important if/when a `beadm` or similar utility is integrated. The `com.sun.auto-snapshot` setting is used by some ZFS snapshot utilities to exclude transient data.
If you do nothing extra, `/tmp` will be stored as part of the root filesystem. Alternatively, you can create a separate dataset for `/tmp`, as shown above. This keeps the `/tmp` data out of snapshots of your root filesystem. It also allows you to set a quota on `rpool/tmp`, if you want to limit the maximum space used. Otherwise, you can use a tmpfs (RAM filesystem) later.
3.4 Install the minimal system:
# debootstrap buster /mnt
# zfs set devices=off rpool
The `debootstrap` command leaves the new system in an unconfigured state. An alternative to using `debootstrap` is to copy the entirety of a working system into the new ZFS root.
## Step 4: System Configuration
4.1 Configure the hostname (change `HOSTNAME` to the desired hostname).
# echo HOSTNAME > /mnt/etc/hostname
# vi /mnt/etc/hosts
Add a line:
127.0.1.1 HOSTNAME
or if the system has a real name in DNS:
127.0.1.1 FQDN HOSTNAME
**Hint:** Use `nano` if you find `vi` confusing.
4.2 Configure the network interface:
Find the interface name:
# ip addr show
# vi /mnt/etc/network/interfaces.d/NAME
auto NAME
iface NAME inet dhcp
Customize this file if the system is not a DHCP client.
4.3 Configure the package sources:
# vi /mnt/etc/apt/sources.list
deb http://deb.debian.org/debian buster main contrib
deb-src http://deb.debian.org/debian buster main contrib
4.4 Bind the virtual filesystems from the LiveCD environment to the new system and `chroot` into it:
# mount --rbind /dev /mnt/dev
# mount --rbind /proc /mnt/proc
# mount --rbind /sys /mnt/sys
# chroot /mnt /bin/bash --login
**Note:** This is using `--rbind`, not `--bind`.
4.5 Configure a basic system environment:
# ln -s /proc/self/mounts /etc/mtab
# apt update
# apt install --yes locales
# dpkg-reconfigure locales
Even if you prefer a non-English system language, always ensure that `en_US.UTF-8` is available.
# dpkg-reconfigure tzdata
4.6 Install ZFS in the chroot environment for the new system:
# apt install --yes dpkg-dev linux-headers-amd64 linux-image-amd64
# apt install --yes git-buildpackage build-essential dkms libattr1-dev \
libblkid-dev libselinux1-dev libssl-dev python3-cffi python3-setuptools \
python3-sphinx python3-all-dev uuid-dev zlib1g-dev
# git clone https://salsa.debian.org/zfsonlinux-team/zfs.git
# cd zfs
# git checkout pristine-tar
# git checkout experimental
# gbp buildpackage --git-debian-branch=experimental -uc -us
# cd ..
# dpkg --install \
libnvpair1linux_0.8.0~rc3-1_amd64.deb \
libuutil1linux_0.8.0~rc3-1_amd64.deb \
libzfs2linux_0.8.0~rc3-1_amd64.deb \
libzpool2linux_0.8.0~rc3-1_amd64.deb \
spl_0.8.0~rc3-1_all.deb \
spl-dkms_0.8.0~rc3-1_all.deb \
zfs-dkms_0.8.0~rc3-1_all.deb \
zfs-initramfs_0.8.0~rc3-1_all.deb \
zfsutils-linux_0.8.0~rc3-1_amd64.deb \
zfs-zed_0.8.0~rc3-1_amd64.deb
4.7 Install GRUB
Choose one of the following options:
4.7a Install GRUB for legacy (BIOS) booting
# apt install --yes grub-pc
Install GRUB to the disk(s), not the partition(s).
4.7b Install GRUB for UEFI booting
# apt install dosfstools
# mkdosfs -F 32 -s 1 -n EFI /dev/disk/by-id/scsi-SATA_disk1-part2
# mkdir /boot/efi
# echo PARTUUID=$(blkid -s PARTUUID -o value \
/dev/disk/by-id/scsi-SATA_disk1-part2) \
/boot/efi vfat nofail,x-systemd.device-timeout=1 0 1 >> /etc/fstab
# mount /boot/efi
# apt install --yes grub-efi-amd64 shim
* The `-s 1` for `mkdosfs` is only necessary for drives which present 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size (given the partition size of 512 MiB) for FAT32. It also works fine on drives which present 512 B sectors.
4.8 Set a root password
# passwd
4.9 Enable importing bpool
This ensures that `bpool` is always imported, regardless of whether `/etc/zfs/zpool.cache` exists, whether it is in the cachefile or not, or whether `zfs-import-scan.service` is enabled.
```
# vi /etc/systemd/system/zfs-import-bpool.service
[Unit]
DefaultDependencies=no
Before=zfs-import-scan.service
Before=zfs-import-cache.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/zpool import -N -o cachefile=none bpool
[Install]
WantedBy=zfs-import.target
# systemctl enable zfs-import-bpool.service
```
4.10 Optional (but recommended): Mount a tmpfs to /tmp
If you chose to create a `/tmp` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put `/tmp` on a tmpfs (RAM filesystem) by enabling the `tmp.mount` unit.
# cp /usr/share/systemd/tmp.mount /etc/systemd/system/
# systemctl enable tmp.mount
## Step 5: GRUB Installation
5.1 Verify that the ZFS boot filesystem is recognized:
# grub-probe /boot
zfs
5.2 Refresh the initrd files:
# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-4.19.0-4-amd64
5.3 Workaround GRUB's missing zpool-features support:
# vi /etc/default/grub
Set: GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/debian"
5.4 Optional (but highly recommended): Make debugging GRUB easier:
# vi /etc/default/grub
Remove quiet from: GRUB_CMDLINE_LINUX_DEFAULT
Uncomment: GRUB_TERMINAL=console
Save and quit.
Later, once the system has rebooted twice and you are sure everything is working, you can undo these changes, if desired.
5.5 Update the boot configuration:
# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.19.0-4-amd64
Found initrd image: /boot/initrd.img-4.19.0-4-amd64
done
* Ignore errors from `osprober`, if present.
5.6 Install the boot loader
5.6a For legacy (BIOS) booting, install GRUB to the MBR:
# grub-install /dev/disk/by-id/scsi-SATA_disk1
Installing for i386-pc platform.
Installation finished. No error reported.
Do not reboot the computer until you get exactly that result message. Note that you are installing GRUB to the whole disk, not a partition.
If you are creating a mirror or raidz topology, repeat the `grub-install` command for each disk in the pool.
5.6b For UEFI booting, install GRUB:
# grub-install --target=x86_64-efi --efi-directory=/boot/efi \
--bootloader-id=debian --recheck --no-floppy
5.7 Verify that the ZFS module is installed:
# ls /boot/grub/*/zfs.mod
5.8 Fix filesystem mount ordering
Until there is support for mounting `/boot` in the initramfs, we also need to mount that, because it was marked `canmount=noauto`. Also, with UEFI, we need to ensure it is mounted before its child filesystem `/boot/efi`.
We need to activate `zfs-mount-generator`. This makes systemd aware of the separate mountpoints, which is important for things like `/var/log` and `/var/tmp`. In turn, `rsyslog.service` depends on `var-log.mount` by way of `local-fs.target` and services using the `PrivateTmp` feature of systemd automatically use `After=var-tmp.mount`.
For UEFI booting, unmount /boot/efi first:
# umount /boot/efi
Everything else applies to both BIOS and UEFI booting:
# zfs set mountpoint=legacy bpool/BOOT/debian
# echo bpool/BOOT/debian /boot zfs \
nodev,relatime,x-systemd.requires=zfs-import-bpool.service 0 0 >> /etc/fstab
# mkdir /etc/zfs/zfs-list.cache
# touch /etc/zfs/zfs-list.cache/rpool
# ln -s /usr/lib/zfs-linux/zed.d/history_event-zfs-list-cacher.sh /etc/zfs/zed.d
# zed -F &
Verify that zed updated the cache by making sure this is not empty:
# cat /etc/zfs/zfs-list.cache/rpool
If it is empty, force a cache update and check again:
# zfs set canmount=noauto rpool/ROOT/debian
Stop zed:
# killall zed
Fix the paths to eliminate /mnt:
# sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/rpool
## Step 6: First Boot
6.1 Snapshot the initial installation:
# zfs snapshot bpool/BOOT/debian@install
# zfs snapshot rpool/ROOT/debian@install
In the future, you will likely want to take snapshots before each upgrade, and remove old snapshots (including this one) at some point to save space.
6.2 Exit from the `chroot` environment back to the LiveCD environment:
# exit
6.3 Run these commands in the LiveCD environment to unmount all filesystems:
# mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
# zpool export -a
6.4 Reboot:
# reboot
6.5 Wait for the newly installed system to boot normally. Login as root.
6.6 Create a user account:
# zfs create rpool/home/YOURUSERNAME
# adduser YOURUSERNAME
# cp -a /etc/skel/.[!.]* /home/YOURUSERNAME
# chown -R YOURUSERNAME:YOURUSERNAME /home/YOURUSERNAME
6.7 Add your user account to the default set of groups for an administrator:
# usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video YOURUSERNAME
6.8 Mirror GRUB
If you installed to multiple disks, install GRUB on the additional disks:
6.8a For legacy (BIOS) booting:
# dpkg-reconfigure grub-pc
Hit enter until you get to the device selection screen.
Select (using the space bar) all of the disks (not partitions) in your pool.
6.8b UEFI
# umount /boot/efi
For the second and subsequent disks (increment debian-2 to -3, etc.):
# dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \
of=/dev/disk/by-id/scsi-SATA_disk2-part2
# efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \
-p 3 -L "debian-2" -l '\EFI\debian\grubx64.efi'
# mount /boot/efi
## Step 7: (Optional) Configure Swap
**Caution**: On systems with extremely high memory pressure, using a zvol for swap can result in lockup, regardless of how much swap is still available. This issue is currently being investigated in: https://github.com/zfsonlinux/zfs/issues/7734
7.1 Create a volume dataset (zvol) for use as a swap device:
# zfs create -V 4G -b $(getconf PAGESIZE) -o compression=zle \
-o logbias=throughput -o sync=always \
-o primarycache=metadata -o secondarycache=none \
-o com.sun:auto-snapshot=false rpool/swap
You can adjust the size (the `4G` part) to your needs.
The compression algorithm is set to `zle` because it is the cheapest available algorithm. As this guide recommends `ashift=12` (4 kiB blocks on disk), the common case of a 4 kiB page size means that no compression algorithm can reduce I/O. The exception is all-zero pages, which are dropped by ZFS; but some form of compression has to be enabled to get this behavior.
7.2 Configure the swap device:
**Caution**: Always use long `/dev/zvol` aliases in configuration files. Never use a short `/dev/zdX` device name.
# mkswap -f /dev/zvol/rpool/swap
# echo /dev/zvol/rpool/swap none swap discard 0 0 >> /etc/fstab
# echo RESUME=none > /etc/initramfs-tools/conf.d/resume
The `RESUME=none` is necessary to disable resuming from hibernation. This does not work, as the zvol is not present (because the pool has not yet been imported) at the time the resume script runs. If it is not disabled, the boot process hangs for 30 seconds waiting for the swap zvol to appear.
7.3 Enable the swap device:
# swapon -av
## Step 8: Full Software Installation
8.1 Upgrade the minimal system:
# apt dist-upgrade --yes
8.2 Install a regular set of software:
# tasksel
8.3 Optional: Disable log compression:
As `/var/log` is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if you are making snapshots of `/var/log`, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in `/etc/logrotate.d` by hand to comment out `compress`, or use this loop (copy-and-paste highly recommended):
# for file in /etc/logrotate.d/* ; do
if grep -Eq "(^|[^#y])compress" "$file" ; then
sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file"
fi
done
8.4 Reboot:
# reboot
### Step 9: Final Cleanup
9.1 Wait for the system to boot normally. Login using the account you created. Ensure the system (including networking) works normally.
9.2 Optional: Delete the snapshots of the initial installation:
$ sudo zfs destroy bpool/BOOT/debian@install
$ sudo zfs destroy rpool/ROOT/debian@install
9.3 Optional: Disable the root password
$ sudo usermod -p '*' root
9.4 Optional:
If you prefer the graphical boot process, you can re-enable it now.
$ sudo vi /etc/default/grub
Add quiet to GRUB_CMDLINE_LINUX_DEFAULT
Comment out GRUB_TERMINAL=console
Save and quit.
$ sudo update-grub
* Ignore errors from `osprober`, if present.
## Troubleshooting
### Rescuing using a Live CD
Go through [Step 1: Prepare The Install Environment](#step-1-prepare-the-install-environment).
This may automatically import your pool. Export it and re-import it to get the mounts right:
# zpool export -a
# zpool import -N -R /mnt rpool
# zpool import -N -R /mnt bpool
# zfs load-key -a
# zfs mount rpool/ROOT/debian
# zfs mount -a
If needed, you can chroot into your installed environment:
# mount --rbind /dev /mnt/dev
# mount --rbind /proc /mnt/proc
# mount --rbind /sys /mnt/sys
# chroot /mnt /bin/bash --login
# mount /boot
# mount -a
Do whatever you need to do to fix your system.
When done, cleanup:
# exit
# mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
# zpool export -a
# reboot
### MPT2SAS
Most problem reports for this tutorial involve `mpt2sas` hardware that does slow asynchronous drive initialization, like some IBM M1015 or OEM-branded cards that have been flashed to the reference LSI firmware.
The basic problem is that disks on these controllers are not visible to the Linux kernel until after the regular system is started, and ZoL does not hotplug pool members. See https://github.com/zfsonlinux/zfs/issues/330.
Most LSI cards are perfectly compatible with ZoL. If your card has this glitch, try setting ZFS_INITRD_PRE_MOUNTROOT_SLEEP=X in /etc/default/zfs. The system will wait X seconds for all drives to appear before importing the pool.
### Areca
Systems that require the `arcsas` blob driver should add it to the `/etc/initramfs-tools/modules` file and run `update-initramfs -u -k all`.
Upgrade or downgrade the Areca driver if something like `RIP: 0010:[<ffffffff8101b316>] [<ffffffff8101b316>] native_read_tsc+0x6/0x20` appears anywhere in kernel log. ZoL is unstable on systems that emit this error message.
### VMware
* Set `disk.EnableUUID = "TRUE"` in the vmx file or vsphere configuration. Doing this ensures that `/dev/disk` aliases are created in the guest.
### QEMU/KVM/XEN
Set a unique serial number on each virtual disk using libvirt or qemu (e.g. `-drive if=none,id=disk1,file=disk1.qcow2,serial=1234567890`).
To be able to use UEFI in guests (instead of only BIOS booting), run this on the host:
$ sudo apt install ovmf
$ sudo vi /etc/libvirt/qemu.conf
Uncomment these lines:
nvram = [
"/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd",
"/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd"
]
$ sudo service libvirt-bin restart