diff --git a/Debian-Buster-Encrypted-Root-on-ZFS.md b/Debian-Buster-Encrypted-Root-on-ZFS.md index 2e1397f..c7138ee 100644 --- a/Debian-Buster-Encrypted-Root-on-ZFS.md +++ b/Debian-Buster-Encrypted-Root-on-ZFS.md @@ -2,11 +2,11 @@ This experimental guide has been made official at [[Debian Buster Root on ZFS]]. If you have an existing system installed from the experimental guide, adjust your sources: - # vi /etc/apt/sources.list.d/buster-backports.list + vi /etc/apt/sources.list.d/buster-backports.list deb http://deb.debian.org/debian buster-backports main contrib deb-src http://deb.debian.org/debian buster-backports main contrib - # vi /etc/apt/preferences.d/90_zfs + vi /etc/apt/preferences.d/90_zfs Package: libnvpair1linux libuutil1linux libzfs2linux libzpool2linux zfs-dkms zfs-initramfs zfs-test zfsutils-linux zfs-zed Pin: release n=buster-backports Pin-Priority: 990 @@ -15,19 +15,19 @@ This will allow you to upgrade from the locally-built packages to the official b You should set a root password before upgrading: - # passwd + passwd Apply updates: - # apt update - # apt dist-upgrade + apt update + apt dist-upgrade Reboot: - # reboot + reboot If the bpool fails to import, then enter the rescue shell (which requires a root password) and run: - # zpool import -f bpool - # zpool export bpool - # reboot + zpool import -f bpool + zpool export bpool + reboot diff --git a/Debian-Buster-Root-on-ZFS.md b/Debian-Buster-Root-on-ZFS.md index f92de34..b6fb718 100644 --- a/Debian-Buster-Root-on-ZFS.md +++ b/Debian-Buster-Root-on-ZFS.md @@ -41,27 +41,27 @@ ZFS native encryption encrypts the data and most metadata in the root pool. It d If you have a second system, using SSH to access the target system can be convenient. - $ sudo apt update - $ sudo apt install --yes openssh-server - $ sudo systemctl restart ssh + sudo apt update + sudo apt install --yes openssh-server + sudo systemctl restart ssh **Hint:** You can find your IP address with `ip addr show scope global | grep inet`. Then, from your main machine, connect with `ssh user@IP`. 1.3 Become root: - $ sudo -i + sudo -i 1.4 Setup and update the repositories: - # echo deb http://deb.debian.org/debian buster contrib >> /etc/apt/sources.list - # echo deb http://deb.debian.org/debian buster-backports main contrib >> /etc/apt/sources.list - # apt update + echo deb http://deb.debian.org/debian buster contrib >> /etc/apt/sources.list + echo deb http://deb.debian.org/debian buster-backports main contrib >> /etc/apt/sources.list + apt update 1.5 Install ZFS in the Live CD environment: - # apt install --yes debootstrap gdisk dkms dpkg-dev linux-headers-$(uname -r) - # apt install --yes -t buster-backports zfs-dkms - # modprobe zfs + apt install --yes debootstrap gdisk dkms dpkg-dev linux-headers-$(uname -r) + apt install --yes -t buster-backports zfs-dkms + modprobe zfs * The dkms dependency is installed manually just so it comes from buster and not buster-backports. This is not critical. @@ -69,33 +69,38 @@ If you have a second system, using SSH to access the target system can be conven 2.1 If you are re-using a disk, clear it as necessary: - If the disk was previously used in an MD array, zero the superblock: - # apt install --yes mdadm - # mdadm --zero-superblock --force /dev/disk/by-id/scsi-SATA_disk1 +If the disk was previously used in an MD array, zero the superblock: - Clear the partition table: - # sgdisk --zap-all /dev/disk/by-id/scsi-SATA_disk1 + apt install --yes mdadm + mdadm --zero-superblock --force /dev/disk/by-id/scsi-SATA_disk1 + +Clear the partition table: + + sgdisk --zap-all /dev/disk/by-id/scsi-SATA_disk1 2.2 Partition your disk(s): - Run this if you need legacy (BIOS) booting: - # sgdisk -a1 -n1:24K:+1000K -t1:EF02 /dev/disk/by-id/scsi-SATA_disk1 +Run this if you need legacy (BIOS) booting: - Run this for UEFI booting (for use now or in the future): - # sgdisk -n2:1M:+512M -t2:EF00 /dev/disk/by-id/scsi-SATA_disk1 + sgdisk -a1 -n1:24K:+1000K -t1:EF02 /dev/disk/by-id/scsi-SATA_disk1 - Run this for the boot pool: - # sgdisk -n3:0:+1G -t3:BF01 /dev/disk/by-id/scsi-SATA_disk1 +Run this for UEFI booting (for use now or in the future): + + sgdisk -n2:1M:+512M -t2:EF00 /dev/disk/by-id/scsi-SATA_disk1 + +Run this for the boot pool: + + sgdisk -n3:0:+1G -t3:BF01 /dev/disk/by-id/scsi-SATA_disk1 Choose one of the following options: 2.2a Unencrypted or ZFS native encryption: - # sgdisk -n4:0:0 -t4:BF01 /dev/disk/by-id/scsi-SATA_disk1 + sgdisk -n4:0:0 -t4:BF01 /dev/disk/by-id/scsi-SATA_disk1 2.2b LUKS: - # sgdisk -n4:0:0 -t4:8300 /dev/disk/by-id/scsi-SATA_disk1 + sgdisk -n4:0:0 -t4:8300 /dev/disk/by-id/scsi-SATA_disk1 Always use the long `/dev/disk/by-id/*` aliases with ZFS. Using the `/dev/sd*` device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool. @@ -106,28 +111,28 @@ Always use the long `/dev/disk/by-id/*` aliases with ZFS. Using the `/dev/sd*` 2.3 Create the boot pool: - # zpool create -o ashift=12 -d \ - -o feature@async_destroy=enabled \ - -o feature@bookmarks=enabled \ - -o feature@embedded_data=enabled \ - -o feature@empty_bpobj=enabled \ - -o feature@enabled_txg=enabled \ - -o feature@extensible_dataset=enabled \ - -o feature@filesystem_limits=enabled \ - -o feature@hole_birth=enabled \ - -o feature@large_blocks=enabled \ - -o feature@lz4_compress=enabled \ - -o feature@spacemap_histogram=enabled \ - -o feature@userobj_accounting=enabled \ - -o feature@zpool_checkpoint=enabled \ - -o feature@spacemap_v2=enabled \ - -o feature@project_quota=enabled \ - -o feature@resilver_defer=enabled \ - -o feature@allocation_classes=enabled \ - -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off \ - -O normalization=formD -O relatime=on -O xattr=sa \ - -O mountpoint=/ -R /mnt \ - bpool /dev/disk/by-id/scsi-SATA_disk1-part3 + zpool create -o ashift=12 -d \ + -o feature@async_destroy=enabled \ + -o feature@bookmarks=enabled \ + -o feature@embedded_data=enabled \ + -o feature@empty_bpobj=enabled \ + -o feature@enabled_txg=enabled \ + -o feature@extensible_dataset=enabled \ + -o feature@filesystem_limits=enabled \ + -o feature@hole_birth=enabled \ + -o feature@large_blocks=enabled \ + -o feature@lz4_compress=enabled \ + -o feature@spacemap_histogram=enabled \ + -o feature@userobj_accounting=enabled \ + -o feature@zpool_checkpoint=enabled \ + -o feature@spacemap_v2=enabled \ + -o feature@project_quota=enabled \ + -o feature@resilver_defer=enabled \ + -o feature@allocation_classes=enabled \ + -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off \ + -O normalization=formD -O relatime=on -O xattr=sa \ + -O mountpoint=/ -R /mnt \ + bpool /dev/disk/by-id/scsi-SATA_disk1-part3 You should not need to customize any of the options for the boot pool. @@ -143,32 +148,32 @@ Choose one of the following options: 2.4a Unencrypted: - # zpool create -o ashift=12 \ - -O acltype=posixacl -O canmount=off -O compression=lz4 \ - -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ - -O mountpoint=/ -R /mnt \ - rpool /dev/disk/by-id/scsi-SATA_disk1-part4 + zpool create -o ashift=12 \ + -O acltype=posixacl -O canmount=off -O compression=lz4 \ + -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ + -O mountpoint=/ -R /mnt \ + rpool /dev/disk/by-id/scsi-SATA_disk1-part4 2.4b LUKS: - # apt install --yes cryptsetup - # cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 \ - /dev/disk/by-id/scsi-SATA_disk1-part4 - # cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 - # zpool create -o ashift=12 \ - -O acltype=posixacl -O canmount=off -O compression=lz4 \ - -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ - -O mountpoint=/ -R /mnt \ - rpool /dev/mapper/luks1 + apt install --yes cryptsetup + cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 \ + /dev/disk/by-id/scsi-SATA_disk1-part4 + cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 + zpool create -o ashift=12 \ + -O acltype=posixacl -O canmount=off -O compression=lz4 \ + -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ + -O mountpoint=/ -R /mnt \ + rpool /dev/mapper/luks1 2.4c ZFS native encryption: - # zpool create -o ashift=12 \ - -O acltype=posixacl -O canmount=off -O compression=lz4 \ - -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ - -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \ - -O mountpoint=/ -R /mnt \ - rpool /dev/disk/by-id/scsi-SATA_disk1-part4 + zpool create -o ashift=12 \ + -O acltype=posixacl -O canmount=off -O compression=lz4 \ + -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ + -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase \ + -O mountpoint=/ -R /mnt \ + rpool /dev/disk/by-id/scsi-SATA_disk1-part4 * The use of `ashift=12` is recommended here because many drives today have 4KiB (or larger) physical sectors, even though they present 512B logical sectors. Also, a future replacement drive may have 4KiB physical sectors (in which case `ashift=12` is desirable) or 4KiB logical sectors (in which case `ashift=12` is required). * Setting `-O acltype=posixacl` enables POSIX ACLs globally. If you do not want this, remove that option, but later add `-o acltype=posixacl` (note: lowercase "o") to the `zfs create` for `/var/log`, as [journald requires ACLs](https://askubuntu.com/questions/970886/journalctl-says-failed-to-search-journal-acl-operation-not-supported) @@ -188,72 +193,84 @@ Choose one of the following options: 3.1 Create filesystem datasets to act as containers: - # zfs create -o canmount=off -o mountpoint=none rpool/ROOT - # zfs create -o canmount=off -o mountpoint=none bpool/BOOT + zfs create -o canmount=off -o mountpoint=none rpool/ROOT + zfs create -o canmount=off -o mountpoint=none bpool/BOOT On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through `pkg image-update` or `beadm`. Similar functionality for APT is possible but currently unimplemented. Even without such a tool, it can still be used for manually created clones. 3.2 Create filesystem datasets for the root and boot filesystems: - # zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/debian - # zfs mount rpool/ROOT/debian + zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/debian + zfs mount rpool/ROOT/debian - # zfs create -o canmount=noauto -o mountpoint=/boot bpool/BOOT/debian - # zfs mount bpool/BOOT/debian + zfs create -o canmount=noauto -o mountpoint=/boot bpool/BOOT/debian + zfs mount bpool/BOOT/debian With ZFS, it is not normally necessary to use a mount command (either `mount` or `zfs mount`). This situation is an exception because of `canmount=noauto`. 3.3 Create datasets: - # zfs create rpool/home - # zfs create -o mountpoint=/root rpool/home/root - # zfs create -o canmount=off rpool/var - # zfs create -o canmount=off rpool/var/lib - # zfs create rpool/var/log - # zfs create rpool/var/spool + zfs create rpool/home + zfs create -o mountpoint=/root rpool/home/root + zfs create -o canmount=off rpool/var + zfs create -o canmount=off rpool/var/lib + zfs create rpool/var/log + zfs create rpool/var/spool - The datasets below are optional, depending on your preferences and/or - software choices: +The datasets below are optional, depending on your preferences and/or software +choices. - If you wish to exclude these from snapshots: - # zfs create -o com.sun:auto-snapshot=false rpool/var/cache - # zfs create -o com.sun:auto-snapshot=false rpool/var/tmp - # chmod 1777 /mnt/var/tmp +If you wish to exclude these from snapshots: - If you use /opt on this system: - # zfs create rpool/opt + zfs create -o com.sun:auto-snapshot=false rpool/var/cache + zfs create -o com.sun:auto-snapshot=false rpool/var/tmp + chmod 1777 /mnt/var/tmp - If you use /srv on this system: - # zfs create rpool/srv +If you use /opt on this system: - If you use /usr/local on this system: - # zfs create -o canmount=off rpool/usr - # zfs create rpool/usr/local + zfs create rpool/opt - If this system will have games installed: - # zfs create rpool/var/games +If you use /srv on this system: - If this system will store local email in /var/mail: - # zfs create rpool/var/mail + zfs create rpool/srv - If this system will use Snap packages: - # zfs create rpool/var/snap +If you use /usr/local on this system: - If you use /var/www on this system: - # zfs create rpool/var/www + zfs create -o canmount=off rpool/usr + zfs create rpool/usr/local - If this system will use GNOME: - # zfs create rpool/var/lib/AccountsService +If this system will have games installed: - If this system will use Docker (which manages its own datasets & snapshots): - # zfs create -o com.sun:auto-snapshot=false rpool/var/lib/docker + zfs create rpool/var/games - If this system will use NFS (locking): - # zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs +If this system will store local email in /var/mail: - A tmpfs is recommended later, but if you want a separate dataset for /tmp: - # zfs create -o com.sun:auto-snapshot=false rpool/tmp - # chmod 1777 /mnt/tmp + zfs create rpool/var/mail + +If this system will use Snap packages: + + zfs create rpool/var/snap + +If you use /var/www on this system: + + zfs create rpool/var/www + +If this system will use GNOME: + + zfs create rpool/var/lib/AccountsService + +If this system will use Docker (which manages its own datasets & snapshots): + + zfs create -o com.sun:auto-snapshot=false rpool/var/lib/docker + +If this system will use NFS (locking): + + zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs + +A tmpfs is recommended later, but if you want a separate dataset for /tmp: + + zfs create -o com.sun:auto-snapshot=false rpool/tmp + chmod 1777 /mnt/tmp The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data such as logs (in `/var/log`). This will be especially important if/when a `beadm` or similar utility is integrated. The `com.sun.auto-snapshot` setting is used by some ZFS snapshot utilities to exclude transient data. @@ -261,8 +278,8 @@ If you do nothing extra, `/tmp` will be stored as part of the root filesystem. A 3.4 Install the minimal system: - # debootstrap buster /mnt - # zfs set devices=off rpool + debootstrap buster /mnt + zfs set devices=off rpool The `debootstrap` command leaves the new system in an unconfigured state. An alternative to using `debootstrap` is to copy the entirety of a working system into the new ZFS root. @@ -270,9 +287,9 @@ The `debootstrap` command leaves the new system in an unconfigured state. An al 4.1 Configure the hostname (change `HOSTNAME` to the desired hostname). - # echo HOSTNAME > /mnt/etc/hostname + echo HOSTNAME > /mnt/etc/hostname - # vi /mnt/etc/hosts + vi /mnt/etc/hosts Add a line: 127.0.1.1 HOSTNAME or if the system has a real name in DNS: @@ -282,10 +299,13 @@ The `debootstrap` command leaves the new system in an unconfigured state. An al 4.2 Configure the network interface: - Find the interface name: - # ip addr show +Find the interface name: - # vi /mnt/etc/network/interfaces.d/NAME + ip addr show + +Adjust NAME below to match your interface name: + + vi /mnt/etc/network/interfaces.d/NAME auto NAME iface NAME inet dhcp @@ -293,52 +313,52 @@ Customize this file if the system is not a DHCP client. 4.3 Configure the package sources: - # vi /mnt/etc/apt/sources.list + vi /mnt/etc/apt/sources.list deb http://deb.debian.org/debian buster main contrib deb-src http://deb.debian.org/debian buster main contrib - # vi /mnt/etc/apt/sources.list.d/buster-backports.list + vi /mnt/etc/apt/sources.list.d/buster-backports.list deb http://deb.debian.org/debian buster-backports main contrib deb-src http://deb.debian.org/debian buster-backports main contrib - # vi /mnt/etc/apt/preferences.d/90_zfs + vi /mnt/etc/apt/preferences.d/90_zfs Package: libnvpair1linux libuutil1linux libzfs2linux libzpool2linux zfs-dkms zfs-initramfs zfs-test zfsutils-linux zfsutils-linux-dev zfs-zed Pin: release n=buster-backports Pin-Priority: 990 4.4 Bind the virtual filesystems from the LiveCD environment to the new system and `chroot` into it: - # mount --rbind /dev /mnt/dev - # mount --rbind /proc /mnt/proc - # mount --rbind /sys /mnt/sys - # chroot /mnt /bin/bash --login + mount --rbind /dev /mnt/dev + mount --rbind /proc /mnt/proc + mount --rbind /sys /mnt/sys + chroot /mnt /bin/bash --login **Note:** This is using `--rbind`, not `--bind`. 4.5 Configure a basic system environment: - # ln -s /proc/self/mounts /etc/mtab - # apt update + ln -s /proc/self/mounts /etc/mtab + apt update - # apt install --yes locales - # dpkg-reconfigure locales + apt install --yes locales + dpkg-reconfigure locales Even if you prefer a non-English system language, always ensure that `en_US.UTF-8` is available. - # dpkg-reconfigure tzdata + dpkg-reconfigure tzdata 4.6 Install ZFS in the chroot environment for the new system: - # apt install --yes dpkg-dev linux-headers-amd64 linux-image-amd64 - # apt install --yes zfs-initramfs + apt install --yes dpkg-dev linux-headers-amd64 linux-image-amd64 + apt install --yes zfs-initramfs 4.7 For LUKS installs only, setup crypttab: - # apt install --yes cryptsetup + apt install --yes cryptsetup - # echo luks1 UUID=$(blkid -s UUID -o value \ - /dev/disk/by-id/scsi-SATA_disk1-part4) none \ - luks,discard,initramfs > /etc/crypttab + echo luks1 UUID=$(blkid -s UUID -o value \ + /dev/disk/by-id/scsi-SATA_disk1-part4) none \ + luks,discard,initramfs > /etc/crypttab * The use of `initramfs` is a work-around for [cryptsetup does not support ZFS](https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/1612906). @@ -350,20 +370,20 @@ Choose one of the following options: 4.8a Install GRUB for legacy (BIOS) booting - # apt install --yes grub-pc + apt install --yes grub-pc Install GRUB to the disk(s), not the partition(s). 4.8b Install GRUB for UEFI booting - # apt install dosfstools - # mkdosfs -F 32 -s 1 -n EFI /dev/disk/by-id/scsi-SATA_disk1-part2 - # mkdir /boot/efi - # echo PARTUUID=$(blkid -s PARTUUID -o value \ - /dev/disk/by-id/scsi-SATA_disk1-part2) \ - /boot/efi vfat nofail,x-systemd.device-timeout=1 0 1 >> /etc/fstab - # mount /boot/efi - # apt install --yes grub-efi-amd64 shim-signed + apt install dosfstools + mkdosfs -F 32 -s 1 -n EFI /dev/disk/by-id/scsi-SATA_disk1-part2 + mkdir /boot/efi + echo PARTUUID=$(blkid -s PARTUUID -o value \ + /dev/disk/by-id/scsi-SATA_disk1-part2) \ + /boot/efi vfat nofail,x-systemd.device-timeout=1 0 1 >> /etc/fstab + mount /boot/efi + apt install --yes grub-efi-amd64 shim-signed * The `-s 1` for `mkdosfs` is only necessary for drives which present 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size (given the partition size of 512 MiB) for FAT32. It also works fine on drives which present 512 B sectors. @@ -371,13 +391,13 @@ Install GRUB to the disk(s), not the partition(s). 4.9 Set a root password - # passwd + passwd 4.10 Enable importing bpool This ensures that `bpool` is always imported, regardless of whether `/etc/zfs/zpool.cache` exists, whether it is in the cachefile or not, or whether `zfs-import-scan.service` is enabled. ``` - # vi /etc/systemd/system/zfs-import-bpool.service + vi /etc/systemd/system/zfs-import-bpool.service [Unit] DefaultDependencies=no Before=zfs-import-scan.service @@ -391,21 +411,21 @@ This ensures that `bpool` is always imported, regardless of whether `/etc/zfs/zp [Install] WantedBy=zfs-import.target - # systemctl enable zfs-import-bpool.service + systemctl enable zfs-import-bpool.service ``` 4.11 Optional (but recommended): Mount a tmpfs to /tmp If you chose to create a `/tmp` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put `/tmp` on a tmpfs (RAM filesystem) by enabling the `tmp.mount` unit. - # cp /usr/share/systemd/tmp.mount /etc/systemd/system/ - # systemctl enable tmp.mount + cp /usr/share/systemd/tmp.mount /etc/systemd/system/ + systemctl enable tmp.mount 4.12 Optional (but kindly requested): Install popcon The `popularity-contest` package reports the list of packages install on your system. Showing that ZFS is popular may be helpful in terms of long-term attention from the distro. - # apt install --yes popularity-contest + apt install --yes popularity-contest Choose Yes at the prompt. @@ -413,24 +433,22 @@ Choose Yes at the prompt. 5.1 Verify that the ZFS boot filesystem is recognized: - # grub-probe /boot - zfs + grub-probe /boot 5.2 Refresh the initrd files: - # update-initramfs -u -k all - update-initramfs: Generating /boot/initrd.img-4.19.0-6-amd64 + update-initramfs -u -k all **Note:** When using LUKS, this will print "WARNING could not determine root device from /etc/fstab". This is because [cryptsetup does not support ZFS](https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/1612906). 5.3 Workaround GRUB's missing zpool-features support: - # vi /etc/default/grub + vi /etc/default/grub Set: GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/debian" 5.4 Optional (but highly recommended): Make debugging GRUB easier: - # vi /etc/default/grub + vi /etc/default/grub Remove quiet from: GRUB_CMDLINE_LINUX_DEFAULT Uncomment: GRUB_TERMINAL=console Save and quit. @@ -439,11 +457,7 @@ Later, once the system has rebooted twice and you are sure everything is working 5.5 Update the boot configuration: - # update-grub - Generating grub configuration file ... - Found linux image: /boot/vmlinuz-4.19.0-6-amd64 - Found initrd image: /boot/initrd.img-4.19.0-6-amd64 - done + update-grub **Note:** Ignore errors from `osprober`, if present. @@ -451,22 +465,20 @@ Later, once the system has rebooted twice and you are sure everything is working 5.6a For legacy (BIOS) booting, install GRUB to the MBR: - # grub-install /dev/disk/by-id/scsi-SATA_disk1 - Installing for i386-pc platform. - Installation finished. No error reported. + grub-install /dev/disk/by-id/scsi-SATA_disk1 -Do not reboot the computer until you get exactly that result message. Note that you are installing GRUB to the whole disk, not a partition. +Note that you are installing GRUB to the whole disk, not a partition. If you are creating a mirror or raidz topology, repeat the `grub-install` command for each disk in the pool. 5.6b For UEFI booting, install GRUB: - # grub-install --target=x86_64-efi --efi-directory=/boot/efi \ - --bootloader-id=debian --recheck --no-floppy + grub-install --target=x86_64-efi --efi-directory=/boot/efi \ + --bootloader-id=debian --recheck --no-floppy 5.7 Verify that the ZFS module is installed: - # ls /boot/grub/*/zfs.mod + ls /boot/grub/*/zfs.mod 5.8 Fix filesystem mount ordering @@ -474,67 +486,72 @@ Until there is support for mounting `/boot` in the initramfs, we also need to mo We need to activate `zfs-mount-generator`. This makes systemd aware of the separate mountpoints, which is important for things like `/var/log` and `/var/tmp`. In turn, `rsyslog.service` depends on `var-log.mount` by way of `local-fs.target` and services using the `PrivateTmp` feature of systemd automatically use `After=var-tmp.mount`. - For UEFI booting, unmount /boot/efi first: - # umount /boot/efi +For UEFI booting, unmount /boot/efi first: - Everything else applies to both BIOS and UEFI booting: + umount /boot/efi - # zfs set mountpoint=legacy bpool/BOOT/debian - # echo bpool/BOOT/debian /boot zfs \ - nodev,relatime,x-systemd.requires=zfs-import-bpool.service 0 0 >> /etc/fstab +Everything else applies to both BIOS and UEFI booting: - # mkdir /etc/zfs/zfs-list.cache - # touch /etc/zfs/zfs-list.cache/rpool - # ln -s /usr/lib/zfs-linux/zed.d/history_event-zfs-list-cacher.sh /etc/zfs/zed.d - # zed -F & + zfs set mountpoint=legacy bpool/BOOT/debian + echo bpool/BOOT/debian /boot zfs \ + nodev,relatime,x-systemd.requires=zfs-import-bpool.service 0 0 >> /etc/fstab - Verify that zed updated the cache by making sure this is not empty: - # cat /etc/zfs/zfs-list.cache/rpool + mkdir /etc/zfs/zfs-list.cache + touch /etc/zfs/zfs-list.cache/rpool + ln -s /usr/lib/zfs-linux/zed.d/history_event-zfs-list-cacher.sh /etc/zfs/zed.d + zed -F & - If it is empty, force a cache update and check again: - # zfs set canmount=noauto rpool/ROOT/debian +Verify that zed updated the cache by making sure this is not empty: - Stop zed: - # fg + cat /etc/zfs/zfs-list.cache/rpool + +If it is empty, force a cache update and check again: + + zfs set canmount=noauto rpool/ROOT/debian + +Stop zed: + + fg Press Ctrl-C. - Fix the paths to eliminate /mnt: - # sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/rpool +Fix the paths to eliminate /mnt: + + sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/rpool ## Step 6: First Boot 6.1 Snapshot the initial installation: - # zfs snapshot bpool/BOOT/debian@install - # zfs snapshot rpool/ROOT/debian@install + zfs snapshot bpool/BOOT/debian@install + zfs snapshot rpool/ROOT/debian@install In the future, you will likely want to take snapshots before each upgrade, and remove old snapshots (including this one) at some point to save space. 6.2 Exit from the `chroot` environment back to the LiveCD environment: - # exit + exit 6.3 Run these commands in the LiveCD environment to unmount all filesystems: - # mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} - # zpool export -a + mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} + zpool export -a 6.4 Reboot: - # reboot + reboot 6.5 Wait for the newly installed system to boot normally. Login as root. 6.6 Create a user account: - # zfs create rpool/home/YOURUSERNAME - # adduser YOURUSERNAME - # cp -a /etc/skel/. /home/YOURUSERNAME - # chown -R YOURUSERNAME:YOURUSERNAME /home/YOURUSERNAME + zfs create rpool/home/YOURUSERNAME + adduser YOURUSERNAME + cp -a /etc/skel/. /home/YOURUSERNAME + chown -R YOURUSERNAME:YOURUSERNAME /home/YOURUSERNAME 6.7 Add your user account to the default set of groups for an administrator: - # usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video YOURUSERNAME + usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video YOURUSERNAME 6.8 Mirror GRUB @@ -542,21 +559,22 @@ If you installed to multiple disks, install GRUB on the additional disks: 6.8a For legacy (BIOS) booting: - # dpkg-reconfigure grub-pc + dpkg-reconfigure grub-pc Hit enter until you get to the device selection screen. Select (using the space bar) all of the disks (not partitions) in your pool. 6.8b UEFI - # umount /boot/efi + umount /boot/efi - For the second and subsequent disks (increment debian-2 to -3, etc.): - # dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \ - of=/dev/disk/by-id/scsi-SATA_disk2-part2 - # efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \ - -p 3 -L "debian-2" -l '\EFI\debian\grubx64.efi' +For the second and subsequent disks (increment debian-2 to -3, etc.): - # mount /boot/efi + dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \ + of=/dev/disk/by-id/scsi-SATA_disk2-part2 + efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \ + -p 3 -L "debian-2" -l '\EFI\debian\grubx64.efi' + + mount /boot/efi ## Step 7: (Optional) Configure Swap @@ -564,10 +582,10 @@ If you installed to multiple disks, install GRUB on the additional disks: 7.1 Create a volume dataset (zvol) for use as a swap device: - # zfs create -V 4G -b $(getconf PAGESIZE) -o compression=zle \ - -o logbias=throughput -o sync=always \ - -o primarycache=metadata -o secondarycache=none \ - -o com.sun:auto-snapshot=false rpool/swap + zfs create -V 4G -b $(getconf PAGESIZE) -o compression=zle \ + -o logbias=throughput -o sync=always \ + -o primarycache=metadata -o secondarycache=none \ + -o com.sun:auto-snapshot=false rpool/swap You can adjust the size (the `4G` part) to your needs. @@ -577,31 +595,31 @@ The compression algorithm is set to `zle` because it is the cheapest available a **Caution**: Always use long `/dev/zvol` aliases in configuration files. Never use a short `/dev/zdX` device name. - # mkswap -f /dev/zvol/rpool/swap - # echo /dev/zvol/rpool/swap none swap discard 0 0 >> /etc/fstab - # echo RESUME=none > /etc/initramfs-tools/conf.d/resume + mkswap -f /dev/zvol/rpool/swap + echo /dev/zvol/rpool/swap none swap discard 0 0 >> /etc/fstab + echo RESUME=none > /etc/initramfs-tools/conf.d/resume The `RESUME=none` is necessary to disable resuming from hibernation. This does not work, as the zvol is not present (because the pool has not yet been imported) at the time the resume script runs. If it is not disabled, the boot process hangs for 30 seconds waiting for the swap zvol to appear. 7.3 Enable the swap device: - # swapon -av + swapon -av ## Step 8: Full Software Installation 8.1 Upgrade the minimal system: - # apt dist-upgrade --yes + apt dist-upgrade --yes 8.2 Install a regular set of software: - # tasksel + tasksel 8.3 Optional: Disable log compression: As `/var/log` is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if you are making snapshots of `/var/log`, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in `/etc/logrotate.d` by hand to comment out `compress`, or use this loop (copy-and-paste highly recommended): - # for file in /etc/logrotate.d/* ; do + for file in /etc/logrotate.d/* ; do if grep -Eq "(^|[^#y])compress" "$file" ; then sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" fi @@ -609,7 +627,7 @@ As `/var/log` is already compressed by ZFS, logrotate’s compression is going t 8.4 Reboot: - # reboot + reboot ### Step 9: Final Cleanup @@ -617,29 +635,29 @@ As `/var/log` is already compressed by ZFS, logrotate’s compression is going t 9.2 Optional: Delete the snapshots of the initial installation: - $ sudo zfs destroy bpool/BOOT/debian@install - $ sudo zfs destroy rpool/ROOT/debian@install + sudo zfs destroy bpool/BOOT/debian@install + sudo zfs destroy rpool/ROOT/debian@install 9.3 Optional: Disable the root password - $ sudo usermod -p '*' root + sudo usermod -p '*' root 9.4 Optional: Re-enable the graphical boot process: If you prefer the graphical boot process, you can re-enable it now. If you are using LUKS, it makes the prompt look nicer. - $ sudo vi /etc/default/grub + sudo vi /etc/default/grub Add quiet to GRUB_CMDLINE_LINUX_DEFAULT Comment out GRUB_TERMINAL=console Save and quit. - $ sudo update-grub + sudo update-grub **Note:** Ignore errors from `osprober`, if present. 9.5 Optional: For LUKS installs only, backup the LUKS header: - $ sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ + sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ --header-backup-file luks1-header.dat Store that backup somewhere safe (e.g. cloud storage). It is protected by your LUKS passphrase, but you may wish to use additional encryption. @@ -654,36 +672,36 @@ Go through [Step 1: Prepare The Install Environment](#step-1-prepare-the-install For LUKS, first unlock the disk(s): - # apt install --yes cryptsetup - # cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 + apt install --yes cryptsetup + cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 Repeat for additional disks, if this is a mirror or raidz topology. Mount everything correctly: - # zpool export -a - # zpool import -N -R /mnt rpool - # zpool import -N -R /mnt bpool - # zfs load-key -a - # zfs mount rpool/ROOT/debian - # zfs mount -a + zpool export -a + zpool import -N -R /mnt rpool + zpool import -N -R /mnt bpool + zfs load-key -a + zfs mount rpool/ROOT/debian + zfs mount -a If needed, you can chroot into your installed environment: - # mount --rbind /dev /mnt/dev - # mount --rbind /proc /mnt/proc - # mount --rbind /sys /mnt/sys - # chroot /mnt /bin/bash --login - # mount /boot - # mount -a + mount --rbind /dev /mnt/dev + mount --rbind /proc /mnt/proc + mount --rbind /sys /mnt/sys + chroot /mnt /bin/bash --login + mount /boot + mount -a Do whatever you need to do to fix your system. When done, cleanup: - # exit - # mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} - # zpool export -a - # reboot + exit + mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} + zpool export -a + reboot ### MPT2SAS @@ -709,11 +727,13 @@ Set a unique serial number on each virtual disk using libvirt or qemu (e.g. `-dr To be able to use UEFI in guests (instead of only BIOS booting), run this on the host: - $ sudo apt install ovmf - $ sudo vi /etc/libvirt/qemu.conf + sudo apt install ovmf + + sudo vi /etc/libvirt/qemu.conf Uncomment these lines: nvram = [ "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd" ] - $ sudo service libvirt-bin restart + + sudo service libvirt-bin restart diff --git a/Ubuntu-18.04-Root-on-ZFS.md b/Ubuntu-18.04-Root-on-ZFS.md index 0c00d15..e0d6ec7 100644 --- a/Ubuntu-18.04-Root-on-ZFS.md +++ b/Ubuntu-18.04-Root-on-ZFS.md @@ -36,58 +36,63 @@ LUKS encrypts almost everything: the OS, swap, home directories, and anything el 1.2 Setup and update the repositories: - $ sudo apt-add-repository universe - $ sudo apt update + sudo apt-add-repository universe + sudo apt update 1.3 Optional: Install and start the OpenSSH server in the Live CD environment: If you have a second system, using SSH to access the target system can be convenient. - $ passwd + passwd There is no current password; hit enter at that prompt. - $ sudo apt install --yes openssh-server + sudo apt install --yes openssh-server **Hint:** You can find your IP address with `ip addr show scope global | grep inet`. Then, from your main machine, connect with `ssh ubuntu@IP`. 1.4 Become root: - $ sudo -i + sudo -i 1.5 Install ZFS in the Live CD environment: - # apt install --yes debootstrap gdisk zfs-initramfs + apt install --yes debootstrap gdisk zfs-initramfs ## Step 2: Disk Formatting 2.1 If you are re-using a disk, clear it as necessary: - If the disk was previously used in an MD array, zero the superblock: - # apt install --yes mdadm - # mdadm --zero-superblock --force /dev/disk/by-id/scsi-SATA_disk1 +If the disk was previously used in an MD array, zero the superblock: - Clear the partition table: - # sgdisk --zap-all /dev/disk/by-id/scsi-SATA_disk1 + apt install --yes mdadm + mdadm --zero-superblock --force /dev/disk/by-id/scsi-SATA_disk1 + +Clear the partition table: + + sgdisk --zap-all /dev/disk/by-id/scsi-SATA_disk1 2.2 Partition your disk(s): - Run this if you need legacy (BIOS) booting: - # sgdisk -a1 -n1:24K:+1000K -t1:EF02 /dev/disk/by-id/scsi-SATA_disk1 +Run this if you need legacy (BIOS) booting: - Run this for UEFI booting (for use now or in the future): - # sgdisk -n2:1M:+512M -t2:EF00 /dev/disk/by-id/scsi-SATA_disk1 + sgdisk -a1 -n1:24K:+1000K -t1:EF02 /dev/disk/by-id/scsi-SATA_disk1 - Run this for the boot pool: - # sgdisk -n3:0:+1G -t3:BF01 /dev/disk/by-id/scsi-SATA_disk1 +Run this for UEFI booting (for use now or in the future): + + sgdisk -n2:1M:+512M -t2:EF00 /dev/disk/by-id/scsi-SATA_disk1 + +Run this for the boot pool: + + sgdisk -n3:0:+1G -t3:BF01 /dev/disk/by-id/scsi-SATA_disk1 Choose one of the following options: 2.2a Unencrypted: - # sgdisk -n4:0:0 -t4:BF01 /dev/disk/by-id/scsi-SATA_disk1 + sgdisk -n4:0:0 -t4:BF01 /dev/disk/by-id/scsi-SATA_disk1 2.2b LUKS: - # sgdisk -n4:0:0 -t4:8300 /dev/disk/by-id/scsi-SATA_disk1 + sgdisk -n4:0:0 -t4:8300 /dev/disk/by-id/scsi-SATA_disk1 Always use the long `/dev/disk/by-id/*` aliases with ZFS. Using the `/dev/sd*` device nodes directly can cause sporadic import failures, especially on systems that have more than one storage pool. @@ -98,23 +103,23 @@ Always use the long `/dev/disk/by-id/*` aliases with ZFS. Using the `/dev/sd*` 2.3 Create the boot pool: - # zpool create -o ashift=12 -d \ - -o feature@async_destroy=enabled \ - -o feature@bookmarks=enabled \ - -o feature@embedded_data=enabled \ - -o feature@empty_bpobj=enabled \ - -o feature@enabled_txg=enabled \ - -o feature@extensible_dataset=enabled \ - -o feature@filesystem_limits=enabled \ - -o feature@hole_birth=enabled \ - -o feature@large_blocks=enabled \ - -o feature@lz4_compress=enabled \ - -o feature@spacemap_histogram=enabled \ - -o feature@userobj_accounting=enabled \ - -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off \ - -O normalization=formD -O relatime=on -O xattr=sa \ - -O mountpoint=/ -R /mnt \ - bpool /dev/disk/by-id/scsi-SATA_disk1-part3 + zpool create -o ashift=12 -d \ + -o feature@async_destroy=enabled \ + -o feature@bookmarks=enabled \ + -o feature@embedded_data=enabled \ + -o feature@empty_bpobj=enabled \ + -o feature@enabled_txg=enabled \ + -o feature@extensible_dataset=enabled \ + -o feature@filesystem_limits=enabled \ + -o feature@hole_birth=enabled \ + -o feature@large_blocks=enabled \ + -o feature@lz4_compress=enabled \ + -o feature@spacemap_histogram=enabled \ + -o feature@userobj_accounting=enabled \ + -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off \ + -O normalization=formD -O relatime=on -O xattr=sa \ + -O mountpoint=/ -R /mnt \ + bpool /dev/disk/by-id/scsi-SATA_disk1-part3 You should not need to customize any of the options for the boot pool. @@ -130,22 +135,22 @@ Choose one of the following options: 2.4a Unencrypted: - # zpool create -o ashift=12 \ - -O acltype=posixacl -O canmount=off -O compression=lz4 \ - -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ - -O mountpoint=/ -R /mnt \ - rpool /dev/disk/by-id/scsi-SATA_disk1-part4 + zpool create -o ashift=12 \ + -O acltype=posixacl -O canmount=off -O compression=lz4 \ + -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ + -O mountpoint=/ -R /mnt \ + rpool /dev/disk/by-id/scsi-SATA_disk1-part4 2.4b LUKS: - # cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 \ - /dev/disk/by-id/scsi-SATA_disk1-part4 - # cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 - # zpool create -o ashift=12 \ - -O acltype=posixacl -O canmount=off -O compression=lz4 \ - -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ - -O mountpoint=/ -R /mnt \ - rpool /dev/mapper/luks1 + cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha256 \ + /dev/disk/by-id/scsi-SATA_disk1-part4 + cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 + zpool create -o ashift=12 \ + -O acltype=posixacl -O canmount=off -O compression=lz4 \ + -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \ + -O mountpoint=/ -R /mnt \ + rpool /dev/mapper/luks1 * The use of `ashift=12` is recommended here because many drives today have 4KiB (or larger) physical sectors, even though they present 512B logical sectors. Also, a future replacement drive may have 4KiB physical sectors (in which case `ashift=12` is desirable) or 4KiB logical sectors (in which case `ashift=12` is required). * Setting `-O acltype=posixacl` enables POSIX ACLs globally. If you do not want this, remove that option, but later add `-o acltype=posixacl` (note: lowercase "o") to the `zfs create` for `/var/log`, as [journald requires ACLs](https://askubuntu.com/questions/970886/journalctl-says-failed-to-search-journal-acl-operation-not-supported) @@ -164,72 +169,84 @@ Choose one of the following options: 3.1 Create filesystem datasets to act as containers: - # zfs create -o canmount=off -o mountpoint=none rpool/ROOT - # zfs create -o canmount=off -o mountpoint=none bpool/BOOT + zfs create -o canmount=off -o mountpoint=none rpool/ROOT + zfs create -o canmount=off -o mountpoint=none bpool/BOOT On Solaris systems, the root filesystem is cloned and the suffix is incremented for major system changes through `pkg image-update` or `beadm`. Similar functionality for APT is possible but currently unimplemented. Even without such a tool, it can still be used for manually created clones. 3.2 Create filesystem datasets for the root and boot filesystems: - # zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/ubuntu - # zfs mount rpool/ROOT/ubuntu + zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/ubuntu + zfs mount rpool/ROOT/ubuntu - # zfs create -o canmount=noauto -o mountpoint=/boot bpool/BOOT/ubuntu - # zfs mount bpool/BOOT/ubuntu + zfs create -o canmount=noauto -o mountpoint=/boot bpool/BOOT/ubuntu + zfs mount bpool/BOOT/ubuntu With ZFS, it is not normally necessary to use a mount command (either `mount` or `zfs mount`). This situation is an exception because of `canmount=noauto`. 3.3 Create datasets: - # zfs create rpool/home - # zfs create -o mountpoint=/root rpool/home/root - # zfs create -o canmount=off rpool/var - # zfs create -o canmount=off rpool/var/lib - # zfs create rpool/var/log - # zfs create rpool/var/spool + zfs create rpool/home + zfs create -o mountpoint=/root rpool/home/root + zfs create -o canmount=off rpool/var + zfs create -o canmount=off rpool/var/lib + zfs create rpool/var/log + zfs create rpool/var/spool - The datasets below are optional, depending on your preferences and/or - software choices: +The datasets below are optional, depending on your preferences and/or software +choices. - If you wish to exclude these from snapshots: - # zfs create -o com.sun:auto-snapshot=false rpool/var/cache - # zfs create -o com.sun:auto-snapshot=false rpool/var/tmp - # chmod 1777 /mnt/var/tmp +If you wish to exclude these from snapshots: - If you use /opt on this system: - # zfs create rpool/opt + zfs create -o com.sun:auto-snapshot=false rpool/var/cache + zfs create -o com.sun:auto-snapshot=false rpool/var/tmp + chmod 1777 /mnt/var/tmp - If you use /srv on this system: - # zfs create rpool/srv +If you use /opt on this system: - If you use /usr/local on this system: - # zfs create -o canmount=off rpool/usr - # zfs create rpool/usr/local + zfs create rpool/opt - If this system will have games installed: - # zfs create rpool/var/games +If you use /srv on this system: - If this system will store local email in /var/mail: - # zfs create rpool/var/mail + zfs create rpool/srv - If this system will use Snap packages: - # zfs create rpool/var/snap +If you use /usr/local on this system: - If you use /var/www on this system: - # zfs create rpool/var/www + zfs create -o canmount=off rpool/usr + zfs create rpool/usr/local - If this system will use GNOME: - # zfs create rpool/var/lib/AccountsService +If this system will have games installed: - If this system will use Docker (which manages its own datasets & snapshots): - # zfs create -o com.sun:auto-snapshot=false rpool/var/lib/docker + zfs create rpool/var/games - If this system will use NFS (locking): - # zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs +If this system will store local email in /var/mail: - A tmpfs is recommended later, but if you want a separate dataset for /tmp: - # zfs create -o com.sun:auto-snapshot=false rpool/tmp - # chmod 1777 /mnt/tmp + zfs create rpool/var/mail + +If this system will use Snap packages: + + zfs create rpool/var/snap + +If you use /var/www on this system: + + zfs create rpool/var/www + +If this system will use GNOME: + + zfs create rpool/var/lib/AccountsService + +If this system will use Docker (which manages its own datasets & snapshots): + + zfs create -o com.sun:auto-snapshot=false rpool/var/lib/docker + +If this system will use NFS (locking): + + zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs + +A tmpfs is recommended later, but if you want a separate dataset for /tmp: + + zfs create -o com.sun:auto-snapshot=false rpool/tmp + chmod 1777 /mnt/tmp The primary goal of this dataset layout is to separate the OS from user data. This allows the root filesystem to be rolled back without rolling back user data such as logs (in `/var/log`). This will be especially important if/when a `beadm` or similar utility is integrated. The `com.sun.auto-snapshot` setting is used by some ZFS snapshot utilities to exclude transient data. @@ -237,8 +254,8 @@ If you do nothing extra, `/tmp` will be stored as part of the root filesystem. A 3.4 Install the minimal system: - # debootstrap bionic /mnt - # zfs set devices=off rpool + debootstrap bionic /mnt + zfs set devices=off rpool The `debootstrap` command leaves the new system in an unconfigured state. An alternative to using `debootstrap` is to copy the entirety of a working system into the new ZFS root. @@ -246,9 +263,9 @@ The `debootstrap` command leaves the new system in an unconfigured state. An al 4.1 Configure the hostname (change `HOSTNAME` to the desired hostname). - # echo HOSTNAME > /mnt/etc/hostname + echo HOSTNAME > /mnt/etc/hostname - # vi /mnt/etc/hosts + vi /mnt/etc/hosts Add a line: 127.0.1.1 HOSTNAME or if the system has a real name in DNS: @@ -258,11 +275,13 @@ The `debootstrap` command leaves the new system in an unconfigured state. An al 4.2 Configure the network interface: - Find the interface name: - # ip addr show +Find the interface name: - Adjust NAME below to match your interface name: - # vi /mnt/etc/netplan/01-netcfg.yaml + ip addr show + +Adjust NAME below to match your interface name: + + vi /mnt/etc/netplan/01-netcfg.yaml network: version: 2 ethernets: @@ -273,7 +292,7 @@ Customize this file if the system is not a DHCP client. 4.3 Configure the package sources: - # vi /mnt/etc/apt/sources.list + vi /mnt/etc/apt/sources.list deb http://archive.ubuntu.com/ubuntu bionic main universe deb-src http://archive.ubuntu.com/ubuntu bionic main universe @@ -285,41 +304,42 @@ Customize this file if the system is not a DHCP client. 4.4 Bind the virtual filesystems from the LiveCD environment to the new system and `chroot` into it: - # mount --rbind /dev /mnt/dev - # mount --rbind /proc /mnt/proc - # mount --rbind /sys /mnt/sys - # chroot /mnt /bin/bash --login + mount --rbind /dev /mnt/dev + mount --rbind /proc /mnt/proc + mount --rbind /sys /mnt/sys + chroot /mnt /bin/bash --login **Note:** This is using `--rbind`, not `--bind`. 4.5 Configure a basic system environment: - # ln -s /proc/self/mounts /etc/mtab - # apt update + ln -s /proc/self/mounts /etc/mtab + apt update - # dpkg-reconfigure locales + dpkg-reconfigure locales Even if you prefer a non-English system language, always ensure that `en_US.UTF-8` is available. - # dpkg-reconfigure tzdata + dpkg-reconfigure tzdata - If you prefer nano over vi, install it: - # apt install --yes nano +If you prefer nano over vi, install it: + + apt install --yes nano 4.6 Install ZFS in the chroot environment for the new system: - # apt install --yes --no-install-recommends linux-image-generic - # apt install --yes zfs-initramfs + apt install --yes --no-install-recommends linux-image-generic + apt install --yes zfs-initramfs **Hint:** For the HWE kernel, install `linux-image-generic-hwe-18.04` instead of `linux-image-generic`. 4.7 For LUKS installs only, setup crypttab: - # apt install --yes cryptsetup + apt install --yes cryptsetup - # echo luks1 UUID=$(blkid -s UUID -o value \ - /dev/disk/by-id/scsi-SATA_disk1-part4) none \ - luks,discard,initramfs > /etc/crypttab + echo luks1 UUID=$(blkid -s UUID -o value \ + /dev/disk/by-id/scsi-SATA_disk1-part4) none \ + luks,discard,initramfs > /etc/crypttab * The use of `initramfs` is a work-around for [cryptsetup does not support ZFS](https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/1612906). @@ -331,20 +351,20 @@ Choose one of the following options: 4.8a Install GRUB for legacy (BIOS) booting - # apt install --yes grub-pc + apt install --yes grub-pc Install GRUB to the disk(s), not the partition(s). 4.8b Install GRUB for UEFI booting - # apt install dosfstools - # mkdosfs -F 32 -s 1 -n EFI /dev/disk/by-id/scsi-SATA_disk1-part2 - # mkdir /boot/efi - # echo PARTUUID=$(blkid -s PARTUUID -o value \ - /dev/disk/by-id/scsi-SATA_disk1-part2) \ - /boot/efi vfat nofail,x-systemd.device-timeout=1 0 1 >> /etc/fstab - # mount /boot/efi - # apt install --yes grub-efi-amd64-signed shim-signed + apt install dosfstools + mkdosfs -F 32 -s 1 -n EFI /dev/disk/by-id/scsi-SATA_disk1-part2 + mkdir /boot/efi + echo PARTUUID=$(blkid -s PARTUUID -o value \ + /dev/disk/by-id/scsi-SATA_disk1-part2) \ + /boot/efi vfat nofail,x-systemd.device-timeout=1 0 1 >> /etc/fstab + mount /boot/efi + apt install --yes grub-efi-amd64-signed shim-signed * The `-s 1` for `mkdosfs` is only necessary for drives which present 4 KiB logical sectors (“4Kn” drives) to meet the minimum cluster size (given the partition size of 512 MiB) for FAT32. It also works fine on drives which present 512 B sectors. @@ -352,13 +372,13 @@ Install GRUB to the disk(s), not the partition(s). 4.9 Set a root password - # passwd + passwd 4.10 Enable importing bpool This ensures that `bpool` is always imported, regardless of whether `/etc/zfs/zpool.cache` exists, whether it is in the cachefile or not, or whether `zfs-import-scan.service` is enabled. ``` - # vi /etc/systemd/system/zfs-import-bpool.service + vi /etc/systemd/system/zfs-import-bpool.service [Unit] DefaultDependencies=no Before=zfs-import-scan.service @@ -372,43 +392,41 @@ This ensures that `bpool` is always imported, regardless of whether `/etc/zfs/zp [Install] WantedBy=zfs-import.target - # systemctl enable zfs-import-bpool.service + systemctl enable zfs-import-bpool.service ``` 4.11 Optional (but recommended): Mount a tmpfs to /tmp If you chose to create a `/tmp` dataset above, skip this step, as they are mutually exclusive choices. Otherwise, you can put `/tmp` on a tmpfs (RAM filesystem) by enabling the `tmp.mount` unit. - # cp /usr/share/systemd/tmp.mount /etc/systemd/system/ - # systemctl enable tmp.mount + cp /usr/share/systemd/tmp.mount /etc/systemd/system/ + systemctl enable tmp.mount 4.12 Setup system groups: - # addgroup --system lpadmin - # addgroup --system sambashare + addgroup --system lpadmin + addgroup --system sambashare ## Step 5: GRUB Installation 5.1 Verify that the ZFS boot filesystem is recognized: - # grub-probe /boot - zfs + grub-probe /boot 5.2 Refresh the initrd files: - # update-initramfs -u -k all - update-initramfs: Generating /boot/initrd.img-4.15.0-46-generic + update-initramfs -u -k all **Note:** When using LUKS, this will print "WARNING could not determine root device from /etc/fstab". This is because [cryptsetup does not support ZFS](https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/1612906). 5.3 Workaround GRUB's missing zpool-features support: - # vi /etc/default/grub + vi /etc/default/grub Set: GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/ubuntu" 5.4 Optional (but highly recommended): Make debugging GRUB easier: - # vi /etc/default/grub + vi /etc/default/grub Comment out: GRUB_TIMEOUT_STYLE=hidden Set: GRUB_TIMEOUT=5 Below GRUB_TIMEOUT, add: GRUB_RECORDFAIL_TIMEOUT=5 @@ -420,11 +438,7 @@ Later, once the system has rebooted twice and you are sure everything is working 5.5 Update the boot configuration: - # update-grub - Generating grub configuration file ... - Found linux image: /boot/vmlinuz-4.15.0-46-generic - Found initrd image: /boot/initrd.img-4.15.0-46-generic - done + update-grub **Note:** Ignore errors from `osprober`, if present. @@ -432,22 +446,20 @@ Later, once the system has rebooted twice and you are sure everything is working 5.6a For legacy (BIOS) booting, install GRUB to the MBR: - # grub-install /dev/disk/by-id/scsi-SATA_disk1 - Installing for i386-pc platform. - Installation finished. No error reported. + grub-install /dev/disk/by-id/scsi-SATA_disk1 -Do not reboot the computer until you get exactly that result message. Note that you are installing GRUB to the whole disk, not a partition. +Note that you are installing GRUB to the whole disk, not a partition. If you are creating a mirror or raidz topology, repeat the `grub-install` command for each disk in the pool. 5.6b For UEFI booting, install GRUB: - # grub-install --target=x86_64-efi --efi-directory=/boot/efi \ - --bootloader-id=ubuntu --recheck --no-floppy + grub-install --target=x86_64-efi --efi-directory=/boot/efi \ + --bootloader-id=ubuntu --recheck --no-floppy 5.7 Verify that the ZFS module is installed: - # ls /boot/grub/*/zfs.mod + ls /boot/grub/*/zfs.mod 5.8 Fix filesystem mount ordering @@ -458,63 +470,66 @@ Until there is support for mounting `/boot` in the initramfs, we also need to mo `rpool` is guaranteed to be imported by the initramfs, so there is no point in adding `x-systemd.requires=zfs-import.target` to those filesystems. - For UEFI booting, unmount /boot/efi first: - # umount /boot/efi +For UEFI booting, unmount /boot/efi first: - Everything else applies to both BIOS and UEFI booting: + umount /boot/efi - # zfs set mountpoint=legacy bpool/BOOT/ubuntu - # echo bpool/BOOT/ubuntu /boot zfs \ - nodev,relatime,x-systemd.requires=zfs-import-bpool.service 0 0 >> /etc/fstab +Everything else applies to both BIOS and UEFI booting: - # zfs set mountpoint=legacy rpool/var/log - # echo rpool/var/log /var/log zfs nodev,relatime 0 0 >> /etc/fstab + zfs set mountpoint=legacy bpool/BOOT/ubuntu + echo bpool/BOOT/ubuntu /boot zfs \ + nodev,relatime,x-systemd.requires=zfs-import-bpool.service 0 0 >> /etc/fstab - # zfs set mountpoint=legacy rpool/var/spool - # echo rpool/var/spool /var/spool zfs nodev,relatime 0 0 >> /etc/fstab + zfs set mountpoint=legacy rpool/var/log + echo rpool/var/log /var/log zfs nodev,relatime 0 0 >> /etc/fstab - If you created a /var/tmp dataset: - # zfs set mountpoint=legacy rpool/var/tmp - # echo rpool/var/tmp /var/tmp zfs nodev,relatime 0 0 >> /etc/fstab + zfs set mountpoint=legacy rpool/var/spool + echo rpool/var/spool /var/spool zfs nodev,relatime 0 0 >> /etc/fstab - If you created a /tmp dataset: - # zfs set mountpoint=legacy rpool/tmp - # echo rpool/tmp /tmp zfs nodev,relatime 0 0 >> /etc/fstab +If you created a /var/tmp dataset: + + zfs set mountpoint=legacy rpool/var/tmp + echo rpool/var/tmp /var/tmp zfs nodev,relatime 0 0 >> /etc/fstab + +If you created a /tmp dataset: + + zfs set mountpoint=legacy rpool/tmp + echo rpool/tmp /tmp zfs nodev,relatime 0 0 >> /etc/fstab ## Step 6: First Boot 6.1 Snapshot the initial installation: - # zfs snapshot bpool/BOOT/ubuntu@install - # zfs snapshot rpool/ROOT/ubuntu@install + zfs snapshot bpool/BOOT/ubuntu@install + zfs snapshot rpool/ROOT/ubuntu@install In the future, you will likely want to take snapshots before each upgrade, and remove old snapshots (including this one) at some point to save space. 6.2 Exit from the `chroot` environment back to the LiveCD environment: - # exit + exit 6.3 Run these commands in the LiveCD environment to unmount all filesystems: - # mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} - # zpool export -a + mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} + zpool export -a 6.4 Reboot: - # reboot + reboot 6.5 Wait for the newly installed system to boot normally. Login as root. 6.6 Create a user account: - # zfs create rpool/home/YOURUSERNAME - # adduser YOURUSERNAME - # cp -a /etc/skel/. /home/YOURUSERNAME - # chown -R YOURUSERNAME:YOURUSERNAME /home/YOURUSERNAME + zfs create rpool/home/YOURUSERNAME + adduser YOURUSERNAME + cp -a /etc/skel/. /home/YOURUSERNAME + chown -R YOURUSERNAME:YOURUSERNAME /home/YOURUSERNAME 6.7 Add your user account to the default set of groups for an administrator: - # usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sambashare,sudo YOURUSERNAME + usermod -a -G adm,cdrom,dip,lpadmin,plugdev,sambashare,sudo YOURUSERNAME 6.8 Mirror GRUB @@ -522,21 +537,22 @@ If you installed to multiple disks, install GRUB on the additional disks: 6.8a For legacy (BIOS) booting: - # dpkg-reconfigure grub-pc + dpkg-reconfigure grub-pc Hit enter until you get to the device selection screen. Select (using the space bar) all of the disks (not partitions) in your pool. 6.8b UEFI - # umount /boot/efi + umount /boot/efi - For the second and subsequent disks (increment ubuntu-2 to -3, etc.): - # dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \ - of=/dev/disk/by-id/scsi-SATA_disk2-part2 - # efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \ - -p 3 -L "ubuntu-2" -l '\EFI\ubuntu\grubx64.efi' +For the second and subsequent disks (increment ubuntu-2 to -3, etc.): - # mount /boot/efi + dd if=/dev/disk/by-id/scsi-SATA_disk1-part2 \ + of=/dev/disk/by-id/scsi-SATA_disk2-part2 + efibootmgr -c -g -d /dev/disk/by-id/scsi-SATA_disk2 \ + -p 3 -L "ubuntu-2" -l '\EFI\ubuntu\grubx64.efi' + + mount /boot/efi ## Step 7: (Optional) Configure Swap @@ -544,10 +560,10 @@ If you installed to multiple disks, install GRUB on the additional disks: 7.1 Create a volume dataset (zvol) for use as a swap device: - # zfs create -V 4G -b $(getconf PAGESIZE) -o compression=zle \ - -o logbias=throughput -o sync=always \ - -o primarycache=metadata -o secondarycache=none \ - -o com.sun:auto-snapshot=false rpool/swap + zfs create -V 4G -b $(getconf PAGESIZE) -o compression=zle \ + -o logbias=throughput -o sync=always \ + -o primarycache=metadata -o secondarycache=none \ + -o com.sun:auto-snapshot=false rpool/swap You can adjust the size (the `4G` part) to your needs. @@ -557,21 +573,21 @@ The compression algorithm is set to `zle` because it is the cheapest available a **Caution**: Always use long `/dev/zvol` aliases in configuration files. Never use a short `/dev/zdX` device name. - # mkswap -f /dev/zvol/rpool/swap - # echo /dev/zvol/rpool/swap none swap discard 0 0 >> /etc/fstab - # echo RESUME=none > /etc/initramfs-tools/conf.d/resume + mkswap -f /dev/zvol/rpool/swap + echo /dev/zvol/rpool/swap none swap discard 0 0 >> /etc/fstab + echo RESUME=none > /etc/initramfs-tools/conf.d/resume The `RESUME=none` is necessary to disable resuming from hibernation. This does not work, as the zvol is not present (because the pool has not yet been imported) at the time the resume script runs. If it is not disabled, the boot process hangs for 30 seconds waiting for the swap zvol to appear. 7.3 Enable the swap device: - # swapon -av + swapon -av ## Step 8: Full Software Installation 8.1 Upgrade the minimal system: - # apt dist-upgrade --yes + apt dist-upgrade --yes 8.2 Install a regular set of software: @@ -579,15 +595,15 @@ Choose one of the following options: 8.2a Install a command-line environment only: - # apt install --yes ubuntu-standard + apt install --yes ubuntu-standard 8.2b Install a full GUI environment: - # apt install --yes ubuntu-desktop + apt install --yes ubuntu-desktop **Hint**: If you are installing a full GUI environment, you will likely want to manage your network with NetworkManager: - # vi /etc/netplan/01-netcfg.yaml + vi /etc/netplan/01-netcfg.yaml network: version: 2 renderer: NetworkManager @@ -596,7 +612,7 @@ Choose one of the following options: As `/var/log` is already compressed by ZFS, logrotate’s compression is going to burn CPU and disk I/O for (in most cases) very little gain. Also, if you are making snapshots of `/var/log`, logrotate’s compression will actually waste space, as the uncompressed data will live on in the snapshot. You can edit the files in `/etc/logrotate.d` by hand to comment out `compress`, or use this loop (copy-and-paste highly recommended): - # for file in /etc/logrotate.d/* ; do + for file in /etc/logrotate.d/* ; do if grep -Eq "(^|[^#y])compress" "$file" ; then sed -i -r "s/(^|[^#y])(compress)/\1#\2/" "$file" fi @@ -604,7 +620,7 @@ As `/var/log` is already compressed by ZFS, logrotate’s compression is going t 8.4 Reboot: - # reboot + reboot ### Step 9: Final Cleanup @@ -612,30 +628,30 @@ As `/var/log` is already compressed by ZFS, logrotate’s compression is going t 9.2 Optional: Delete the snapshots of the initial installation: - $ sudo zfs destroy bpool/BOOT/ubuntu@install - $ sudo zfs destroy rpool/ROOT/ubuntu@install + sudo zfs destroy bpool/BOOT/ubuntu@install + sudo zfs destroy rpool/ROOT/ubuntu@install 9.3 Optional: Disable the root password - $ sudo usermod -p '*' root + sudo usermod -p '*' root 9.4 Optional: Re-enable the graphical boot process: If you prefer the graphical boot process, you can re-enable it now. If you are using LUKS, it makes the prompt look nicer. - $ sudo vi /etc/default/grub + sudo vi /etc/default/grub Uncomment: GRUB_TIMEOUT_STYLE=hidden Add quiet and splash to: GRUB_CMDLINE_LINUX_DEFAULT Comment out: GRUB_TERMINAL=console Save and quit. - $ sudo update-grub + sudo update-grub **Note:** Ignore errors from `osprober`, if present. 9.5 Optional: For LUKS installs only, backup the LUKS header: - $ sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ + sudo cryptsetup luksHeaderBackup /dev/disk/by-id/scsi-SATA_disk1-part4 \ --header-backup-file luks1-header.dat Store that backup somewhere safe (e.g. cloud storage). It is protected by your LUKS passphrase, but you may wish to use additional encryption. @@ -650,34 +666,34 @@ Go through [Step 1: Prepare The Install Environment](#step-1-prepare-the-install For LUKS, first unlock the disk(s): - # cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 + cryptsetup luksOpen /dev/disk/by-id/scsi-SATA_disk1-part4 luks1 Repeat for additional disks, if this is a mirror or raidz topology. Mount everything correctly: - # zpool export -a - # zpool import -N -R /mnt rpool - # zpool import -N -R /mnt bpool - # zfs mount rpool/ROOT/ubuntu - # zfs mount -a + zpool export -a + zpool import -N -R /mnt rpool + zpool import -N -R /mnt bpool + zfs mount rpool/ROOT/ubuntu + zfs mount -a If needed, you can chroot into your installed environment: - # mount --rbind /dev /mnt/dev - # mount --rbind /proc /mnt/proc - # mount --rbind /sys /mnt/sys - # chroot /mnt /bin/bash --login - # mount /boot - # mount -a + mount --rbind /dev /mnt/dev + mount --rbind /proc /mnt/proc + mount --rbind /sys /mnt/sys + chroot /mnt /bin/bash --login + mount /boot + mount -a Do whatever you need to do to fix your system. When done, cleanup: - # exit - # mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} - # zpool export -a - # reboot + exit + mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {} + zpool export -a + reboot ### MPT2SAS @@ -703,11 +719,13 @@ Set a unique serial number on each virtual disk using libvirt or qemu (e.g. `-dr To be able to use UEFI in guests (instead of only BIOS booting), run this on the host: - $ sudo apt install ovmf - $ sudo vi /etc/libvirt/qemu.conf + sudo apt install ovmf + + sudo vi /etc/libvirt/qemu.conf Uncomment these lines: nvram = [ "/usr/share/OVMF/OVMF_CODE.fd:/usr/share/OVMF/OVMF_VARS.fd", "/usr/share/AAVMF/AAVMF_CODE.fd:/usr/share/AAVMF/AAVMF_VARS.fd" ] - $ sudo service libvirt-bin restart + + sudo service libvirt-bin restart