ZFS root fs on FreeBSD 9.0

Gone are the old MFS root fs image and the sysinstall utility. To create a useable ZFS root fs from the 9.0-RELEASE DVD we must specify an alternate location for the temporary zpool.cache file and an alternate mountpoint for the ZFS root fs by using a command like:

zpool create -o cachefile=/tmp/zpool.cache -m /tmp/zroot zroot /dev/gpt/disk0

Also, the FreeBSD distribution is now stored as a handful of xz compressed tar files in /usr/freebsd-dist on the DVD. Everything else remain largely the same. The instructions below also apply to 9.1-RELEASE.

See http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot and http://www.aisecure.net/2011/05/01/root-on-zfs-freebsd-current/ for further details.

Keep on reading if you want my recipe for creating a ZFS root fs.

  1. Boot from the FreeBSD 9.0-RELEASE DVD. You may choose the AMD64 or the i386 ISO file. You may choose the FreeBSD 9.1-RELEASE DVD, either AMD64 or i386.
  2. Interrupt the FreeBSD boot loader by pressing the escape key.
    Boot menu for FreeBSD 9 and onwards
    Boot menu for FreeBSD 9 onwards

  3. Issue the following commands, adjust to taste if necessary:
    load opensolaris
    load zfs
    boot
    
  4. Choose Shell by hitting the right arrow and the enter keys.
    FreeBSD Installer with Shell option selected
    FreeBSD Installer with Shell option selected

  5. Assuming you want to use GPT partition scheme, issue these commands, adjust to taste if necessary:
    gpart create -s gpt ada0
    gpart add -s 64K -t freebsd-boot ada0
    gpart add -s 4G -t freebsd-swap -l swap0 ada0
    gpart add -t freebsd-zfs -l disk0 ada0
    gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
    
  6. Create the root pool and populate the pool with the required and other file systems as you see fit:
    zpool create -o cachefile=/tmp/zpool.cache -m /tmp/zroot zroot /dev/gpt/disk0
    zpool set bootfs=zroot zroot
    
    zfs set checksum=fletcher4 zroot
    
    zfs create -o setuid=off zroot/home
    
    zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp
    chmod 1777 /tmp/zroot/tmp
    
    zfs create zroot/usr
    zfs create zroot/usr/compat
    ln -s usr/compat /tmp/zroot/compat
    mkdir -p /tmp/zroot/compat/linux/proc
    
    zfs create zroot/usr/local
    
    zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/local/certs
    zfs create -o compression=lzjb -o exec=on  -o setuid=off zroot/usr/local/etc
    
    zfs create zroot/usr/local/www
    
    zfs create zroot/usr/obj
    
    zfs create -o exec=off -o setuid=off zroot/usr/packages
    
    zfs create -o compression=lzjb -o exec=on  -o setuid=off zroot/usr/ports
    zfs create -o compression=off  -o exec=off -o setuid=off zroot/usr/ports/distfiles
    zfs create -o compression=off  -o exec=off -o setuid=off zroot/usr/ports/packages
    zfs create -o compression=off  -o exec=on  -o setuid=off zroot/usr/ports/workdirs
    
    zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src
    
    zfs create zroot/var
    
    zfs create -o compression=lzjb   -o exec=off -o setuid=off zroot/var/crash
    zfs create                       -o exec=off -o setuid=off zroot/var/db
    zfs create -o compression=lzjb   -o exec=on  -o setuid=off zroot/var/db/pkg
    zfs create -o compression=lzjb   -o exec=on  -o setuid=off zroot/var/db/ports
    zfs create -o compression=lzjb   -o exec=off -o setuid=off zroot/var/db/sup
    zfs create                       -o exec=off -o setuid=off zroot/var/empty
    zfs create -o compression=lzjb   -o exec=off -o setuid=off zroot/var/log
    zfs create -o compression=gzip-9 -o exec=off -o setuid=off zroot/var/mail
    zfs create                       -o exec=off -o setuid=off zroot/var/run
    zfs create -o compression=lzjb   -o exec=on  -o setuid=off zroot/var/tmp
    chmod 1777 /tmp/zroot/var/tmp
    
  7. It might be a Good Thing™ to reserve some of the storage capacity for the root fs by setting the refreservation property. This property won’t propagate to the child filesystems. 10G might be sufficient for your needs or else take your pick:
    zfs set refreservation=10G zroot
    
  8. Change the working directory to /tmp/zroot and extract the necessary tar files:
    cd /tmp/zroot
    tar xvvf /usr/freebsd-dist/kernel.txz
    tar xvvf /usr/freebsd-dist/base.txz
    tar xvvf /usr/freebsd-dist/lib32.txz # Only on AMD64.
    
  9. Write protect /var/empty:
    zfs set readonly=on zroot/var/empty
    
  10. Copy the zpool.cache file to the ZFS root fs:
    cp -p /tmp/zpool.cache /tmp/zroot/boot/zfs/zpool.cache
    
  11. chroot into the new root fs:
    chroot /tmp/zroot
    set -E
    
  12. Create separate mountpoints for CDs (ISO 9660) and DVDs (UDF), as FreeBSD lacks autofs:
    mkdir /media/cdrom
    mkdir /media/dvdrom
    
  13. Create a minimal /etc/fstab file, adjust the following to taste:
    # Device        Mountpoint              FStype          Options         Dump    Pass#
    /dev/gpt/swap0  none                    swap            sw              0       0
    /dev/cd0        /media/cdrom            cd9660          ro,noauto       0       0
    /dev/cd0        /media/dvdrom           udf             ro,noauto       0       0
    proc            /proc                   procfs          rw,late         0       0
    linproc         /compat/linux/proc      linprocfs       rw,late         0       0
    
  14. Create a minimal /etc/rc.conf file, adjust the following to taste:
    zfs_enable="YES"
    background_fsck="NO"
    hostname="zfstest.example.org"
    ifconfig_em0="DHCP"
    ipv6_activate_all_interfaces="YES"
    keymap="norwegian.iso"
    keyrate="fast"
    keybell="off"
    font8x16="iso-8x16"
    font8x14="iso-8x14"
    font8x8="iso-8x8"
    dumpdev="/dev/gpt/swap0"
    
  15. If you decide not to use DHCP, then create a /etc/resolv.conf file and adjust the contents of this file to fit your environment:
    search example.org
    nameserver 2001:db8:1234:1::2
    nameserver 2001:db8:1234:1::3
    nameserver 192.0.2.2
    
  16. Create a minimal /boot/loader.conf, adjust the following to taste:
    zfs_load="YES"
    vfs.root.mountfrom="zfs:zroot"
    

    Update 2013-07-21: Newer versions of {,gpt}zfsboot are able to compute the correct value for vfs.root.mountfrom based on the bootfs zpool attribute. Simply avoid setting a value for vfs.root.mountfrom in /boot/loader.conf.

  17. It might make sense to store the workdirs for the ports hierarchy in a filesystem without any compression being applied, so create a minimal /etc/make.conf, adjust the following to taste:
    WRKDIRPREFIX=/usr/ports/workdirs
    
  18. Set root’s password (and optionally toor’s password):
    passwd root
    passwd toor
    
  19. Set the time zone:
    tzsetup
    
  20. Create the /etc/mail/aliases.db file:
    cd /etc/mail
    make aliases
    
  21. Leave the chroot environment:
    exit
    
  22. Unmount the ZFS file systems:
    cd /
    zfs unmount -a
    
  23. Set the final mountpoints:
    zfs set mountpoint=legacy zroot
    zfs set mountpoint=/home  zroot/home
    zfs set mountpoint=/tmp   zroot/tmp
    zfs set mountpoint=/usr   zroot/usr
    zfs set mountpoint=/var   zroot/var
    
  24. Unmount the ZFS file systems once more:
    zfs unmount -a
    
  25. Reboot the system, unload the DVD, and cross your fingers:
    1. reboot
    2. Unload DVD.
    3. Cross fingers.