ZFS root fs on FreeBSD 9.0

Gone are the old MFS root fs image and the sysinstall utility. To create a useable ZFS root fs from the 9.0-RELEASE DVD we must specify an alternate location for the temporary zpool.cache file and an alternate mountpoint for the ZFS root fs by using a command like:

zpool create -o cachefile=/tmp/zpool.cache -m /tmp/zroot zroot /dev/gpt/disk0

Also, the FreeBSD distribution is now stored as a handful of xz compressed tar files in /usr/freebsd-dist on the DVD. Everything else remain largely the same. The instructions below also apply to 9.1-RELEASE.

See http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot and http://www.aisecure.net/2011/05/01/root-on-zfs-freebsd-current/ for further details.

Keep on reading if you want my recipe for creating a ZFS root fs.

Boot from the FreeBSD 9.0-RELEASE DVD. You may choose the AMD64 or the i386 ISO file. You may choose the FreeBSD 9.1-RELEASE DVD, either AMD64 or i386.
Interrupt the FreeBSD boot loader by pressing the escape key.

Boot menu for FreeBSD 9 onwards
Issue the following commands, adjust to taste if necessary:
```
load opensolaris
load zfs
boot
```
Choose Shell by hitting the right arrow and the enter keys.

FreeBSD Installer with Shell option selected

Assuming you want to use GPT partition scheme, issue these commands, adjust to taste if necessary:

gpart create -s gpt ada0
gpart add -s 64K -t freebsd-boot ada0
gpart add -s 4G -t freebsd-swap -l swap0 ada0
gpart add -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

Create the root pool and populate the pool with the required and other file systems as you see fit:

zpool create -o cachefile=/tmp/zpool.cache -m /tmp/zroot zroot /dev/gpt/disk0
zpool set bootfs=zroot zroot

zfs set checksum=fletcher4 zroot

zfs create -o setuid=off zroot/home

zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp
chmod 1777 /tmp/zroot/tmp

zfs create zroot/usr
zfs create zroot/usr/compat
ln -s usr/compat /tmp/zroot/compat
mkdir -p /tmp/zroot/compat/linux/proc

zfs create zroot/usr/local

zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/local/certs
zfs create -o compression=lzjb -o exec=on  -o setuid=off zroot/usr/local/etc

zfs create zroot/usr/local/www

zfs create zroot/usr/obj

zfs create -o exec=off -o setuid=off zroot/usr/packages

zfs create -o compression=lzjb -o exec=on  -o setuid=off zroot/usr/ports
zfs create -o compression=off  -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off  -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create -o compression=off  -o exec=on  -o setuid=off zroot/usr/ports/workdirs

zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src

zfs create zroot/var

zfs create -o compression=lzjb   -o exec=off -o setuid=off zroot/var/crash
zfs create                       -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb   -o exec=on  -o setuid=off zroot/var/db/pkg
zfs create -o compression=lzjb   -o exec=on  -o setuid=off zroot/var/db/ports
zfs create -o compression=lzjb   -o exec=off -o setuid=off zroot/var/db/sup
zfs create                       -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb   -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip-9 -o exec=off -o setuid=off zroot/var/mail
zfs create                       -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb   -o exec=on  -o setuid=off zroot/var/tmp
chmod 1777 /tmp/zroot/var/tmp

It might be a Good Thing™ to reserve some of the storage capacity for the root fs by setting the refreservation property. This property won’t propagate to the child filesystems. 10G might be sufficient for your needs or else take your pick:
```
zfs set refreservation=10G zroot
```

Change the working directory to /tmp/zroot and extract the necessary tar files:

cd /tmp/zroot
tar xvvf /usr/freebsd-dist/kernel.txz
tar xvvf /usr/freebsd-dist/base.txz
tar xvvf /usr/freebsd-dist/lib32.txz # Only on AMD64.

Write protect /var/empty:
```
zfs set readonly=on zroot/var/empty
```

Copy the zpool.cache file to the ZFS root fs:

cp -p /tmp/zpool.cache /tmp/zroot/boot/zfs/zpool.cache

chroot into the new root fs:
```
chroot /tmp/zroot
set -E
```
Create separate mountpoints for CDs (ISO 9660) and DVDs (UDF), as FreeBSD lacks autofs:
```
mkdir /media/cdrom
mkdir /media/dvdrom
```

Create a minimal /etc/fstab file, adjust the following to taste:

# Device        Mountpoint              FStype          Options         Dump    Pass#
/dev/gpt/swap0  none                    swap            sw              0       0
/dev/cd0        /media/cdrom            cd9660          ro,noauto       0       0
/dev/cd0        /media/dvdrom           udf             ro,noauto       0       0
proc            /proc                   procfs          rw,late         0       0
linproc         /compat/linux/proc      linprocfs       rw,late         0       0

Create a minimal /etc/rc.conf file, adjust the following to taste:

zfs_enable="YES"
background_fsck="NO"
hostname="zfstest.example.org"
ifconfig_em0="DHCP"
ipv6_activate_all_interfaces="YES"
keymap="norwegian.iso"
keyrate="fast"
keybell="off"
font8x16="iso-8x16"
font8x14="iso-8x14"
font8x8="iso-8x8"
dumpdev="/dev/gpt/swap0"

If you decide not to use DHCP, then create a /etc/resolv.conf file and adjust the contents of this file to fit your environment:
```
search example.org
nameserver 2001:db8:1234:1::2
nameserver 2001:db8:1234:1::3
nameserver 192.0.2.2
```
Create a minimal /boot/loader.conf, adjust the following to taste:
```
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"
```
Update 2013-07-21: Newer versions of {,gpt}zfsboot are able to compute the correct value for vfs.root.mountfrom based on the bootfs zpool attribute. Simply avoid setting a value for vfs.root.mountfrom in /boot/loader.conf.
It might make sense to store the workdirs for the ports hierarchy in a filesystem without any compression being applied, so create a minimal /etc/make.conf, adjust the following to taste:
```
WRKDIRPREFIX=/usr/ports/workdirs
```
Set root’s password (and optionally toor’s password):
```
passwd root
passwd toor
```
Set the time zone:
```
tzsetup
```
Create the /etc/mail/aliases.db file:
```
cd /etc/mail
make aliases
```
Leave the chroot environment:
```
exit
```
Unmount the ZFS file systems:
```
cd /
zfs unmount -a
```

Set the final mountpoints:

zfs set mountpoint=legacy zroot
zfs set mountpoint=/home  zroot/home
zfs set mountpoint=/tmp   zroot/tmp
zfs set mountpoint=/usr   zroot/usr
zfs set mountpoint=/var   zroot/var

Unmount the ZFS file systems once more:
```
zfs unmount -a
```
Reboot the system, unload the DVD, and cross your fingers:
1. reboot
2. Unload DVD.
3. Cross fingers.