Resizing mirrored ZFS root pool

George Mamalakis was having trouble with ZFS not detecting and utilizing his resized GPT freebsd-zfs partitions. Marco van Tol came to the rescue with a corrective procedure.

Credit goes to George Mamalakis for raising the issue and to Marco van Tol for the solution. I’m just a simple bystander in comparison to these two, fine gentlemen.

Below are my notes from a simple experiment performed late this evening demonstrating one way of making ZFS grow its pool size. The experiment is based on George Mamalakis’ setup as explained in his post linked above. Note, some system administrator’s liberty has been applied.

Begin by creating a virtual machine in VirtualBox. Create two virtual hard drives, both of size 10 GiB.
Boot off the FreeBSD/amd64 9.0-RELEASE dvd1. (You may choose the i386 version instead, it shouldn’t really matter.)
Interrupt the boot menu by pressing the escape key. Enter the commands load zfs and boot. Choose the Shell menu item when the FreeBSD Installer appears.
Create a freebsd-boot partition, a freebsd-zfs partition, and a freebsd-swap partition, in this particular order for each hard drive. The boot partition should be of size 64 KiB, the ZFS partition should be of size 7 GiB, and the swap partition should consume the remaining free space (about 3 GiB).
```
gpart create -s gpt ada0
gpart create -s gpt ada1
gpart add -t freebsd-boot -s 64k ada0
gpart add -t freebsd-boot -s 64k ada1
gpart add -t freebsd-zfs -s 7G ada0
gpart add -t freebsd-zfs -s 7G ada1
gpart add -t freebsd-swap ada0
gpart add -t freebsd-swap ada1
```

A gpart show command should give this output:

=>      34  20971453  ada0  GPT  (10G)
        34       128     1  freebsd-boot  (64k)
       162  14680064     2  freebsd-zfs  (7.0G)
  14680226   6291261     3  freebsd-swap  (3G)

=>      34  20971453  ada1  GPT  (10G)
        34       128     1  freebsd-boot  (64k)
       162  14680064     2  freebsd-zfs  (7.0G)
  14680226   6291261     3  freebsd-swap  (3G)

Next, write boot code into the PMBR region and into the freebsd-boot partition on each hard drive:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

Continue with creating the mirrored ZFS root pool:

zpool create -o cachefile=/tmp/zpool.cache -m /tmp/zroot zroot mirror ada0p2 ada1p2

The command zpool list should give the following output:

NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zroot  6.94G   120K  6.94G     0%  1.00x  ONLINE  -

Do various housekeeping commands like these two:

zpool set bootfs=zroot zroot
zfs set checksum=fletcher4 zroot

Extract the base OS into place:

cd /tmp/zroot
tar xvvf /usr/freebsd-dist/kernel.txz
tar xvvf /usr/freebsd-dist/base.txz
tar xvvf /usr/freebsd-dist/lib32.txz # Only on AMD64

Copy the zpool.cache file to its final resting place:

cp -p /tmp/zpool.cache /tmp/zroot/boot/zfs/zpool.cache

Create a minimal /boot/loader.conf file:

zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"

Change the mountpoint to legacy and reboot the system:
```
cd /
zfs set mountpoint=legacy zroot
reboot
```
Verify that the system boots, although the boot sequence will complain about missing /etc/rc.conf and /etc/fstab files.

NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zroot  6.94G   651M  6.30G     9%  1.00x  ONLINE  -

Shutdown the system using the command:

shutdown -p now Resizing the hard drives.; logout

Resize the virtual hard drives to 20 GiB each, by issuing these commands on the host system:
```
VBoxManage modifyhd --resize 20480 FreeBSD-9-growpool-ada0.vdi
VBoxManage modifyhd --resize 20480 FreeBSD-9-growpool-ada1.vdi
```
Substitute the names you gave the files when creating the virtual hard drives.
Relaunch the virtual machine, shortly press the F12 key to access VirtualBox’ boot menu, and select booting from the CD-ROM drive by pressing the c key. Hit the enter key as soon as the FreeBSD boot menu appears. We won’t bother loading the zfs.ko nor the opensolaris.ko kernel modules on this occasion. Choose the Shell menu item when the FreeBSD Installer appears.

Running the gpart show command should give this output:

=>      34  20971453  ada0  GPT  (20G) [CORRUPT]
        34       128     1  freebsd-boot  (64k)
       162  14680064     2  freebsd-zfs  (7.0G)
  14689226   6291261     3  freebsd-swap  (3G)

=>      34  20971453  ada1  GPT  (20G) [CORRUPT]
        34       128     1  freebsd-boot  (64k)
       162  14680064     2  freebsd-zfs  (7.0G)
  14689226   6291261     3  freebsd-swap  (3G)

Notice the GPT is corrupt due to the secondary copy not being located at end of the drives.

Correct this fault by issuing the commands:
```
gpart recover ada0
gpart recover ada1
```

A new run of the command gpart show should produce:

=>      34  41942973  ada0  GPT  (20G)
        34       128     1  freebsd-boot  (64k)
       162  14680064     2  freebsd-zfs  (7.0G)
  14689226   6291261     3  freebsd-swap  (3G)
  20971487  20971520        - free -  (10G)

=>      34  41942973  ada1  GPT  (20G)
        34       128     1  freebsd-boot  (64k)
       162  14680064     2  freebsd-zfs  (7.0G)
  14689226   6291261     3  freebsd-swap  (3G)
  20971487  20971520        - free -  (10G)

Remove the freebsd-swap partitions by issuing these two commands:
```
gpart delete -i 3 ada0
gpart delete -i 3 ada1
```
Resize the freebsd-zfs partition to 17 GiB each, leaving about 3 GiB for the swap partitions:
```
gpart resize -i 2 -s 17G ada0
gpart resize -i 2 -s 17G ada1
```

Add the missing freebsd-swap partitions:

gpart add -t freebsd-swap ada0
gpart add -t freebsd-swap ada1

This run of the command gpart show should produce:

=>      34  41942973  ada0  GPT  (20G)
        34       128     1  freebsd-boot  (64k)
       162  35651584     2  freebsd-zfs  (17G)
  35651746   6291261     3  freebsd-swap  (3G)

=>      34  41942973  ada1  GPT  (20G)
        34       128     1  freebsd-boot  (64k)
       162  35651584     2  freebsd-zfs  (17G)
  35651746   6291261     3  freebsd-swap  (3G)

Reboot the system once more, letting it fully boot to multiuser mode.

NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zroot  6.94G   651M  6.30G     9%  1.00x  ONLINE  -

Notice the size attribute is unchanged.

Turn on the autoexpand property:
```
zpool set autoexpand=on zroot
```
This could be done in step 9 of this list, however, it will not make a difference in 9.0-RELEASE.
Put the secondary mirrored partition, i.e. ada1p2 into the offline state, then back to the online state:
```
zpool offline zroot ada1p2
zpool online -e zroot ada1p2
```
Wait until the secondary mirrored partition is fully online and quite possibly resilvered.
Put the primary mirrored partition, i.e. ada0p2 into the offline state, then back to the online state:
```
zpool offline zroot ada0p2
zpool online -e zroot ada0p2
```
Wait until the primary mirrored partition is fully online and quite possibly resilvered.

Issue the zpool status command and verify the zpool zroot is indeed fully operational, i.e. in the ONLINE state.

 pool: zroot
state: ONLINE
scan: resilvered 22K in 0h0m with 0 errors on Thu Mar 15 20:20:32 2012
config:
        NAME        STATE    READ WRITE CKSUM
        zroot       ONLINE      0     0     0
          mirror-0  ONLINE      0     0     0
            ada0p2  ONLINE      0     0     0
            ada1p2  ONLINE      0     0     0

errors: No known data errors

Finally, verify the zpool zroot has indeed grown in size by issuing the zpool list command:

NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zroot  16.9G   651M  16.3G     3%  1.00x  ONLINE  -

Turn off the autoexpand property, if so desired:
```
zpool set autoexpand=off zroot
```