Resizing mirrored ZFS root pool

George Mamalakis was having trouble with ZFS not detecting and utilizing his resized GPT freebsd-zfs partitions. Marco van Tol came to the rescue with a corrective procedure.

Credit goes to George Mamalakis for raising the issue and to Marco van Tol for the solution. I’m just a simple bystander in comparison to these two, fine gentlemen.

Below are my notes from a simple experiment performed late this evening demonstrating one way of making ZFS grow its pool size. The experiment is based on George Mamalakis’ setup as explained in his post linked above. Note, some system administrator’s liberty has been applied.

  1. Begin by creating a virtual machine in VirtualBox. Create two virtual hard drives, both of size 10 GiB.
  2. Boot off the FreeBSD/amd64 9.0-RELEASE dvd1. (You may choose the i386 version instead, it shouldn’t really matter.)
  3. Interrupt the boot menu by pressing the escape key. Enter the commands load zfs and boot. Choose the Shell menu item when the FreeBSD Installer appears.
  4. Create a freebsd-boot partition, a freebsd-zfs partition, and a freebsd-swap partition, in this particular order for each hard drive. The boot partition should be of size 64 KiB, the ZFS partition should be of size 7 GiB, and the swap partition should consume the remaining free space (about 3 GiB).
    gpart create -s gpt ada0
    gpart create -s gpt ada1
    gpart add -t freebsd-boot -s 64k ada0
    gpart add -t freebsd-boot -s 64k ada1
    gpart add -t freebsd-zfs -s 7G ada0
    gpart add -t freebsd-zfs -s 7G ada1
    gpart add -t freebsd-swap ada0
    gpart add -t freebsd-swap ada1
    
  5. A gpart show command should give this output:
    =>      34  20971453  ada0  GPT  (10G)
            34       128     1  freebsd-boot  (64k)
           162  14680064     2  freebsd-zfs  (7.0G)
      14680226   6291261     3  freebsd-swap  (3G)
    
    =>      34  20971453  ada1  GPT  (10G)
            34       128     1  freebsd-boot  (64k)
           162  14680064     2  freebsd-zfs  (7.0G)
      14680226   6291261     3  freebsd-swap  (3G)
    
    
  6. Next, write boot code into the PMBR region and into the freebsd-boot partition on each hard drive:
    gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
    gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
    
  7. Continue with creating the mirrored ZFS root pool:
    zpool create -o cachefile=/tmp/zpool.cache -m /tmp/zroot zroot mirror ada0p2 ada1p2
    
  8. The command zpool list should give the following output:
    NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
    zroot  6.94G   120K  6.94G     0%  1.00x  ONLINE  -
    
  9. Do various housekeeping commands like these two:
    zpool set bootfs=zroot zroot
    zfs set checksum=fletcher4 zroot
    
  10. Extract the base OS into place:
    cd /tmp/zroot
    tar xvvf /usr/freebsd-dist/kernel.txz
    tar xvvf /usr/freebsd-dist/base.txz
    tar xvvf /usr/freebsd-dist/lib32.txz # Only on AMD64
    
  11. Copy the zpool.cache file to its final resting place:
    cp -p /tmp/zpool.cache /tmp/zroot/boot/zfs/zpool.cache
    
  12. Create a minimal /boot/loader.conf file:
    zfs_load="YES"
    vfs.root.mountfrom="zfs:zroot"
    
  13. Change the mountpoint to legacy and reboot the system:
    cd /
    zfs set mountpoint=legacy zroot
    reboot
    
  14. Verify that the system boots, although the boot sequence will complain about missing /etc/rc.conf and /etc/fstab files.
  15. Log in as root and issue the command zpool list. You should see the following output:
    NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
    zroot  6.94G   651M  6.30G     9%  1.00x  ONLINE  -
    
  16. Shutdown the system using the command:
    shutdown -p now Resizing the hard drives.; logout
    
  17. Resize the virtual hard drives to 20 GiB each, by issuing these commands on the host system:
    VBoxManage modifyhd --resize 20480 FreeBSD-9-growpool-ada0.vdi
    VBoxManage modifyhd --resize 20480 FreeBSD-9-growpool-ada1.vdi
    

    Substitute the names you gave the files when creating the virtual hard drives.

  18. Relaunch the virtual machine, shortly press the F12 key to access VirtualBox’ boot menu, and select booting from the CD-ROM drive by pressing the c key. Hit the enter key as soon as the FreeBSD boot menu appears. We won’t bother loading the zfs.ko nor the opensolaris.ko kernel modules on this occasion. Choose the Shell menu item when the FreeBSD Installer appears.
  19. Running the gpart show command should give this output:
    =>      34  20971453  ada0  GPT  (20G) [CORRUPT]
            34       128     1  freebsd-boot  (64k)
           162  14680064     2  freebsd-zfs  (7.0G)
      14689226   6291261     3  freebsd-swap  (3G)
    
    =>      34  20971453  ada1  GPT  (20G) [CORRUPT]
            34       128     1  freebsd-boot  (64k)
           162  14680064     2  freebsd-zfs  (7.0G)
      14689226   6291261     3  freebsd-swap  (3G)
    
    

    Notice the GPT is corrupt due to the secondary copy not being located at end of the drives.

  20. Correct this fault by issuing the commands:
    gpart recover ada0
    gpart recover ada1
    
  21. A new run of the command gpart show should produce:
    =>      34  41942973  ada0  GPT  (20G)
            34       128     1  freebsd-boot  (64k)
           162  14680064     2  freebsd-zfs  (7.0G)
      14689226   6291261     3  freebsd-swap  (3G)
      20971487  20971520        - free -  (10G)
    
    =>      34  41942973  ada1  GPT  (20G)
            34       128     1  freebsd-boot  (64k)
           162  14680064     2  freebsd-zfs  (7.0G)
      14689226   6291261     3  freebsd-swap  (3G)
      20971487  20971520        - free -  (10G)
    
    
  22. Remove the freebsd-swap partitions by issuing these two commands:
    gpart delete -i 3 ada0
    gpart delete -i 3 ada1
    
  23. Resize the freebsd-zfs partition to 17 GiB each, leaving about 3 GiB for the swap partitions:
    gpart resize -i 2 -s 17G ada0
    gpart resize -i 2 -s 17G ada1
    
  24. Add the missing freebsd-swap partitions:
    gpart add -t freebsd-swap ada0
    gpart add -t freebsd-swap ada1
    
  25. This run of the command gpart show should produce:
    =>      34  41942973  ada0  GPT  (20G)
            34       128     1  freebsd-boot  (64k)
           162  35651584     2  freebsd-zfs  (17G)
      35651746   6291261     3  freebsd-swap  (3G)
    
    =>      34  41942973  ada1  GPT  (20G)
            34       128     1  freebsd-boot  (64k)
           162  35651584     2  freebsd-zfs  (17G)
      35651746   6291261     3  freebsd-swap  (3G)
    
    
  26. Reboot the system once more, letting it fully boot to multiuser mode.
  27. Log in as the root user and issue the zpool list command:
    NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
    zroot  6.94G   651M  6.30G     9%  1.00x  ONLINE  -
    

    Notice the size attribute is unchanged.

  28. Turn on the autoexpand property:
    zpool set autoexpand=on zroot
    

    This could be done in step 9 of this list, however, it will not make a difference in 9.0-RELEASE.

  29. Put the secondary mirrored partition, i.e. ada1p2 into the offline state, then back to the online state:
    zpool offline zroot ada1p2
    zpool online -e zroot ada1p2
    
  30. Wait until the secondary mirrored partition is fully online and quite possibly resilvered.
  31. Put the primary mirrored partition, i.e. ada0p2 into the offline state, then back to the online state:
    zpool offline zroot ada0p2
    zpool online -e zroot ada0p2
    
  32. Wait until the primary mirrored partition is fully online and quite possibly resilvered.
  33. Issue the zpool status command and verify the zpool zroot is indeed fully operational, i.e. in the ONLINE state.
     pool: zroot
    state: ONLINE
    scan: resilvered 22K in 0h0m with 0 errors on Thu Mar 15 20:20:32 2012
    config:
            NAME        STATE    READ WRITE CKSUM
            zroot       ONLINE      0     0     0
              mirror-0  ONLINE      0     0     0
                ada0p2  ONLINE      0     0     0
                ada1p2  ONLINE      0     0     0
    
    errors: No known data errors
    
  34. Finally, verify the zpool zroot has indeed grown in size by issuing the zpool list command:
    NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
    zroot  16.9G   651M  16.3G     3%  1.00x  ONLINE  -
    
  35. Turn off the autoexpand property, if so desired:
    zpool set autoexpand=off zroot