Replacing drives on AMANDA server

I spent some days last week converting our 32-bit AMANDA server to a 64-bit counterpart using spare but aged hardware. The former AMANDA server ran on very aged hardware in comparison. Going 64-bit also ment turning to ZFS-based storage.

Today, I replaced the two 320 GB first generation SATA drives with two 1 TB third generation SATA drives. The new drives, like their predecessors, are connected to the second generation SATA controller on the motherboard. Replacing the drives is nevertheless an improvement.

In other words, the system went from looking like this:

# gpart show -lp ada0 ada1
=>       34  625142381    ada0  GPT  (298G)
         34          6          - free -  (3.0K)
         40       1024  ada0p1  amanda_gptboot0  (512K)
       1064        984          - free -  (492K)
       2048    8388608  ada0p2  amanda_swap0  (4.0G)
    8390656  616546304  ada0p3  amanda_zroot0  (294G)
  624936960     205455          - free -  (100M)

=>       34  625142381    ada1  GPT  (298G)
         34          6          - free -  (3.0K)
         40       1024  ada1p1  amanda_gptboot1  (512K)
       1064        984          - free -  (492K)
       2048    8388608  ada1p2  amanda_swap1  (4.0G)
    8390656  616546304  ada1p3  amanda_zroot1  (294G)
  624936960     205455          - free -  (100M)

To look like this:

# gpart show -lp ada0 ada1
=>        34  1953525101    ada0  GPT  (932G)
          34           6          - free -  (3.0K)
          40        1024  ada0p1  amanda_gptboot0  (512K)
        1064         984          - free -  (492K)
        2048     8388608  ada0p2  amanda_swap0  (4.0G)
     8390656  1944929672  ada0p3  amanda_zroot0  (927G)
  1953320328      204807          - free -  (100M)

=>        34  1953525101    ada1  GPT  (932G)
          34           6          - free -  (3.0K)
          40        1024  ada1p1  amanda_gptboot1  (512K)
        1064         984          - free -  (492K)
        2048     8388608  ada1p2  amanda_swap1  (4.0G)
     8390656  1944929672  ada1p3  amanda_zroot1  (927G)
  1953320328      204807          - free -  (100M)

Leaving a small piece of free space at the end of each drive will be helpful if and when a future replacement drive is a few blocks shorter than these drives.

I began by replacing the ada1 drive, booted the system into single-user mode off the ada0 drive, partitioned the new ada1 drive as shown above, and wrote new bootcode using

gpart -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

I told ZFS to replace the old gpt/amanda_zroot1 with the new gpt/amanda_zroot1, i.e.

zpool replace amanda_zroot gpt/amanda_zroot1 gpt/amanda_zroot1

ZFS started resilvering the ada1 drive.

# zpool status -v
  pool: amanda_zroot
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Apr 15 10:35:47 2016
        3.31G scanned out of 14.7G at 21.6M/s, 0h8m to go
        3.31G resilvered, 22.56% done
config:

        NAME                        STATE     READ WRITE CKSUM
        amanda_zroot                ONLINE       0     0     0
          mirror-0                  ONLINE       0     0     0
            gpt/amanda_zroot0       ONLINE       0     0     0
            replacing-1             UNAVAIL      0     0     0
              12300099375623714003  UNAVAIL      0     0     0  was /dev/gpt/amanda_zroot1/old
            gpt/amanda_zroot1       ONLINE       0     0     0  (resilvering)

errors: No known data errors
# zpool status -v
  pool: amanda_zroot
 state: ONLINE
  scan: resilvered 14.7G in 0h11m with 0 errors on Fri Apr 15 10:47:37 2016
config:

        NAME                   STATE     READ WRITE CKSUM
        amanda_zroot           ONLINE       0     0     0
          mirror-0             ONLINE       0     0     0
            gpt/amanda_zroot0  ONLINE       0     0     0
            gpt/amanda_zroot1  ONLINE       0     0     0

errors: No known data errors

12 minutes later could I replace the ada0 drive and repeat the steps above.

The resilvering of ada0 took only 8 minutes, clearly showing an improved throughput.

# zpool status -v
  pool: amanda_zroot
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Apr 15 11:03:42 2016
        12.8G scanned out of 14.7G at 29.6M/s, 0h1m to go
        12.7G resilvered, 86.95% done
config:

        NAME                        STATE     READ WRITE CKSUM
        amanda_zroot                ONLINE       0     0     0
          mirror-0                  ONLINE       0     0     0
            replacing-0             UNAVAIL      0     0     0
              16248583458849512152  UNAVAIL      0     0     0  was /dev/gpt/amanda_zroot0/old
              gpt/amanda_zroot0     ONLINE       0     0     0  (resilvering)
            gpt/amanda_zroot1       ONLINE       0     0     0

errors: No known data errors
# zpool status -v
  pool: amanda_zroot
 state: ONLINE
  scan: resilvered 14.7G in 0h7m with 0 errors on Fri Apr 15 11:11:32 2016
config:

        NAME                   STATE     READ WRITE CKSUM
        amanda_zroot           ONLINE       0     0     0
          mirror-0             ONLINE       0     0     0
            gpt/amanda_zroot0  ONLINE       0     0     0
            gpt/amanda_zroot1  ONLINE       0     0     0

errors: No known data errors

Now I told ZFS to start using the extra space by uttering:

zpool online -e amanda_zroot gpt/amanda_zroot0

Here’s how the pool looked before and after the expansion:

# zpool list -v
NAME                    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
amanda_zroot            292G  14.7G   277G      633G     3%     5%  1.00x  ONLINE  -
  mirror                292G  14.7G   277G      633G     3%     5%
    gpt/amanda_zroot0      -      -      -         -      -      -
    gpt/amanda_zroot1      -      -      -         -      -      -
# zpool online -e amanda_zroot gpt/amanda_zroot0
# zpool list -v
NAME                    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
amanda_zroot            926G  14.7G   911G         -     1%     1%  1.00x  ONLINE  -
  mirror                926G  14.7G   911G         -     1%     1%
    gpt/amanda_zroot0      -      -      -         -      -      -
    gpt/amanda_zroot1      -      -      -         -      -      -

A subsequent reboot confirmed the system’s ability to boot off the ada0 drive.

The next time I need to replace one of the current drives, I can run gpart restore -l (lowercase ell) on ada0 or ada1 using these files as (redirected) input:

GPT 128
1 freebsd-boot      40       1024 amanda_gptboot0
2 freebsd-swap    2048    8388608 amanda_swap0
3 freebsd-zfs  8390656 1944929672 amanda_zroot0
GPT 128
1 freebsd-boot      40       1024 amanda_gptboot1
2 freebsd-swap    2048    8388608 amanda_swap1
3 freebsd-zfs  8390656 1944929672 amanda_zroot1