Replacing drives on AMANDA server
I spent some days last week converting our 32-bit AMANDA server to a 64-bit counterpart using spare but aged hardware. The former AMANDA server ran on very aged hardware in comparison. Going 64-bit also ment turning to ZFS-based storage.
Today, I replaced the two 320 GB first generation SATA drives with two 1 TB third generation SATA drives. The new drives, like their predecessors, are connected to the second generation SATA controller on the motherboard. Replacing the drives is nevertheless an improvement.
In other words, the system went from looking like this:
# gpart show -lp ada0 ada1
=> 34 625142381 ada0 GPT (298G)
34 6 - free - (3.0K)
40 1024 ada0p1 amanda_gptboot0 (512K)
1064 984 - free - (492K)
2048 8388608 ada0p2 amanda_swap0 (4.0G)
8390656 616546304 ada0p3 amanda_zroot0 (294G)
624936960 205455 - free - (100M)
=> 34 625142381 ada1 GPT (298G)
34 6 - free - (3.0K)
40 1024 ada1p1 amanda_gptboot1 (512K)
1064 984 - free - (492K)
2048 8388608 ada1p2 amanda_swap1 (4.0G)
8390656 616546304 ada1p3 amanda_zroot1 (294G)
624936960 205455 - free - (100M)
To look like this:
# gpart show -lp ada0 ada1
=> 34 1953525101 ada0 GPT (932G)
34 6 - free - (3.0K)
40 1024 ada0p1 amanda_gptboot0 (512K)
1064 984 - free - (492K)
2048 8388608 ada0p2 amanda_swap0 (4.0G)
8390656 1944929672 ada0p3 amanda_zroot0 (927G)
1953320328 204807 - free - (100M)
=> 34 1953525101 ada1 GPT (932G)
34 6 - free - (3.0K)
40 1024 ada1p1 amanda_gptboot1 (512K)
1064 984 - free - (492K)
2048 8388608 ada1p2 amanda_swap1 (4.0G)
8390656 1944929672 ada1p3 amanda_zroot1 (927G)
1953320328 204807 - free - (100M)
Leaving a small piece of free space at the end of each drive will be helpful if and when a future replacement drive is a few blocks shorter than these drives.
I began by replacing the ada1 drive, booted the system into single-user mode off the ada0 drive, partitioned the new ada1 drive as shown above, and wrote new bootcode using
gpart -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
I told ZFS to replace the old gpt/amanda_zroot1 with the new gpt/amanda_zroot1, i.e.
zpool replace amanda_zroot gpt/amanda_zroot1 gpt/amanda_zroot1
ZFS started resilvering the ada1 drive.
# zpool status -v
pool: amanda_zroot
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Apr 15 10:35:47 2016
3.31G scanned out of 14.7G at 21.6M/s, 0h8m to go
3.31G resilvered, 22.56% done
config:
NAME STATE READ WRITE CKSUM
amanda_zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/amanda_zroot0 ONLINE 0 0 0
replacing-1 UNAVAIL 0 0 0
12300099375623714003 UNAVAIL 0 0 0 was /dev/gpt/amanda_zroot1/old
gpt/amanda_zroot1 ONLINE 0 0 0 (resilvering)
errors: No known data errors
# zpool status -v
pool: amanda_zroot
state: ONLINE
scan: resilvered 14.7G in 0h11m with 0 errors on Fri Apr 15 10:47:37 2016
config:
NAME STATE READ WRITE CKSUM
amanda_zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/amanda_zroot0 ONLINE 0 0 0
gpt/amanda_zroot1 ONLINE 0 0 0
errors: No known data errors
12 minutes later could I replace the ada0 drive and repeat the steps above.
The resilvering of ada0 took only 8 minutes, clearly showing an improved throughput.
# zpool status -v
pool: amanda_zroot
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Apr 15 11:03:42 2016
12.8G scanned out of 14.7G at 29.6M/s, 0h1m to go
12.7G resilvered, 86.95% done
config:
NAME STATE READ WRITE CKSUM
amanda_zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
replacing-0 UNAVAIL 0 0 0
16248583458849512152 UNAVAIL 0 0 0 was /dev/gpt/amanda_zroot0/old
gpt/amanda_zroot0 ONLINE 0 0 0 (resilvering)
gpt/amanda_zroot1 ONLINE 0 0 0
errors: No known data errors
# zpool status -v
pool: amanda_zroot
state: ONLINE
scan: resilvered 14.7G in 0h7m with 0 errors on Fri Apr 15 11:11:32 2016
config:
NAME STATE READ WRITE CKSUM
amanda_zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gpt/amanda_zroot0 ONLINE 0 0 0
gpt/amanda_zroot1 ONLINE 0 0 0
errors: No known data errors
Now I told ZFS to start using the extra space by uttering:
zpool online -e amanda_zroot gpt/amanda_zroot0
Here’s how the pool looked before and after the expansion:
# zpool list -v
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
amanda_zroot 292G 14.7G 277G 633G 3% 5% 1.00x ONLINE -
mirror 292G 14.7G 277G 633G 3% 5%
gpt/amanda_zroot0 - - - - - -
gpt/amanda_zroot1 - - - - - -
# zpool online -e amanda_zroot gpt/amanda_zroot0
# zpool list -v
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
amanda_zroot 926G 14.7G 911G - 1% 1% 1.00x ONLINE -
mirror 926G 14.7G 911G - 1% 1%
gpt/amanda_zroot0 - - - - - -
gpt/amanda_zroot1 - - - - - -
A subsequent reboot confirmed the system’s ability to boot off the ada0 drive.
The next time I need to replace one of the current drives, I can run gpart restore -l (lowercase ell) on ada0 or ada1 using these files as (redirected) input:
GPT 128 1 freebsd-boot 40 1024 amanda_gptboot0 2 freebsd-swap 2048 8388608 amanda_swap0 3 freebsd-zfs 8390656 1944929672 amanda_zroot0
GPT 128 1 freebsd-boot 40 1024 amanda_gptboot1 2 freebsd-swap 2048 8388608 amanda_swap1 3 freebsd-zfs 8390656 1944929672 amanda_zroot1