Another geeky post here, because these are somehow easier to write than more personal posts. I recently decided to upgrade my network attached storage (NAS) server from two hard drives to four in order to take advantage of the RAID-Z filesystem. I had always meant to do this, but I was only able to afford two hard disks when I built the server, and had just planned on upgrading eventually.
The most immediate benefit of upgrading to RAID-Z over mirrored disks is the ability to get ~2/3 usable space rather than ~1/2, which means that by doubling the number of hard drives, I’ll be tripling the amount of usable storage space! (not that I was using more than a quarter of the space I had before, but that’s beside the point.)
The challenge here was doing the whole upgrade in-place, because I don’t have a fifth hard drive to back everything up to during the swap. I found a nice description of how to do exactly this on Emil Mikulic’s blog but his description is based on an old version of FreeBSD, which I think caused a couple of problems trying to use his method on my computer. (My server is running FreeBSD 10.1-RELEASE-p15 for amd64 and I think he was using FreeBSD 8.2-RELEASE).
The gist of his technique is to offline one of the mirrored disks in the original zpool:
zpool offline M31 label/m31d2.eli
then create a new zpool backed with the recently offlined disk, the two new disks, and a file-backed vnode:
truncate -s 2000398929920 /disk-backing mdconfig -a -t vnode -S 4096 -f /disk-backing -u 0 zpool create M32 raidz md0 label/m31d2.eli label/m31d3.eli \ label/m31d4.eli
and finally, offline the file-backed vnode
zpool offline M32 md0
It’s then possible to copy everything over from the (now non-redundant) old zpool to the (degraded) new raidz pool, and finally destroy the old zpool and replace the offlined vnode with the fourth hard disk.
The problem I had was with creating a zpool from a sparse file. For some reason
`zpool create` wants to fill the file (which obviously doesn’t work because it’s supposed to be the size of the entire hard disk). I found a forum post suggesting that this is due to ZFS TRIM support (which was introduced in FreeBSD 10, and can be disabled by following the instructions in the FreeBSD manual) but that didn’t seem to fix the problem. The forum post did point out that
`zpool replace` doesn’t fill new disks in the pool. He had a problem with autoexpansion, but I figured I’d give it a shot and fortunately I didn’t have the same issue. The following is my workaround to Emil’s instructions, with all steps included for completeness:
(note: My original zpool was called M31, and it was backed by geli-encrypted disks on /dev/label/m31d1 and /dev/label/m31d2)
glabel label m31d3 /dev/ada3 glabel label m31d4 /dev/ada4 # set up full-disk encryption on the new disks: dd if=/dev/random of=/root/m31d3.key bs=64 count=1 dd if=/dev/random of=/root/m31d4.key bs=64 count=1 geli init -s 4096 -K /root/m31d3.key /dev/label/m31d3 geli init -s 4096 -K /root/m31d4.key /dev/label/m31d4 geli attach -k /root/m31d3.key label/m31d3 geli attach -k /root/m31d4.key label/m31d4 ## Detach one of the original mirrored disks ## from the original zpool zpool detach M31 label/m31d2.eli ## Make new zpool based on the three free disks, ## and one (small) file: truncate -s 1G /d0 mdconfig -a -t vnode -S 4096 -f /d0 -u 0 zpool create -f M32 raidz md0 label/m31d2.eli \ label/m31d3.eli label/m31d4.eli zpool set autoexpand=on M32 zpool offline M32 md0 ## Now, /d0 got filled with 1G of zeros ## but we want M32 to be based on four 2TB disks. truncate -s 2000398929920 /d1 # get actual size from `geom ELI list` mdconfig -a -t vnode -S 4096 -f /d1 -u 1 zpool replace M32 md0 md1 ## Somehow that leaves /d1 sparse, but allows the ## zpool to autoexpand ## Now be sure to remove that vnode, and its file back: zpool offline M32 md1 mdconfig -d -u 0 mdconfig -d -u 1 rm /d0 /d1 ## Now we have two zpools, both degraded ## but usable, and we need to copy the data from ## M31 to M32: zfs get all > /zfs-get-all.txt zfs snapshot -r M31@copier1 zfs send -R M31@copier1 | zfs recv -v -F -d M32 ## You can apparently continue using M31, even making ## snapshots. After that last command finishes you can ## catch up with any changes that were made by shutting ## off any snapshot scripts you have, and running the following: zfs umount -a zfs snapshot -r M31@copier2 zfs rollback M32@copier1 zfs send -R -I M31@copier1 M31@copier2 | zfs recv -v -d M32 ## Then just swap out the new zpool with the old one: zpool destroy M31 zpool export M32 zpool import M32 M31 ## And replace the offlined disk with the real disk zpool replace M31 md1 label/m31d1.eli zfs get all > /zfs-get-all-new.txt diff /zfs-get-all.txt /zfs-get-all-new.txt ## Files should be identical except things like ## creation dates, available space, etc.
And that’s it! Hopefully after that’s all done what’s left is a ~5TB file system whose contents are identical to the pre-upgrade state of the original zpool. Any one of the four hard disks can fail without any data loss, and data quality is verifiable, and recoverable.