## ZFS Mirror to RAID-Z

Another geeky post here, because these are somehow easier to write than more personal posts. I recently decided to upgrade my network attached storage (NAS) server from two hard drives to four in order to take advantage of the RAID-Z filesystem. I had always meant to do this, but I was only able to afford two hard disks when I built the server, and had just planned on upgrading eventually.

The most immediate benefit of upgrading to RAID-Z over mirrored disks is the ability to get ~2/3 usable space rather than ~1/2, which means that by doubling the number of hard drives, I’ll be tripling the amount of usable storage space! (not that I was using more than a quarter of the space I had before, but that’s beside the point.)

The challenge here was doing the whole upgrade in-place, because I don’t have a fifth hard drive to back everything up to during the swap. I found a nice description of how to do exactly this on Emil Mikulic’s blog but his description is based on an old version of FreeBSD, which I think caused a couple of problems trying to use his method on my computer. (My server is running FreeBSD 10.1-RELEASE-p15 for amd64 and I think he was using FreeBSD 8.2-RELEASE).

The gist of his technique is to offline one of the mirrored disks in the original zpool:

zpool offline M31 label/m31d2.eli

then create a new zpool backed with the recently offlined disk, the two new disks, and a file-backed vnode:

truncate -s 2000398929920 /disk-backing mdconfig -a -t vnode -S 4096 -f /disk-backing -u 0 zpool create M32 raidz md0 label/m31d2.eli label/m31d3.eli \ label/m31d4.eli

and finally, offline the file-backed vnode

zpool offline M32 md0

It’s then possible to copy everything over from the (now non-redundant) old zpool to the (degraded) new raidz pool, and finally destroy the old zpool and replace the offlined vnode with the fourth hard disk.

The problem I had was with creating a zpool from a sparse file. For some reason zpool create wants to fill the file (which obviously doesn’t work because it’s supposed to be the size of the entire hard disk). I found a forum post suggesting that this is due to ZFS TRIM support (which was introduced in FreeBSD 10, and can be disabled by following the instructions in the FreeBSD manual) but that didn’t seem to fix the problem. The forum post did point out that zpool replace doesn’t fill new disks in the pool. He had a problem with autoexpansion, but I figured I’d give it a shot and fortunately I didn’t have the same issue. The following is my workaround to Emil’s instructions, with all steps included for completeness:

(note: My original zpool was called M31, and it was backed by geli-encrypted disks on /dev/label/m31d1 and /dev/label/m31d2)

glabel label m31d3 /dev/ada3 glabel label m31d4 /dev/ada4 # set up full-disk encryption on the new disks: dd if=/dev/random of=/root/m31d3.key bs=64 count=1 dd if=/dev/random of=/root/m31d4.key bs=64 count=1 geli init -s 4096 -K /root/m31d3.key /dev/label/m31d3 geli init -s 4096 -K /root/m31d4.key /dev/label/m31d4 geli attach -k /root/m31d3.key label/m31d3 geli attach -k /root/m31d4.key label/m31d4   ## Detach one of the original mirrored disks ## from the original zpool zpool detach M31 label/m31d2.eli   ## Make new zpool based on the three free disks, ## and one (small) file: truncate -s 1G /d0 mdconfig -a -t vnode -S 4096 -f /d0 -u 0 zpool create -f M32 raidz md0 label/m31d2.eli \ label/m31d3.eli label/m31d4.eli zpool set autoexpand=on M32 zpool offline M32 md0   ## Now, /d0 got filled with 1G of zeros ## but we want M32 to be based on four 2TB disks. truncate -s 2000398929920 /d1 # get actual size from geom ELI list mdconfig -a -t vnode -S 4096 -f /d1 -u 1 zpool replace M32 md0 md1 ## Somehow that leaves /d1 sparse, but allows the ## zpool to autoexpand   ## Now be sure to remove that vnode, and its file back: zpool offline M32 md1 mdconfig -d -u 0 mdconfig -d -u 1 rm /d0 /d1   ## Now we have two zpools, both degraded ## but usable, and we need to copy the data from ## M31 to M32: zfs get all > /zfs-get-all.txt zfs snapshot -r M31@copier1 zfs send -R M31@copier1 | zfs recv -v -F -d M32   ## You can apparently continue using M31, even making ## snapshots. After that last command finishes you can ## catch up with any changes that were made by shutting ## off any snapshot scripts you have, and running the following: zfs umount -a zfs snapshot -r M31@copier2 zfs rollback M32@copier1 zfs send -R -I M31@copier1 M31@copier2 | zfs recv -v -d M32   ## Then just swap out the new zpool with the old one: zpool destroy M31 zpool export M32 zpool import M32 M31   ## And replace the offlined disk with the real disk zpool replace M31 md1 label/m31d1.eli   zfs get all > /zfs-get-all-new.txt diff /zfs-get-all.txt /zfs-get-all-new.txt ## Files should be identical except things like ## creation dates, available space, etc.

And that’s it! Hopefully after that’s all done what’s left is a ~5TB file system whose contents are identical to the pre-upgrade state of the original zpool. Any one of the four hard disks can fail without any data loss, and data quality is verifiable, and recoverable.

## WTF, File synchronization?

As I mentioned before, this particular super basic computer task seems far from solved. Most recently, I upgraded unison on my server to the latest version from FreeBSD Ports. Immediately after the upgrade, my synchronization script began to fail with this error:

Uncaught exception Failure(“input_value: bad bigarray kind”)

As is the case with most computer bugs, it turns out lots of people on the internet have been complaining about this. Apparently the oCaml compiler on FreeBSD is a version ahead of the one distributed with every linux distribution I use (and apparently OSX, too). I guess something super significant and backwards-incompatible changed between version 4.01 and 4.02 (WTF oCaml, and really WTF unison for being written in such an obscure language).

In any case, I could have reverted the server version, but instead I chose to install ocaml 4.02 from Debian experimental, and compile unison against it. If you’re in a similar situation, and willing to accept a binary from some random stranger on the internet, you can get my copy here. Just put the binary somewhere in your PATH (ie /usr/bin) and rename it to something useful like “unison”. It’s compiled with static links, so it should work on any 64 bit Linux distribution (tested on Debian and Ubuntu).

unison-2.48.3 (for Linux amd-64) sha256: f797c6cd43546faf6e2bf57f04f2d2729a52ed9792b277800087774617b3b3c4

And the corresponding binary for FreeBSD (Compiled from ports, not tested across machines. Might also work on OSX):

## 2002 Digital Gauges

I have a ton of stuff to catch up on posting here. Between AGU, comprehensive exams, several additional sets of photos, and even back to what was supposed to be a series of posts on Kyrgyzstan, I could be kept busy for days. Instead of any of those, I’m gonna do a quick post about this funky little project I’ve been working on lately to bring another small part of my car in to the 21st century.

Quick background: I’ve had this little 1976 BMW 2002 for almost six years now. It’s been a great car, but I’ve been on a slow mission to modernize various aspects of how it works. I love all its funky 1970’s features and quirks, but I also can’t help but tinker with it to try and make it more efficient and reliable. The biggest of these projects so far was the upgrade from mechanical (carbureted) engine management to fully solid-state electronic fuel injection and ignition timing. I used various DIY techniques, and all the information I could find on the interwebs and it took years of tinkering to get it working well, but that project is finally basically over.

The one thing it left me lacking was a tachometer. The old tachometer was driven by an extra pin on the side of the distributor which delivered an oscillating signal between 12V and ground. The new system has no such mechanism. Sure, I could make a circuit to do exactly that, but that got me thinking about all the various inaccuracies associated with mechanical gauges. Why bother computerizing something if the result isn’t more accurate? Of course the look of the gauges is one of those nice ’70s touches I like, so I’d prefer any modifications I do to be behind the scenes and invisible from the driver’s perspective.

With some cursory interweb searching I found out how modern automotive gauges work, and got to work researching everything I could about circuit design and microcontrollers.

Although the project was initially intended to just fix my tachometer, I ended up deciding to first fix my aging speedometer/odometer. The odometer hasn’t been working for ages, the speedometer is wearing out, and it’s a more interesting challenge than the tach in many ways.

I don’t really feel like getting too involved with the technical details right now. You can read more about the details and technical shenanigans here. Instead, here are some pictures of the first prototype:

Aside

Yesterday I decided I was curious enough about how many people read this blog to add web analytics software to count page views. I’m using a local installation of the Piwik web analytics platform. It’s fully free software and endorsed by the Free Software Foundation and many other privacy rights organizations. All data collected stays safely on my server. Basically, I feel weird using tracking software of any sort but I think this solution is sufficiently respectful of users rights that I’m willing to give it a try.

If you want to opt out of having visits to this site recorded, simply set your browser to “tell websites you don’t want to be tracked.” This doesn’t guarantee your privacy in all cases, but my website respects it. I can also recommend the browser plug-ins suggested HERE and the software alternatives to common applications HERE for better privacy in all aspects of computing.

## Duck, Duck, Back to Work!

The bargaining team was in mediation this week from 2 PM Monday to 2:30 AM Tuesday. Then back at it from 10 AM Tuesday to 8 AM Wednesday. They’ve spent 88 hours in mediation sessions over all. They came to a tentative agreement with the administration this morning and we all finally get to go back to work with a new contract (pending ratification by a vote of the general membership).

Our new CBA includes pay raises of 5% per year for the next two years, and a “medical hardship fund” for graduate students who need to take paid leave. The fund is guaranteed in our contract to exist, and to have a specific value based on the number of grads enrolled. There is a committee made up of one member appointed by the president, one appointed by the dean, one appointed by the faculty senate, and four grad students. This committee needs to vote unanimously to change the terms by which the fund operates. If someone feels they were unreasonably denied access to the fund, they can file a grievance through the union and have a 3rd party arbitrator decide. Unfortunately, we did not get a guarantee that anyone who needs it will have access, only that if the fund administrators act “unreasonably” we can file a grievance. This is not paid leave, but hopefully it will be effective enough.

It’s unfortunate that the administration still will not recognize us as employees who play an indispensable role in the university’s operation and community. They prefer to make it as clear as possible that this fund is for students, not workers. Then again, it is nice that this fund will now cover the other 1500 graduate students who are not GTFs, and who are paying tuition to come here for a graduate education. They deserve paid leave too, and it’s good that now they can have access to the same benefits we fought so hard for.

In any case, I’m just glad that the strike is over. Technically because my work is the same as my research this should not have affected me, but I’d be lying to say that it hasn’t impacted my own research. I’ve spent between 5 and 9 hours on picket lines every day for over a week. Walking in a circle can be surprisingly exhausting. Now it’s time to finally finish that AGU poster!

## Duck, Duck, Strike! (Day 2; Wednesday)

Picket lines still strong. I was only out there for a few hours because NSF geodynamics proposals were due and I need funding for my research. Cool news of the day: GTFs at the Oregon Institute for Marine Biology out in Coos Bay have been holding their own pickets and today they went out on a private motorboat to hold a picket line on the ocean. The GTFF officially has a navy! Mediation resumes tomorrow.