ZFS Mirror to RAID-Z

Another geeky post here, because these are somehow easier to write than more personal posts. I recently decided to upgrade my network attached storage (NAS) server from two hard drives to four in order to take advantage of the RAID-Z filesystem. I had always meant to do this, but I was only able to afford two hard disks when I built the server, and had just planned on upgrading eventually.

The most immediate benefit of upgrading to RAID-Z over mirrored disks is the ability to get ~2/3 usable space rather than ~1/2, which means that by doubling the number of hard drives, I’ll be tripling the amount of usable storage space! (not that I was using more than a quarter of the space I had before, but that’s beside the point.)

The challenge here was doing the whole upgrade in-place, because I don’t have a fifth hard drive to back everything up to during the swap. I found a nice description of how to do exactly this on Emil Mikulic’s blog but his description is based on an old version of FreeBSD, which I think caused a couple of problems trying to use his method on my computer. (My server is running FreeBSD 10.1-RELEASE-p15 for amd64 and I think he was using FreeBSD 8.2-RELEASE).

The gist of his technique is to offline one of the mirrored disks in the original zpool:

zpool offline M31 label/m31d2.eli

then create a new zpool backed with the recently offlined disk, the two new disks, and a file-backed vnode:

truncate -s 2000398929920 /disk-backing
mdconfig -a -t vnode -S 4096 -f /disk-backing -u 0
zpool create M32 raidz md0 label/m31d2.eli label/m31d3.eli \

and finally, offline the file-backed vnode

zpool offline M32 md0

It’s then possible to copy everything over from the (now non-redundant) old zpool to the (degraded) new raidz pool, and finally destroy the old zpool and replace the offlined vnode with the fourth hard disk.

The problem I had was with creating a zpool from a sparse file. For some reason `zpool create` wants to fill the file (which obviously doesn’t work because it’s supposed to be the size of the entire hard disk). I found a forum post suggesting that this is due to ZFS TRIM support (which was introduced in FreeBSD 10, and can be disabled by following the instructions in the FreeBSD manual) but that didn’t seem to fix the problem. The forum post did point out that `zpool replace` doesn’t fill new disks in the pool. He had a problem with autoexpansion, but I figured I’d give it a shot and fortunately I didn’t have the same issue. The following is my workaround to Emil’s instructions, with all steps included for completeness:

(note: My original zpool was called M31, and it was backed by geli-encrypted disks on /dev/label/m31d1 and /dev/label/m31d2)

glabel label m31d3 /dev/ada3
glabel label m31d4 /dev/ada4
# set up full-disk encryption on the new disks:
dd if=/dev/random of=/root/m31d3.key bs=64 count=1
dd if=/dev/random of=/root/m31d4.key bs=64 count=1
geli init -s 4096 -K /root/m31d3.key /dev/label/m31d3
geli init -s 4096 -K /root/m31d4.key /dev/label/m31d4
geli attach -k /root/m31d3.key label/m31d3
geli attach -k /root/m31d4.key label/m31d4
## Detach one of the original mirrored disks
## from the original zpool
zpool detach M31 label/m31d2.eli
## Make new zpool based on the three free disks,
## and one (small) file:
truncate -s 1G /d0
mdconfig -a -t vnode -S 4096 -f /d0 -u 0
zpool create -f M32 raidz md0 label/m31d2.eli \
    label/m31d3.eli label/m31d4.eli
zpool set autoexpand=on M32
zpool offline M32 md0
## Now, /d0 got filled with 1G of zeros
## but we want M32 to be based on four 2TB disks.
truncate -s 2000398929920 /d1
    # get actual size from `geom ELI list`
mdconfig -a -t vnode -S 4096 -f /d1 -u 1
zpool replace M32 md0 md1
## Somehow that leaves /d1 sparse, but allows the
## zpool to autoexpand
## Now be sure to remove that vnode, and its file back:
zpool offline M32 md1
mdconfig -d -u 0
mdconfig -d -u 1
rm /d0 /d1
## Now we have two zpools, both degraded
## but usable, and we need to copy the data from
## M31 to M32:
zfs get all > /zfs-get-all.txt
zfs snapshot -r M31@copier1
zfs send -R M31@copier1 | zfs recv -v -F -d M32
## You can apparently continue using M31, even making
## snapshots. After that last command finishes you can
## catch up with any changes that were made by shutting
## off any snapshot scripts you have, and running the following:
zfs umount -a
zfs snapshot -r M31@copier2
zfs rollback M32@copier1
zfs send -R -I M31@copier1 M31@copier2 | zfs recv -v -d M32
## Then just swap out the new zpool with the old one:
zpool destroy M31
zpool export M32
zpool import M32 M31
## And replace the offlined disk with the real disk
zpool replace M31 md1 label/m31d1.eli
zfs get all > /zfs-get-all-new.txt
diff /zfs-get-all.txt /zfs-get-all-new.txt
## Files should be identical except things like
## creation dates, available space, etc.

And that’s it! Hopefully after that’s all done what’s left is a ~5TB file system whose contents are identical to the pre-upgrade state of the original zpool. Any one of the four hard disks can fail without any data loss, and data quality is verifiable, and recoverable.

WTF, File synchronization?

As I mentioned before, this particular super basic computer task seems far from solved. Most recently, I upgraded unison on my server to the latest version from FreeBSD Ports. Immediately after the upgrade, my synchronization script began to fail with this error:

Uncaught exception Failure(“input_value: bad bigarray kind”)

As is the case with most computer bugs, it turns out lots of people on the internet have been complaining about this. Apparently the oCaml compiler on FreeBSD is a version ahead of the one distributed with every linux distribution I use (and apparently OSX, too). I guess something super significant and backwards-incompatible changed between version 4.01 and 4.02 (WTF oCaml, and really WTF unison for being written in such an obscure language).

In any case, I could have reverted the server version, but instead I chose to install ocaml 4.02 from Debian experimental, and compile unison against it. If you’re in a similar situation, and willing to accept a binary from some random stranger on the internet, you can get my copy here. Just put the binary somewhere in your PATH (ie /usr/bin) and rename it to something useful like “unison”. It’s compiled with static links, so it should work on any 64 bit Linux distribution (tested on Debian and Ubuntu).

unison-2.48.3 (for Linux amd-64) sha256: f797c6cd43546faf6e2bf57f04f2d2729a52ed9792b277800087774617b3b3c4

And the corresponding binary for FreeBSD (Compiled from ports, not tested across machines. Might also work on OSX):

unison-2.48.3 (for FreeBSD-10.1 amd-64) sha256: 29e9cbaf1044b121fc25855663087ebce9b0a7d4ee72c285928b9a1e74eab8d4

2002 Digital Gauges

I have a ton of stuff to catch up on posting here. Between AGU, comprehensive exams, several additional sets of photos, and even back to what was supposed to be a series of posts on Kyrgyzstan, I could be kept busy for days. Instead of any of those, I’m gonna do a quick post about this funky little project I’ve been working on lately to bring another small part of my car in to the 21st century.

Quick background: I’ve had this little 1976 BMW 2002 for almost six years now. It’s been a great car, but I’ve been on a slow mission to modernize various aspects of how it works. I love all its funky 1970’s features and quirks, but I also can’t help but tinker with it to try and make it more efficient and reliable. The biggest of these projects so far was the upgrade from mechanical (carbureted) engine management to fully solid-state electronic fuel injection and ignition timing. I used various DIY techniques, and all the information I could find on the interwebs and it took years of tinkering to get it working well, but that project is finally basically over.

The one thing it left me lacking was a tachometer. The old tachometer was driven by an extra pin on the side of the distributor which delivered an oscillating signal between 12V and ground. The new system has no such mechanism. Sure, I could make a circuit to do exactly that, but that got me thinking about all the various inaccuracies associated with mechanical gauges. Why bother computerizing something if the result isn’t more accurate? Of course the look of the gauges is one of those nice ’70s touches I like, so I’d prefer any modifications I do to be behind the scenes and invisible from the driver’s perspective.

With some cursory interweb searching I found out how modern automotive gauges work, and got to work researching everything I could about circuit design and microcontrollers.

Although the project was initially intended to just fix my tachometer, I ended up deciding to first fix my aging speedometer/odometer. The odometer hasn’t been working for ages, the speedometer is wearing out, and it’s a more interesting challenge than the tach in many ways.

I don’t really feel like getting too involved with the technical details right now. You can read more about the details and technical shenanigans here. Instead, here are some pictures of the first prototype:


Yesterday I decided I was curious enough about how many people read this blog to add web analytics software to count page views. I’m using a local installation of the Piwik web analytics platform. It’s fully free software and endorsed by the Free Software Foundation and many other privacy rights organizations. All data collected stays safely on my server. Basically, I feel weird using tracking software of any sort but I think this solution is sufficiently respectful of users rights that I’m willing to give it a try.

If you want to opt out of having visits to this site recorded, simply set your browser to “tell websites you don’t want to be tracked.” This doesn’t guarantee your privacy in all cases, but my website respects it. I can also recommend the browser plug-ins suggested HERE and the software alternatives to common applications HERE for better privacy in all aspects of computing.

Duck, Duck, Back to Work!

The bargaining team was in mediation this week from 2 PM Monday to 2:30 AM Tuesday. Then back at it from 10 AM Tuesday to 8 AM Wednesday. They’ve spent 88 hours in mediation sessions over all. They came to a tentative agreement with the administration this morning and we all finally get to go back to work with a new contract (pending ratification by a vote of the general membership).

Our new CBA includes pay raises of 5% per year for the next two years, and a “medical hardship fund” for graduate students who need to take paid leave. The fund is guaranteed in our contract to exist, and to have a specific value based on the number of grads enrolled. There is a committee made up of one member appointed by the president, one appointed by the dean, one appointed by the faculty senate, and four grad students. This committee needs to vote unanimously to change the terms by which the fund operates. If someone feels they were unreasonably denied access to the fund, they can file a grievance through the union and have a 3rd party arbitrator decide. Unfortunately, we did not get a guarantee that anyone who needs it will have access, only that if the fund administrators act “unreasonably” we can file a grievance. This is not paid leave, but hopefully it will be effective enough.

It’s unfortunate that the administration still will not recognize us as employees who play an indispensable role in the university’s operation and community. They prefer to make it as clear as possible that this fund is for students, not workers. Then again, it is nice that this fund will now cover the other 1500 graduate students who are not GTFs, and who are paying tuition to come here for a graduate education. They deserve paid leave too, and it’s good that now they can have access to the same benefits we fought so hard for.

In any case, I’m just glad that the strike is over. Technically because my work is the same as my research this should not have affected me, but I’d be lying to say that it hasn’t impacted my own research. I’ve spent between 5 and 9 hours on picket lines every day for over a week. Walking in a circle can be surprisingly exhausting. Now it’s time to finally finish that AGU poster!

Duck, Duck, Strike! (Day 8; Tuesday)

Our bargaining team was in mediation yesterday until 2:30 AM, and they’ve been back at it since 10 AM today. I feel like a wimp for complaining about walking in a circle all day. The bargaining team is made up of other GTFs who volunteer their time to make this successful. They aren’t getting paid $300/hr like Jeff, the administration’s union-busting lawyer (who interestingly has no background in labor law, and is currently out on paid sick leave being replaced by a pregnant woman who will soon be taking paid parental leave from her law firm).

As of last Friday this became the longest legal, academic strike in Oregon. It’s now been twice that long, which is very scary. Numbers are still high on the picket lines though. The administration has been bragging that only 22% of scheduled classes have gone untaught, but fail to mention that GTFs teach 33% of classes, which makes that 66% of GTF-run classes that have been affected. Not to mention the grading and research that’s been on hold since last Monday.

After the philosophy department head refused to enter grades for striking GTFs, many students were assigned grades by administrators based on their December 1st grades. When the students complained that they had been counting on taking the finals (as outlined in their syllabus) to improve their grades, their grades got “recalculated” and magically increased. Academic integrity was compromised a long time ago, now the admin is threatening our accreditation. This is absolutely absurd.

In other news, Meaghan just got another blog post about this published in the Huffington Post!

Duck, Duck, Strike! (Day 7; Monday)

Pickets are still holding in there but a lot of GTFs are starting to head home for winter break on plane tickets they bought months ago so numbers seem to be declining. The impact is also less visible (or audible at least) because we’re doing our best to avoid interrupting finals. We’ve moved our loud picket to the front of the alumni center and the other pickets are staying quiet and handing out “finals care packages” with pens and candy to undergrads. The bargaining team is back at the table as of 2:15pm and is still there now.

Positive stuff from today: Our moveon.org petition now has over 15,000 signatures. That means Scott Coltrane has received over 15,000 emails in support of the GTFF. Also, and I hesitate to call this positive, the university is starting to look really really bad in the media. Unfortunately a lot of the nuance of admin vs university gets lost there so the reputation of the whole school is at stake which is why this is only somewhat positive. If you do a news.google.com search for “University of Oregon” (in a clean browser with cleared cookies and logged out of Google) the first two results are about the GTFF strike, the third result is about the admin’s effort to cover up a major sexual assault case earlier this year, the 4th-6th results finally get to sports, and the 7th result is back to news on UO losing students to OSU.

I have to say, the GTFF is one of the most important things to me about UO and the admin’s behavior toward us in bargaining makes me sort of ashamed to study and work here. Phil Knight’s and other wealthy, notoriously anti-union donors involvement in campus funding has always been troublesome but never more apparent than now. Before his job as interim president, Scott Coltrane made a name for himself in sociology with his research “on families, in particular the ways mothers and fathers divide parenting and housework.” By pushing back on paid parental and medical leave he is actively backtracking on his entire academic career. He wouldn’t do that if there weren’t some serious pressure from big donors. If they win this one, it will be clear that this was never our university. It belongs to Chuck Lillis and Nike.

Anyway, for folks who live in Oregon, I highly recommend that you call the governor’s constituents’ line at (503) 378.4582 to tell him that you support the GTFF. The board of trustees answers directly to him so he has a lot of weight in ending this. We know he’s sympathetic to our strike, but actual action from him on our behalf could make or break this.

Duck, Duck, Strike! (Day 4; Friday)

Mediation continued all day with no progress. The terms of the offers the administration is proposing seem to be flexible, but still no movement on any contractual guarantee. Without a legal guarantee, any offer they make is basically meaningless to us. The strike will continue in to next week. Mediation will not resume until the administration brings us an offer with a legal guarantee.

Positive things from pickets today: lots of support from Teamsters. I was at a picket that turned away two semi truck deliveries. One of the drivers tipped me off about three other locations around campus where he was supposed to make deliveries and we got flash pickets to those places in time. Trash is beginning to pile up because Sanipac drivers respect our picket lines, and UPS hasn’t delivered a single package that we know of. Different professors brought us fresh baked bread, fancy cookies, coffee, and pastries. Undergrads have been buying us snacks. The classified staff union bought everyone lunch today. AFT National is taking out web ads on our behalf. Donations have been pouring in to our strike fund. Alumni have been withholding donations to the university and started a letter campaign to the alumni association (that 10k donation to our strike fund would have otherwise gone to the university). We have a moveon.org petition with over 12,000 signatures so far (every time someone signs the petition, president Scott Coltrane gets an email). Over 700 GTFs have been out at pickets in total, with around 200 at any given time, plus the faculty, classified staff, and students who march with us. Even one of the board of trustees members marched with us!

We’re going strong but the admin’s lack of movement has been disturbing. They’ve been destroying the reputation of the University of Oregon, and diminishing our academic integrity. They’ve offered undergrads the choice to accept their grades as they stood on Dec 1, or to have finals administered and graded by random unqualified people from the community and other undergrads. President Coltrane has backtracked on his entire body of research as an academic to deny us something so trivially inexpensive as to barely amount to a rounding error in the university’s annual budget surplus. It’s very disheartening to see the administration burn this university to the ground over a power struggle like this. They know they can afford to wait us out financially, but they lost the public support battle a long time ago. Restoring any trust that the administration prioritizes academic integrity going forward will be very difficult. They’ve all but said directly that academics takes second stage to athletics and branding. As president Coltrane ended his email to the campus community on today’s negotiations, “I hope to be able to take some time, either at the Matthew Knight Arena watch party or at home, to enjoy tonight’s PAC 12 championship game as a reminder of the passion we feel for our University of Oregon.”

Duck, Duck, Strike! (Day 3; Thursday)

Mediation resumed today at 8am, and as of 5pm remains unsuccessful. With NSF proposals out of the way I was on the picket line for 9 hours. The good news today is that we received a very public donation of $10,000 from the partner of one of the anthropology GTFs toward our strike fund (now $170,000). Also, other unions in Eugene have bought us lunch every day so far! Today it came from faculty at Lane Community College.

Duck, Duck, Strike! (Day 2; Wednesday)

Picket lines still strong. I was only out there for a few hours because NSF geodynamics proposals were due and I need funding for my research. Cool news of the day: GTFs at the Oregon Institute for Marine Biology out in Coos Bay have been holding their own pickets and today they went out on a private motorboat to hold a picket line on the ocean. The GTFF officially has a navy! Mediation resumes tomorrow.