Those who know me well also know that I'm incredibly particular in how I organize things. It should be no surprise that this extends to my expanding collection of e-books. Unfortunately, my recent efforts at organizing things turned into a nightmare that I shall now recount. In short: Screw you, Apple.

My wife, Kestrel, is an extremely avid reader. When I asked her how she was organizing the huge amount of material she's been reading, she responded with Calibre. Calibre doesn't appear to be a very well-written program, but it did get the job done and its interface was easy enough to navigate.

After fixing up the metadata on my EPUBs the way I wanted, I ran into my first annoying problem: Calibre requires that it maintains your library. It's pretty bad when an open source program gives you less choice than Apple's iTunes (which at least allows me to keep audio files where I want them on-disk). So, I scripted up something that would find all the books in Calibre's library and overwrite their counterparts in my library. Stupid, but not insurmountable.

I then synced all my newly-tagged EPUBs to my iPhone with iTunes...or, so I thought. After the sync, all the files still appeared to have the old metadata. I tried syncing again after:

• Deleting everything from my phone
• Starting a new iTunes library
• Deleting any and all cached files I could find from iBooks

...nothing.

I began to think the problem was with Calibre itself, so I picked an EPUB and ripped it apart. It turns out EPUBs are really just zip files (PKZIP is the first 5 bytes) and I pretty quickly found the problem: a file named iTunesMetadata.plist. Turns out, Apple couldn't stand using the same metadata as everyone else, so it had to go and insert its own. Typical.

Of course, after removing the file from every EPUB in my iTunes library (with find . -name "*.epub" -exec zip -d {} iTunesMetadata.plist \;), the metadata still wasn't correct after syncing the books to my device. Stunned, I walked away for a few minutes to think about what I might have missed. I eventually decided to have a look at the original files. This is where I got angry.

When you add a book to the iTunes library with iBooks, iBooks makes a copy of the book and inserts the iTunesMetadata.plist file. If you're going to modify a user's file, this seems reasonable. What isn't reasonable is that iBooks also modifies the original EPUB, inserting the same gorram file. WHY?!

Anyway, removing the iTunesMetadata.plist from the original files, copying them somewhere, then adding those files with iBooks and syncing worked. Finally!

It's really annoying, but it turns out doing all of this is mostly fine for me in the end. I have a file server that is responsible for back-ups of my data, so I want to keep all of my EPUBs there. When adding books, I can simply rsync them locally, add them with iBooks, sync them to my device, and delete the local copies when I'm done. I already have to do this with my music (since iTunes refuses to acknowledge iTunes libraries on file shares).

When I attempted to rsync my files, I noticed a strange problem. rsync was deleting and re-copying a bunch of files that never actually changed. I noticed these files always contained unicode characters. After some searching, I turned up an explanation - OS X sucks at decomposing UTF-8 filenames. Argh.

More searching suggested that rsync 3.0+ would solve my issue. I checked brew and noticed that it had packages for rsync. After installing them, I had rsync version 3.1.1. Unfortunately, my problems weren't solved:

iconv_open("UTF-8", "UTF-8-MAC") failed
rsync error: requested action not supported (code 4) at rsync.c(121) [sender=3.1.0]
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.1]

After raging for a bit, I realized that the original poster above had the same problem. His follow-up question had the answer: Always specify the encoding of the source machine first. If you're copying from Linux to OS X, you want --iconv=utf-8-mac,utf-8. If you're going in the opposite direction, you want --iconv=utf-8,utf-8-mac. In hindsight, I suppose this makes sense. Still frustrating.

Anyway, I finally have my EPUBs set up exactly the way I want them. All I have to do now is wait for Apple to change something and break all of my scripts!

UPDATE (2015-11-19): Or, simply wait for a new version of OS X to come out, apparently. The upgrade to El Capitan removed the symlinking that brew had put there to provide rsync 3.1.1. Took me a few minutes, but a re-install smoothed it all out. If anyone follows the above, but has rsync break on them randomly, make sure it wasn't due to an OS upgrade.