Ibid: Incremental backups to infinite disk

You back up the files on your computer regularly, right? You don’t? You’re just asking for trouble. But you already knew that, didn’t you? You’d do periodic backups if you could, but there just are no good backup tools, right? They’re too complicated to use? You don’t have enough CD-R’s to hold all your files? Or you already have too many and are drowning in the clutter?

Well now you can save the lame excuses for why you don’t floss. I’ve written ibid, a simple tool for performing incremental backups on Linux and Unix-like systems — possibly also Mac OS X, I haven’t checked yet. And who knows, maybe it can even be made to run under Windows.

Incremental backups

An incremental backup is a backup of just those files that have changed since the last backup.

Many people use mirroring for backups, periodically duplicating essential files to some destination, such as a spare external hard drive, replacing those copies with fresh copies on each backup. This approach is economical in terms of the storage space used, but it only allows you to restore the latest version of a given file. If you have a file called my_novel.doc and you mirror it to a spare hard drive, then you accidentally delete half of it but don’t notice until after the next time you mirror it to that hard drive, then the pre-accident version of my_novel.doc is gone forever.

With a true backup system like ibid, the pre-accident backup copy of my_novel.doc still exists; the post-accident backup copy is separate. The main drawback to this safer approach, of course, is that the more backups you do, the more space you need.

For a long time I used a makeshift system that backed up files to blank CDs or DVDs, but it was slow and difficult to use and it required my constant attendance (for swapping discs in and out) and very often the disc burner would crap out at the last minute after half an hour of burning and produce an unreadable disc. I didn’t trust the archives that I created with this system; it was annoying to have to render botched discs unreadable with scissors all the time (for privacy reasons); and the more backups I did, the more crowded with discs my safe-deposit box got.

Then I discovered Jungledisk, a commercial “infinite filesystem” service that’s easy to use and has good privacy and security features, not to mention reasonable fees. In conjunction with davfs it can be made to act just like a mounted filesystem.

Ibid doesn’t care if you’re using Jungledisk or some other destination service or media. It simply copies the files you ask for to the target directory tree. The backed-up copies are just plain files residing under their original names in a plain filesystem, so retrieving old data is simple.

Here are ibid’s other main features:

  • Written as a single Perl file, no complicated build/install process;
  • Limit sessions by size;
  • Exclude files and directories by pattern (using Perl “regular expressions”);
  • Files with multiple “hard links” stored only once — link names stored separately;
  • Files re-backed up when they change;
  • Files not re-backed up if they’re merely renamed (the new name is stored separately);
  • Can run unattended, and even at automated intervals (e.g. via cron).

You can download it here. At some point soon perhaps I will create a proper website for it, at which time I will update this blog post with the new information; meanwhile, this is the definitive home of ibid (and I will keep the copy at the above link up to date with bugfixes and other changes).

Here is the documentation for ibid.

Continue reading “Ibid: Incremental backups to infinite disk”

Why I owe Adam Stoller three apologies

In the late 1980s and early ’90s, I worked on the Andrew Project at the Information Technology Center at Carnegie Mellon University, where Adam Stoller was a fellow programmer and system administrator. On at least three occasions I opposed Adam’s advocacy of something or other, only to reverse my opinion later.

Adam kept trying to interest me in XPilot, a space-battle game; but I found it confusing and could never get interested in it. Years later I became a huge XPilot fan. I’ve even contributed several new features.


American pop culture regarded sushi askance during my formative years.

In a 1978 episode of Columbo, “Murder Under Glass” [directed by Jonathan Demme!], chef Louis Jourdan serves a special sushi meal to a visiting Japanese dignitary, who exclaims, “Fugu sashimi — in Los Angeles!”

In The Breakfast Club (1985) there’s this exchange between Judd Nelson and Molly Ringwald: “What’s that?” “Sushi.” “Sushi?” “Rice, raw fish, and seaweed.” “You won’t accept a guy’s tongue in your mouth, and you’re going to eat that?”

And from Desperately Seeking Susan that same year: “Now you have these sushi restaurants. Everyone goes for sushi.” “Sushi… I hate the stuff.” “Although, I tell you, I had some the other day. I took it home, I cooked it, it wasn’t bad. It tasted like fish.”

Adam was also known for his abiding love of sushi. At that time, sushi hadn’t permeated American culture to the extent that it has now. I found the mere idea of eating raw fish to be repellent, and never hesitated to say so to Adam — perhaps even cruelly. Now hardly a day goes by that I don’t crave a meal of sushi.

Once, after a considerable research effort, Adam tried to convince the ITC to switch from using RCS for source code control to using CVS. CVS was fairly new at the time, and I was among those who were distrustful of CVS’s fundamentally different approach than that of RCS. It didn’t require files to be locked before being edited; instead, if two developers edited the same file at the same time, they were later obliged to use a dubious merging algorithm to resolve any conflicts that may have arisen. Collectively we defeated Adam’s proposal and continued using RCS. Within a few years, though, CVS became the de facto standard for source code control in organizations with multiple software developers; no one even considered using RCS anymore. I’ve even contributed some bugfixes to CVS.

Adam, you were right about so many things, and I was wrong.

Greatest hits: The Webby Awards

[Reproduced and edited from e-mail.]

In our last episode, I co-founded the Internet Movie Database. The IMDb team consisted of 15 or 20 film geeks scattered around the globe. We were a “virtual company,” coordinating all our activity via e-mail and the rare conference call. Of all the team members, I was the only one in Northern California and thus became the IMDb’s representative at the first Webby Awards ceremony in 1997.

Andrea and I prepared by shopping for new clothes; the invitation instructed us to “dress swanky.” I wrote and rehearsed an acceptance speech just in case the IMDb beat the other four sites nominated in the Film category.

The dot-com boom had not yet really begun in earnest, and so I was surprised to see that many large corporate sponsors were behind the awards ceremony; several “celebrity judges” had voted on the winners; San Francisco Mayor Willie Brown was the official “welcomer”; and it was slated to be telecast on KRON, PBS, and The Discovery Channel. For a bunch of computer geeks it was an unaccustomed level of attention and glamour, but not unwelcome. With this event, the revenge of the nerds had officially begun.

When I saw all the trendy corporate sponsors with their brands emblazoned here and there at the club, I despaired of our chances of winning. I was sure that the fix was in, and only the moneyed sites would be walking away with the awards. The IMDb was strictly an enthusiast site, not part of a big media conglomerate.

We arrived at Bimbo’s 365 Club a little after 8pm. Being nominees, we were allowed to bypass the long queue of people waiting to get in. We saw a line of limousines dropping off dignitaries. Big searchlights shone upwards to mark the location of the event.

Inside, we received our “nominee” badges and drink tokens. We milled about along with a large number of trendy Multimedia Gulch folks. I’d been to Bimbo’s a few times before, but this was the first time I’d seen all the rooms of the club open and in use.

In the main room, a swing band was playing dance tunes. Andrea and I found a table and had a couple of drinks. I took a last look at my speech, which by now I had comfortably memorized.

The event began. Mayor Brown came out to say a few words about how he loved The Web magazine (which had organized the show), how proud he was that San Francisco was hosting this event, the first of its kind, and so on. He told a couple of good jokes, too, which I promptly forgot. Mayor Brown was something of a national celebrity, and in person it was easy to understand his legendary charm.

The mistress of ceremonies, columnist and playwright Cintra Wilson, then came out, made a few very funny remarks, and got the show under way. Her first rule was that, to keep things moving along, winners would be limited to an acceptance speech of no longer than five words! I tore up my speech — oh well!

There followed a rapid-fire sequence of category and nominee announcements, followed by winners and very quick acceptances. Film was the third category and the IMDb won! I ran up on stage and got a kiss and a trophy. Then I thanked our many thousands of contributors over the past seven years (in almost as many words) and ran off. It was very exciting. As I left the stage, some guy pulled me aside and told me to join the “winners’ circle” in the lounge after the ceremony.

The trophy itself was ugly as sin and tremendously heavy — cubical black base made of solid neutronium, near as I could tell, with a badly-etched plaque stuck on it and supporting a freakish oblong colored glass ovoid, which actually looked kind of cool at one point when I set it down on a table and some light came from behind it. The fifteen winners hefted their trophies around the club like Sisyphus. Some who weren’t careful enough with theirs found that the glass ovoid snapped easily off of the base.

Most of what followed was a blur because I was so thrilled at having won. I do remember the presentation for the best Sex site, though. One of the guys from Bianca’s Smut Shack who came up on stage to accept their award was wearing a Hugh Hefner style robe, smoking a pipe. Big laughs.

After the ceremony, we milled about some more and made our way to the lounge, where several camera crews were at work. Production people from various TV shows asked me to stick around so I could be interviewed. While waiting, Andrea and I met Mayor Brown. He was talking to a woman who had also won a Webby. He asked her in what category she’d won. “Politics,” she said. Mayor Brown turned to me and said, in mock confidentiality, “I want to see the people who won for their sex site!” I thought, “He’d get my vote, if I lived in San Francisco.”

Then I met an interviewer from a Web-related program on PBS. I underwent a very short interview in which I waxed enthusiastic about having won, espoused the IMDb philosophy (i.e., by film fans, for film fans), described the site, and said a few words about the team. I managed to work in much of what had been in my acceptance speech.

A still from The Internet Café

After that, there were two more interviews that were almost identical in content. One was for The Discovery Channel’s web show. The other was for C|net. The Discovery folks told me that my footage would be edited into a segment they’d already done about how the IMDb blows away the corporate movie sites. Some PC magazine reporters spoke to me too.

Andrea and I stuck around for a little while longer as the interviewing wound down and music, dancing, and drinking picked up again. Then we left, drove across town, and had a bite to eat at Mel’s Drive-In — suitable, I thought, since it was the setting for a movie (namely, American Graffiti).

It was great fun. Andrea and I resolved to embark on a career of ingratiating ourselves with politicians, celebrities, and captains of industry in order to get invited to events like this all the time. And in fact we did show up for the 1998 and 1999 Webbies…

(…to be continued…)

I said, “The mree juffer ang flot the marr wuk.” Mree juffer! Mree juffer!

From the I’m-sure-they-meant-well department…

Just now I tried to get some online customer support for my AT&T (formerly SBC [formerly Pacbell]) DSL account. Their website invited me to try an “online chat” with a customer service agent. I gamely clicked the “chat” button and a Java applet window popped up with a text-entry area. In another part of the window I was greeted like this:

Hi Bob Glickstein. How can I assist you?

I typed, “Hi! I have been providing my own SMTP service for my domains but would like to start using an AT&T relay as a smart host (to get around some ISPs’ IP-blocks). What server address can I use and what settings (if any) do I need?”

Here’s how it appeared in the chat window after I pressed return:


I typed, “Whoa, wtf happened to my message?”


“I’ve gotten gibberish from tech support before but this is ridiculous.”


The AT&T guy wrote:

Bob, I apologize, I am unable to understand your words.

I shouldn’t wonder!

As I am only able to assist you in English language.

Thanks for clearing that up. I decided to try putting a space in between each letter. “M a y b e i f I t y p e l i k e t h i s”


“T h e w e b U I i s m a n g l i n g m y w o r d s”


“I h a v e t o p u t a s p a c e b e t w e e n e a c h l e t t e r”


AT&T wrote:

Bob, I suggest that you restart this chat session.

“O K”



The words are really appearing as disturbed.

No kidding. Clicking the “hang up” icon to end the session didn’t work. I dismissed the window and went back through the website to launch a new chat but was placed into the same session, with the same bug. A bit more hilarity ensued (of the rapidly diminishing kind) and then I tried to tell the guy “This is the most broken thing I have ever used,” “I give up,” and “Thanks anyway.”


Though it prevented me from doing what I wanted to do, I can still admire this bug for the impressively gigantic fuck-up it is. I can’t help but wonder if AT&T’s customer support tool was written by disgruntled post-merger SBC or Pacbell engineers in a kind of removing the W’s moment.

Time to ditch AOL

AOL is not your friend.

They use dirty tricks to prevent you from canceling your account. (Someone recorded his attempt to cancel his account. Check it out, it’s unbelievable.)

They censor your e-mail if it’s critical of AOL.

They released your search history along with that of 658,000 other users, complete with identifying information so everyone in the world can tell who did the search for e.g. “cocaine in urine.”

And of course their arbitrary e-mail blocking policies prevent about half of the mail that I send from ever reaching you or any of the other folks on AOL with whom I (try to) maintain a correspondence.

The good news is that there are plenty of better alternatives:

Switch now. Switch now. Switch now!