iBanjo

As many folks know, I’m a huge fan of Interactive Fiction (text adventures). I’ve been working on a z-machine interpreter for Android, as well one that runs in python.

As of today, my iPhone is able to download ‘legitimate’ apps from the AppStore, and there’s still no z-machine interpreter available. For over a year now, there’s been one which requires a jailbroken iPhone out there, but jailbreaking isn’t an option for me (since Google owns my phone.)

My hopes got up when I discovered that the author of Zoom (the best z-machine app for Mac OS X) is actively working on a legitimate iPhone port. However, something he posted really disturbs me:

“A more serious issue is that Apple’s SDK license prohibits downloading code to interpret: this means that it would be impossible to load any games that were not bundled with the interpreter. I think this is probably a fatal problem: it seems doubtful that many IF authors will be willing to pay the $99 required to get their work onto the iPhone – plus it would mean no Zork, ever.”

While it sure is convenient that iPhone users only have one place to check for apps, this really scares me. I’m guessing that the license is intended to prevent people from distributing generic JVM or CLR machines that can download and run any old code, thereby circumventing Apple’s ability to vet applications. But clearly their Safari web browser already downloads and interprets Javascript apps, right? Where does one draw the line? We’re talking about a VM to play text-adventures — would Apple consider the fetching of text-adventures dangerous?

More and more, it’s clear to me that Apple is just as evil as Microsoft, they’re just not as big and powerful (yet)… and they have better taste. Maybe I should give up on issue and just wait for my Android phone at the end of the year. It will be a truly open OS, and I’ll be able to download and run whatever the heck I want.

I’m done with a nice 12-day vacation. I was definitely on the verge of burning out; unproductive at the office and cranky at home all the time. I really needed to step back and forget about computers for a while.

What I did instead:

• Went on a family vacation to Dubuque, Iowa, a mere 3 hour drive from Chicago.
  • Stayed in a hotel-waterpark with wife and 2 year old. Waterpark every day.
  • Horse-carriage rides.
  • Mississippi paddleboat rides.
  • Mississippi Aquarium Museum.
  • Lots of ice cream.
• Finally finished reading Watership Down.
• Started reading The Compassionate Carnivore, which gives me a warm fuzzy feeling that I’m not alone in being a freerange-itarian.
• Read the through the new 4th edition D&D Player’s Handbook. I’m a geek. I like roleplaying games. Sue me.
• Went to the Indiana Dunes — at a friend’s house — for the 4th of July.
• Went to the Bristol Renaissance Faire
• Shot 300 photos. A few of them were pretty good.
• Beta-tested an excellent new text-adventure game written by a friend; it should be available to the public by August 1st.
• Played my banjo at the usual Friday night jam.

Time to go back to Google; I’m continuing to lead a team whose goal is to make Google Code’s Subversion service as fast and scalable as possible.

No, really. Seriously!

After nearly two years of work, it’s been released to the public. Semi-intelligent tracking of merges, sparse directories, interactive conflict resolution, and much much more all described here. I had previously posted about how easy it is to manage feature branches; now you can try it yourself.

Visit subversion.tigris.org to download. (It make take a few days for volunteers to update binary packages from the source code.) Also, we’ve just about finished up the new edition of the Subversion Book, which now covers 1.5. You can read it online, or wait a couple of months to buy a hardcopy from O’Reilly.

I’ve got a lot to say today!

I want to chat about something that I’ve never noticed before, but probably should have. There’s always been a stereotype out there of programmers being nerdy, anti-social people (Q: How do you know when an engineer is outgoing? A: He looks at your shoes!). But my revelation of the week is that most programmers seem to be really insecure about their work. I mean: really, really insecure.

My buddy Fitz and I have long preached about best practices in open source software development — how one should be open and transparent with one’s work, accept code reviews, give constructive criticism, and generally communicate as actively as possible with peers. One of the main community “anti-patterns” we’ve talked about is people writing “code bombs”. That is, what do you do when somebody shows up to an open source project with a gigantic new feature that took months to write? Who has the time to review thousands of lines of code? What if there was a bad design decision made early in the process — does it even make sense to point it out? Dropping code-bombs on communities is rarely good for the project: the team is either forced to reject it outright, or accept it and deal with a giant opaque blob that is hard to understand, change, or maintain. It moves the project decidedly in one direction without much discussion or consensus.

And yet over and over, I’m gathering stories that point to the fact that programmers do not want to write code out in the open. Programmers don’t want their peers to see mistakes or failures. They want to work privately, in a cave, then spring “perfect” code on their community, as if no mistakes had ever been made. I don’t think it’s hubris so much as fear of embarrassment. Rather than think of programming as an inherently social activity, most coders seem to treat it as an arena for personal heroics, and will do anything to protect that myth. They’re fine with sharing code, as long as they present themselves as infallible, it seems. Maybe it’s just human nature.

Check out some of these stories I’ve collected:

• Requests at the Google I/O booth: A couple of weeks ago when my team was at the Google I/O conference, we ran a booth demonstrating our Open Source Project Hosting service. Over and over, we kept getting requests like this:
  

  “Can you guys please give subversion on Google Code the ability to hide specific branches?”

  “Can you guys make it possible to create open source projects that start out hidden to the world, then get ‘revealed’ when they’re ready?”

  Translation: “I don’t want people to see my work-in-progress until it’s perfect.”

• Requests on the Google Code mailing list: Sometimes users need their googlecode.com svn repositories wiped clean. Legitimate reasons include the accidental commit of sensitive data, or the need to load code history in from a different svn repository. But most of the time we get (invalid) requests like this:
  

  “Hi, I want to rewrite all my code from scratch, can you please wipe all the history?”

  Translation: “I don’t want people to be able to find my old code, it’s too embarrassing.” Call it vanity, call it insecurity… the bottom line is that coders want prior mistakes or failures to be erased from history.

• Code-reviews taken as personal attacks. Fitz tells a funny anecdote about a friend of his who went from the open source world to a corporate software job. Vastly paraphrased:
  

  During his first week, he started emailing friendly code reviews to each of his coworkers, receiving strange stares in turn. Eventually his boss called him into his office:

  “You know, you really need to stop with the negative energy. Your peers say that you’re constantly criticizing everything they do.”

  Moral: not only is code review not the norm in corporate environments, most programmers are unable to separate their fragile egos from the code they write. Repeat after me: you are not your code!

• Distributed version control — in a cave. A friend of mine works on several projects that use git or mercurial. He gave me this story recently. Basically, he was working with two groups on a project. One group published changes frequently…
  

  “…and as a result, I was able to review consistently throughout the semester, offering design tweaks and code reviews regularly. And as a result of that, [their work] is now in the mainline, and mostly functional. The other group […] I haven’t heard a peep out of for 5 months. Despite many emails and IRC conversations inviting them to discuss their design and publish changes regularly, there is not a single line of code anywhere that I can see it. […] Last weekend, one of them walked up to me with a bug […] and I finally got to see the code to help them debug. I failed, because there are about 5000 lines of crappy code, and just reading through a single file I pointed out two or three major design flaws and a dozen wonky implementation issues. I had admonished them many times during these 5 months to publish their changes, so that we (the others) could take a look and offer feedback… but each time met with stony silence. I don’t know if they were afraid to publish it, or just don’t care. But either way, given the code I’ve seen, the net result is 5 wasted months.”

  Before you scream; yes yes, I know that the potential for cave-hiding and writing code bombs is also possible with a centralized version control system like Subversion. But my friend has an interesting point:

  “I think this failure is at least partially due to the fact that [DVCS] makes it so damn easy to wall yourself into a cave. Had we been using svn, I think the barrier to caving would have been too high, and I’d have seen the code.”

  In other words, yes, this was fundamentally a social problem. A team was embarrassed to share code. But because they were using distributed version control, it gave them a sense of false security. “See, we’re committing changes to our repository every day… making progress!” If they had been using Subversion, it’s much less likely they would have sat on a 5000 line patch in their working copy for 5 months; they would have had to share the work much earlier. Moral: even though one shouldn’t depend on technical solutions to social problems, default tool behaviors matter a lot. This was my main theme way back when I wrote about the risks of distributed version control.

OK, so what’s the conclusion here? People are scared of sharing their unfinished work, plain and simple. I know this isn’t headline news to most people, but I really think I’ve been in deep denial about this. I’m so used to throwing my creative output up for constant criticism, that I simply expect everyone else to do it as well. I think of it as the norm, and I can’t comprehend why someone wouldn’t want to do that… and yet clearly, the growing popularity of distributed version control shows just how thrilled people are to hide their work from each other. It’s the classic “testimonial” for systems like git (taken from a blog comment):

“Don’t tell me I should cooperate with other people at the beginning and publish my modification as early as possible. I do cooperate with other people but I do want to do some work alone sometimes.”

Hm, okay. Please just don’t work alone for too long!

Sidetracking a wee bit, I think this is why I prefer Mercurial over Git, having done a bit of research and reading on both systems. Git leans much more heavily towards cave-hiding, and I don’t like that. For example, the ‘git rebase’ command is a way of effectively destroying an entire line of history: very powerful, sure, but it’s also a way of erasing your tracks. Rather than being forced to merge your branch into a parent line, just pretend that your branch was always based on the latest parent line! Another example: when it comes to pushing and pulling changesets, Mercurial’s default behavior is to exchange all history with the remote repository, while git’s default behavior is to only push or pull a single branch — presumably one that the user has deemed fit for sharing with the public. In other words, git defaults to all work being private cave-work, and is happy to destroy history. Mercurial shares everything by default, and cannot erase history.

I know this post has been long, but let me stand on my soapbox for a moment.

Be transparent. Share your work constantly. Solicit feedback. Appreciate critiques. Let other people point out your mistakes. You are not your code. Do not be afraid of day-to-day failures — learn from them. (As they say at Google, “don’t run from failure — fail often, fail quickly, and learn.”) Cherish your history, both the successes and mistakes. All of these behaviors are the way to get better at programming. If you don’t follow them, you’re cheating your own personal development.

Phew, I feel better now.

(This page has been translated into Spanish language by Maria Ramos.)

In case you forgot, I used to have an active career in as a composer for theater. From 1995-2005 my buddy Andre and I worked our way up the theater ladder, starting as a couple of naive college kids who had written a college rock-musical adaptation based on Dante’s Inferno. We wrote hundreds of scores for plays, and at least five more musicals as contracted by various small theaters. Over the years, though, our career paths slowly diverged. As the theaters got bigger and more professional, rehearsals moved to daytime hours — and thus we had to quit our day jobs to keep going. Andre took the leap to become a “pro” designer; with my family and mortgage, though, I wasn’t able to bring myself to walk away from the lucrative and exciting world of professional software development.

Andre now travels around the country writing scores for dozens of regional theaters, and he’s been gracious enough to let me ride his coattails now and then. When a rare “musical theater” opportunity presents itself, we’ll still work together in the studio like the old days. Since I had a kid in 2005, I’ve had even less time to work with Andre, though we did write a children’s musical adaptation of Ray Bradbury’s “Dandelion Wine” for the Steppenwolf theater last year.

The work I’m most proud of, though, was a very dark musical adaptation of Sherwood Anderson’s Winesburg, Ohio. If you’re not familiar with the book, you should be — it’s a collection of twisted short stories about life in a small town around 1900. Think “Our Town” on crack. Sherwoood Anderson’s book was scandalous when it was released about about 100 years ago, but his groundbreaking style had a (self-admitted) heavy influence on later writers such as Steinbeck, Hemingway, and Faulkner.

In any case, as with all decent musical theater, our “Winesburg” show has gone through numereous evolutionary iterations. It started as a tiny production in 2002, followed by more workshops, followed by a bigger production in Chicago which garnered a regional Jeff Award… then a workshop in NYC, a production by some kids at an arts college in Connecticut, and finally a much larger regional production in Philadelphia which won a Barrymore Award. (The Philly show actually produced box-office royalities, which paid for my banjo!) While I’m skeptical the show will ever go to Broadway, I do hope it gets refined and popular enough to get published and make the rounds at regional theaters around the country.

So: we’re ready for round 7! The show will be playing at the Kansas City Repertory in March 2009. Mark your calendars. One more chance to improve, rewrite, or add new music. I’m pretty excited to be involved again.

Incidentally, this musical is the reason why — about five years ago — I gave up my life as a jazz pianist for folk music on stringed instruments. It was the first show Andre and I had written where the piano plays only a minor role. Many songs have no piano at all, in fact. I remember running rehearsals without Andre, and discovering that certain songs simply couldn’t be rendered at the piano… the only solution was for me to quickly take some guitar lessons. From there, my teacher introduced me bluegrass, and then off I sailed into banjo-land. I’ve not really gone back to the piano since then!

Allow me to geek out for a second. I’ve come up with a new recipe for ultimate geekiness:

1. Build this c-compiler for the z-machine platform.
2. Use the c-compiler to build this super-simple lisp interpreter.
3. Distribute the lisp interpreter as a .z5 game
4. Profit!

If you have no idea what any of this means, it’s ok. I have a strange sense of humor today.

As I’ve mentioned in other posts, the Subversion project is on the verge of releasing version 1.5, a culmination of nearly two years of work. The release is jam-packed with some huge new features, but the one everyone’s excited about is “merge tracking”.

Merge-tracking is when your version control system keeps track of how lines of development (branches) diverge and re-form together. Historically, open source tools such as CVS and Subversion haven’t done this at all; they’ve relied on “advanced” users carefully examining history and typing arcane commands with just the right arguments. Branching and merging is possible, but it sure ain’t easy. Of course, distributed version control systems have now started to remove the fear and paranoia around branching and merging—they’re actually designed around merging as a core competency. While Subversion 1.5 doesn’t make it merging as easy as a system like Git or Mercurial, it certainly solves common points of pain. As a famous quote goes, “it makes easy things easy, and hard things possible.” Subversion is now beginning to match features in larger, commercial tools such as Clearcase and Perforce.

My collaborators and I are gearing up to release a 2nd Edition of the free online Subversion book soon (and you should be able to buy it from O’Reilly in hardcopy this summer.) If you want gritty details about how merging works, you can glance over Chapter 4 right now, but I thought a “nutshell” summary would make a great short blog post, just to show people how easy the common case now is.

1. Make a branch for your experimental work:
   
   $ svn cp trunkURL branchURL
   $ svn switch branchURL

2. Work on the branch for a while:
   
   # ...edit files
   $ svn commit
   # ...edit files
   $ svn commit

3. Sync your branch with the trunk, so it doesn’t fall behind:
   
   $ svn merge trunkURL
   --- Merging r3452 through r3580 into '.':
   U button.c
   U integer.c
   ...

   $ svn commit

4. Repeat the prior two steps until you’re done coding.
5. Merge your branch back into the trunk:
   
   $ svn switch trunkURL
   $ svn merge --reintegrate branchURL
   --- Merging differences between repository URLs into '.':
   U button.c
   U integer.c
   ...

   $ svn commit


6. Go have a beer, and live in fear of feature branches no more.

Notice how I never had to type a single revision number in my example: Subversion 1.5 knows when the branch was created, which changes need to be synced from branch to trunk, and which changes need to be merged back into the trunk when I’m done. It’s all magic now. This is how it should have been in the first place. 🙂

Subversion 1.5 isn’t officially released yet, but we’re looking for people to test one of our final release candidate source tarballs. CollabNet has also created some nice binary packages for testing, as part of their early adopter program. Try it out and report any bugs!

Remember that awesome folding travel banjo I bought a year ago? It was really cool, but I traded it away for something better.

My original itch was the fact that I was flying to California (and other places) at least five times per year, and wanted to be able to join in jams in other cities. It’s really nervewracking to carry a banjo on a plane — usually the attendants will let you store it in the coat-hanging closet, but you live in constant fear of being forced to “gate check” the thing into the bottom of the plane. And that means burly men throwing the thing around. You can fill your hardshell case with bubble wrap all you want, but there are still a lot of horror stories out there. So the Tranjo was perfect: the neck just popped off (with strings still attached!), and the whole thing fit in a backpack. I went through airport security five times last year before anyone even noticed I had a banjo in my bag — it only got searched on the 6th flight. Great peace of mind!

Unfortunately, the Tranjo had one big drawback: it was too quiet. I’d take it to jams and couldn’t even hear myself playing. The instrument was great for practicing quietly in hotel rooms, or playing solo around a campfire (which I did once)… but that’s it. No group jamming. So, I stopped carrying it with me on trips, and discovered it was easier to just ask friends to lend me banjos on the other side. Turns out there are several Googlers in Mountain View who have banjos I can borrow. 🙂

Thus, I decided to trade in my Tranjo for a really nice “traditional” open-back banjo. It’s a beautiful, light-weight thing, with a rock maple rim and mahogany neck. It sounds great, and is perfect for old-time “clawhammer” frailing. (Clawhammer is a whole different school of banjo-playing that doesn’t involve picks at all—the music pre-dates bluegrass by centuries, long before the bluegrass-heads added resonators to make banjos louder.) The banjo is made by a single artisan in Michigan, Bart Reiter. So now I have my big-ass resonator banjo for bluegrass jams, and my smaller (less expensive) open-back banjo for camping, car trips, and practicing at my office. It’s a joy to play!

(Pictures below are taken from Turtle Hill Banjo’s website.)

According to Google Analytics, one of the most heavily trafficked posts on my blog is the one I wrote years ago, the Risks of Distributed Version Control. It’s full of a lot of semi-angry comments about how wrong I am. I thought I would follow up to that post with some newer thoughts and news.

I have to say, after using Mercurial for a bit, I think distributed version control is pretty neat stuff. As Subversion tests a final release candidate for 1.5 (which features limited merge-tracking abilities), there’s a bit of angst going on in the Subversion developer community about what exactly the future of Subversion is. Mercurial and Git are everywhere, getting more popular all the time (certainly among the 20% trailblazers). What role does Subversion — a “best of breed” centralized version control system — have in a world where everyone is slowly moving to decentralized systems? Subversion has clearly accomplished the mission we established back in 2000 (“to replace CVS”). But you can’t hold still. If Subversion doesn’t have a clear mission going into the future, it will be replaced by something shinier. It might be Mercurial or Git, or maybe something else. Ideally, Subversion would replace itself. 🙂 If we were to design Subversion 2.0, how would we do it?

Last week one of our developers wrote an elegant email that summarizes a potential new mission statement very well. You should really read the whole thing here. Here’s a nice excerpt:

I'm pretty confident that, for a new open source project of non-huge
size, I would not choose Subversion to host it [...]
 
So does that mean Subversion is dead? That we should all jump ship
and just write a new front-end for git and make sure it runs on
windows?

Nah. Centralized version control is still good for some things:

* Working on huge projects where putting all of the *current* source
  code on everyone's machine is infeasible, let alone complete
  history (but where atomic commits across arbitrary pieces of the
  project are required).
* Read authorization! A client/server model is pretty key if you
  just plain aren't allowed to give everyone all the data. (Sure,
  there are theoretical ways to do read authorization in distributed
  systems, but they aren't that easy.)

My opinion? The Subversion project shouldn't spend any more time
trying to make Subversion a better version control tool for non-huge
open source projects. Subversion is already decent for that task, and
other tools have greater potential than it. We need to focus on
making Subversion the best tool for organizations whose users need to
interact with repositories in complex ways[...]

I’ve chatted with other developers, and we’ve all come to some similar private conclusions about Subversion’s future. First, we think that this will probably be the “final” centralized system that gets written in the open source world — it represents the end-of-the-line for this model of code collaboration. It will continue to be used for many years, but specifically it will gain huge mindshare in the corporate world, while (eventually) losing mindshare to distributed systems in the open-source arena. Those of us living in the open source universe really have a skewed view of reality. From where we stand, it may seem like “everyone’s switching to git”, but then when you look at a graph like the one below (which shows all public (not private!) Apache Subversion servers discoverable on the internet), you can see that Subversion isn’t anywhere near “fading away”. Quite the opposite: its adoption is still growing quadratically in the corporate world, with no sign of slowing down. This is happening independently of open source trailblazers losing interest in it. It may end up becoming a mainly “corporate” open source project (that is, all development funded by corporations that depend on it), but that’s a fine way for a piece of mature software to settle down. 🙂

It’s astounding to me that Americans tolerate our mobile phone landscape. Imagine going into a computer store, and being told that the computer you buy can only run on 1 of 3 different internets. And that it comes with all the software pre-installed, and that the software can’t be changed. (OK, well, you can add more software from a small, restricted supply provided by the computer vendor, and only for steep prices.) The hardware is opaque. The operating system is opaque. You have no freedom whatsover. Would you buy this computer? Millions of Americans buy cellphones like this, and don’t think twice. Meanwhile in Japan (which is 5 years in the future) I’ve been told that the phones are so powerful and usable that they’ve actually become a replacement for laptop computers. People spend more time emailing, instant messaging, websurfing, and shopping on their phones than they do on regular computers. Have you seen any phone in the U.S. with a user interface that would allow you to do that?

Starting last fall, I’ve been pretty excited about the new generation of “smartphones” coming out. We’re starting to get closer to the ideal of “computer in the pocket.” The iPhone I carry around with me is the first phone I’ve ever had which hasn’t frustrated me. It’s a real pleasure to use. And it has set the bar incredibly high — it’s like carrying a 1998 computer wherever you go. No, it can’t do everything my desktop computer can do, but even a computer from 1998 is still pretty handy: a real web browser with fonts and CSS and javascript; a beautiful finger-driven email reader; and now that Apple has finally allowed developers to write native applications with the iPhone SDK, it can even play OpenGL games that appear to be from 1998. But again… it’s in my pocket. In my pocket. As the VC guy in the Apple iPhone SDK video, the whole paradigm is changing. The computer in my pocket knows who I am, and it knows where I am. It’s intensely personal, and will change the computing game as much as the personal computer changed things.

I sound like an Apple commercial, for sure, but I’m also still really hopeful for Google’s Android platform as a major contender. Android isn’t a specific phone like the iPhone (or “gPhone”, as some have said): it’s a whole class of phones. Google got a bunch of phone manufacturers together, got them to agree on a hardware platform, and then wrote a complete Linux/Java-based phone operating system to run on this hardware specification. The entire operating system will be 100% open source when it gets released later this year, and I’ve even started learning how to write applications for it, using the Android SDK. (My first project has been to help someone write a z-machine of course, so you can run text adventures on your Android phone!) You can bet that when the first batch of Android phones is released later this year, I’ll be ditching my iPhone for one. 🙂

What’s interesting to me, it seems, is that Apple and Google are now about to compete head-to-head in this market, but with completely different philosophies. Apple is the Cathedral, Google is betting on the Bazaar.

Over in Apple’s universe, there is only one single phone. The hardware and software of the phone are completely secret, and tightly controlled by a single entity. Even the distribution of applications is centralized and tightly controlled: authors must distribute them only through Apple’s iTunes Store, and only after Apple has approved them as legitimate.

In Google’s corner, though, everything is open. The hardware is merely a spec — dozens or hundreds of phones will be created that are compatible, allowing users to choose the form factors and features they want. The operating system is completely open, effectively part of the public domain. Anyone can examine or modify the system, and I expect multiple ‘distributions’ to be released with different purposes, just as there are multiple Linux distros for my computer. And as with any normal computer platform, absolutely anyone can write an application anywhere and give it to anyone else (“caveat emptor” — which means “we hope you like it!”).

I know it’s a cliche analogy, but the two worlds sort of feel like the difference between a centrally-planned, tightly-controlled economy versus a big free market. Who will win? People could argue that capitalism has historically been more successful than planned economies. People could also argue that the chaotic Windows/PC market has historically been more successful than the centralized universe of Mac computers. But Macs are starting to make a big comeback now. There’s clearly a large segment of the population that’s willing to give up some freedom for the convenience of not having to make choices. Heck, I have a Mac and love it. My days of building PCs from parts and messing around with Linux software packages are long over; my time is too valuable, and Macs Just Work. I wonder if after I have a long affair with my Android phone I’ll eventually end up going back to an iPhone? 🙂