iBanjo

11/17/07: Before posting an angry comment about this post, please see the follow-up post!

Disclaimer: I’m going to make some crazy sweeping generalizations — ones which are based on my 12 years of observing the software development industry. I’m aware that I’m drawing some oversimplified stereotypes, but I think most of my peers who work in this industry will nod their head at some point, able to see the grains of truth in my characterizations.

Two Types of Programmers

There are two “classes” of programmers in the world of software development: I’m going to call them the 20% and the 80%.

The 20% folks are what many would call “alpha” programmers — the leaders, trailblazers, trendsetters, the kind of folks that places like Google and Fog Creek software are obsessed with hiring. These folks were the first ones to install Linux at home in the 90’s; the people who write lisp compilers and learn Haskell on weekends “just for fun”; they actively participate in open source projects; they’re always aware of the latest, coolest new trends in programming and tools.

The 80% folks make up the bulk of the software development industry. They’re not stupid; they’re merely vocational. They went to school, learned just enough Java/C#/C++, then got a job writing internal apps for banks, governments, travel firms, law firms, etc. The world usually never sees their software. They use whatever tools Microsoft hands down to them — usally VS.NET if they’re doing C++, or maybe a GUI IDE like Eclipse or IntelliJ for Java development. They’ve never used Linux, and aren’t very interested in it anyway. Many have never even used version control. If they have, it’s only whatever tool shipped in the Microsoft box (like SourceSafe), or some ancient thing handed down to them. They know exactly enough to get their job done, then go home on the weekend and forget about computers.

Shocking statement #1: Most of the software industry is made up of 80% programmers. Yes, most of the world is small Windows development shops, or small firms hiring internal programmers. Most companies have a few 20% folks, and they’re usually the ones lobbying against pointy-haired bosses to change policies, or upgrade tools, or to use a sane version-control system.

Shocking statement #2: Most alpha-geeks forget about shocking statement #1. People who work on open source software, participate in passionate cryptography arguments on Slashdot, and download the latest GIT releases are extremely likely to lose sight of the fact that “the 80%” exists at all. They get all excited about the latest Linux distro or AJAX toolkit or distributed SCM system, spend all weekend on it, blog about it… and then are confounded about why they can’t get their office to start using it.

I will be the first to admit that I completely lost sight of the 80% as well. When I was first hired by Collabnet to “design a replacement for CVS” back in 2000, my two collaborators and I were really excited. All the 20% folks were using CVS, especially for open source projects. We viewed this as an opportunity to win the hearts and minds of the open source world, and to especially attract the attention of all those alpha-geeks. But things turned out differently. When we finally released Subversion 1.0 in early 2004, guess what happened? Did we have flocks of 20% people converting open source projects to Subversion? No, actually, just a few small projects did that. Instead, we were overwhelmed with dozens of small companies tossing out Microsoft SourceSafe, and hundreds of 80% people flocking to our user lists for tech support.

Today, Subversion has now gone from “cool subversive product” to “the default safe choice” for both 80% and 20% audiences. The 80% companies who were once using crappy version control (or no version control at all) are now blogging to one another — web developers giving “hot tips” to each other about using version control (and Subversion in particular) to manage their web sites at their small web-development shops. What was once new and hot to 20% people has finally trickled down to everyday-tool status among the 80%.

The great irony here (as Karl Fogel points out in one of his recent OSCON slides) is that Subversion was originally intended to subvert the open source world. It’s done that to a reasonable degree, but it’s proven far more subversive in the corporate world!

Enter Distributed Version Control

In 2007, Distributed Version Control Systems (DVCS) are all the range among the alpha-geeks. They’re thrilled with tools like git, mercurial, bazaar-ng, darcs, monotone… and they view Subversion as a dinosaur. Bleeding-edge open source projects are switching to DVCS. Many of these early adopters come off as either incredibly pretentious and self-righteous (like Linus Torvalds!), or are just obnoxious fanboys who love DVCS because it’s new and shiny.

And what’s not to love about DVCS? It is really cool. It liberates users, empowers them to work in disconnected situations, makes branching and merging into trivial operations.

Shocking statement #3: No matter how cool DVCS is, anyone who tells you that DVCS is perfect for everyone is completely out of touch with reality.

Why? Because (1) DVCS has tradeoffs that are not appropriate for all teams, and (2) DVCS completely blows over the head of the 80%.

Let’s talk about tradeoffs first. While DVCS dramatically lowers the bar for participation in a project (just clone the repository and start making local commits!), it also encourages anti-social behavior. I already wrote a long essay about this (see The Risks of Distributed Version Control). In a nutshell: with a centralized system, people are forced to collaborate and review each other’s work; in a decentralized system, the default behavior is for each developer to privately fork the project. They have to put in some extra effort to share code and organize themselves into some sort of collaborative structure. Yes, I’m aware that a DVCS is able to emulate a centralized system; but defaults matter. The default action is to fork, not to collaborate! This encourages people to crawl into caves and write huge new features, then “dump” these code-bombs on their peers, at which point the code is unreviewable. Yes, best practices are possible with DVCS, but they’re not encouraged. It makes me nervous about the future of open source development. (Maybe the great liberation is worth it; time will tell.)

Second, how about all those 80% folks working in small Windows development shops? How would we go about deploying DVCS to them?

• Most DVCS systems don’t run on Windows at all.
• Most DVCS have no shell or GUI tool integrations; they’re command-line only.
• Most 80% coders find TortoiseSVN full of new, challenging concepts like “update” and “commit”. They often struggle to use version control at all; are you now going to teach them the difference between “pull” and “update”, between “commit” and “push”? Look me in the eyes and say that with a straight face.
• Corporations are inherently centralized entities. Not only is their power-structure centralized, but their shared resources are centralized as well.
  • Managers don’t want 20 different private forks of a codebase; they want one codebase that they can monitor all activity on.
  • Cloning a repository is bad for corporate security. Most corporations have an absolute need for access control on their code; sensitive intellectual property in specific parts of the repository is only readable/writeable by certain teams. No DVCS is able to provide fine-grained access control; the entire code history is sitting on local disk.
  • Cloning is often unscalable for corporations. Many companies have huge codebases — repositories which are dozens or even hundreds of gigabytes in size. When a new developer starts out, it’s simply a waste of time (and disk space) to clone a repository that big.

Again, I repeat the irony: Subversion was designed for open source geeks, but the reality is that it’s become much more of a “home run”for corporate development. Subversion is centralized. Subversion runs on Windows, both client and server. Subversion has fine-grained access control. It has an absolutely killer GUI (TortoiseSVN) that makes version control accessible to people who barely know what it is. It integrates with all the GUI IDEs like VS.NET and Eclipse. In short, it’s an absolute perfect fit for the 80%, and it’s why Collabnet is doing so well in supporting this audience.

DVCS and Subversion’s Future

Most Subversion developers are well aware of the cool new ground being broken by DVCS, and there’s already a lot of discussion out there to “evolve” Subversion 2.0 in those directions. However, as Karl Fogel pointed out in a long email, the challenge before us is to keep Subversion simple, while still co-opting many of the features of DVCS. We will not forget about the 80%!

Subversion 1.5 is getting very close to a release candidate, and this fixes the long-standing DVCS criticism that “Subversion merging is awful”. Branching is still a constant-time operation, but you can now repeatedly merge one branch to another without searching history for the exact arguments you need. Subversion automatically keeps track of which changes you’ve merged already, and which still need merging. We even allow cherry-picking of changes. We’ve also got nice interactive conflict resolution now, so you can plug in your favorite Mercurial
merging tool and away you go. A portable patch format is also coming soon.

For Subversion 2.0, a few of us are imagining a centralized system, but with certain decentralized features. We’d like to allow working copies to store “offline commits” and manage “local branches”, which can then be pushed to the central repository when you’re online again. Our prime directive is to keep the UI simple, and avoid the curse of DVCS UI (which often have 40, 50, or even 100 different commands!)

We also plan to centralize our working copy metadata into one place, which will make many client operations much faster. We may also end up stealing Mercurial’s “revlog” repository format as a replacement for the severely I/O bottlenecked FSFS format.

A Last Plea

Allow me to make a plea to all the DVCS fanatics out there: yes, it’s awesome, but please have some perspective! Understand that all tools have tradeoffs and that different teams have different needs. There is no magic bullet for version control. Anyone who argues that DVCS is “the bullet” is either selling you something or utterly forgetting about the 80%. They need to pull their head out of Slashdot and pay attention to the rest of the industry.

Update, 10/18/07: A number of comments indicate that my post should have been clearer in some ways. It was never my intent to say that “Subversion is good enough for everyone” or that “most of the world is too dumb to use DVCS, so don’t use it.” Instead, I’m simply presenting a checklist — a list of obstacles that DVCS needs to overcome in order to be accepted into mainstream corporate software development. I have no doubt that DVCS systems will get there someday, and that will be a great thing. And I’m imploring DVCS evangelists to be aware of these issues, rather than running around thoughtlessly trashing centralized systems. 🙂

My conspirator Fitz and I have given a talk (about 12 times now!) called “How to Protect Your Open Source Project From Poisonous People”. (You can see the slides or watch the video.)

And today, we found an actual poisonous person in the Wild! His posts are textbook-classic; we should bottle him up and show him to Summer of Code students.

The scenario: a guy found some javascript bugs and/or confusing behaviors on a Google product. So he came to the mailing list that provides tech-support for the service and enumerated his complaints. The problem is, his attitude was so condescending and full of bile, he ended up attracting more attention to himself than to the bugs. If you search the net, you can see that this guy hangs out on various javascript lists, and does seem to be a javascript guru of some kind, so his criticisms are probably legit (and valuable!)… but putting on my psychologist hat, it’s pretty clear that his primary goal isn’t to report bugs. His goal is to get people riled up. I guess some people get a kick out of that.

Let’s examine his posts by beginning with what a reasonable, respectful person would say, and then compare that to what this person actually said.

What he meant to say

Hey folks, I found a lot of bugs when using this service. Is this the right place to report them?

What he actually said

This is the worst interface in history. Who designed this miserable thing?

Yes, this is actually how he started the thread. No actual bug reports, just a trolling insult to get people’s attention. Eventually after people posted some “OMG what a jerk” responses, he comes back:

What he meant to say

These bugs have frustrated me and made me not want to use the service. […] Here are some bugs I found: […] I’m suprised these bugs are present, given Google’s reputation for building web applications.

What he actually said

I’m sure I won’t [use the service]. I tried it out once and realized it was a typical piece of Google incompetence. […] For starters: [list of bugs] […] And as usual for Google, JavaScript errors appear randomly. How is it that a company this large that relies on the Web for everything can’t find competent Web developers? It boggles the mind.

Notice that he’s unable to simply enumerate the problems: he attempts to directly insult Google (and its employees) at every opportunity.

When somebody asks him to please “go elsewhere and spare us these useless, trolly comments”, his reply makes it clear that he’s deliberately looking for a fight. Rather than apologize for his attitude, he tells the person to screw off and asserts that he has the right to be a jerk anywhere, anytime:

I you don’t like it, then ignore it and read the next post. I’ll post where I feel like it.

When someone else (nicely) points out that his ranting just makes him look like an idiot, his response indicates that he thinks of his initial post as a sort of “gift” to the rest of us:

I posted one sentence to express my displeasure. Subsequent posts were replies to queries for details as Google’s clueless developers clearly wanted some direction. I’m sure they are glad I clued them in on some of the more odious aspects of this thing.

The guru has expressed his displeasure, and fired a shot across our bow! We better beg him for his thoughts… thank goodness he’s here to help us!

I’m proud the response from Jason, our lead UI developer. He completely ignores the troller’s attitude, and addresses only the criticisms themselves in a technical discussion. This is the proper way to deal with such folk: rather than feed the energy beast, extract bug reports calmly. If it becomes clear that no more bug reports are forthcoming, then simply detach from the conversation. These sorts of people just wither away when they’re no longer able to drum up attention.

All my life I’ve had a secret wish to be a vegetarian. Hm, no, scratch that… but I’ve had the wish for at least the last twelve years, ever since I met my wife. Why? Because after over a decade of discussing and thinking about it, I’m persuaded that (1) it’s generally healthier, (2) it’s better for the planet, (3) it moots the entire issue of trying to define “animal cruelty”.

It all began when I met my wife in college. She told me she was a pescatarian. Well, actually, nobody used that word in 1995, but it’s basically someone who won’t eat any animals other than fish and seafood. I asked her if it was a moral/ethical issue for her (“meat is murder?”), and she calmly explained that she had stopped eating birds and mammals at age 13 because she didn’t want to support animal torture via “factory farms”. (At this point in my blog entry, I don’t have time to go into a long diversion about what factory farms are… but they are terrifying things that torture millions of animals and poison the environment. And they’re the norm for all animal farms; 99% of all meat you consume comes from them. To learn more, read the famous book Diet For a New America.)

I was horrified to learn about factory farming, but was also relieved that my wife didn’t actually believe that eating animals was a sin. I’ve always been of the opinion that there’s nothing inherently unethical about eating animals, considering that millions of years of evolution (and the planet’s entire biosphere) is based on the idea. What matters is the way in which you participate in the food chain. Do you do it respectfully, the way the Native Americans did? Or do you systematically torture creatures and destroy the planet in the process?

Unfortunately, there’s always been one big obstacle between me and vegetarianism: I love meat. I was raised in a meat-centric U.S. culture, doubled by a meat-centric Jewish culture of deli-cured delicacies. My childhood and teen years are peppered with fondly-told stories of how much meat I ate at various events. The taste is in my mouth… I drool when driving by the hot-dog factory, despite my knowing of what disgusting animal parts are poured into a hot dog. When I want to really celebrate, I still have initial thoughts of finding a fancy steak somewhere. In a nutshell, whenever I’ve experimented with vegetarianism, I’ve always crumbled after a week or two, like a nicotine addict reaching back for one more cigarette. It’s embarrassing. (People are sometimes surprised when they find out that we’re raising our kid pescatarian; I think of it as an act of mercy, sparing him from a lifetime of guilty cravings.)

However, there’s finally some hope to my story.

When I moved in with my wife, I learned to live by the rules of her kitchen and house. Her rule was simple: no meat in the house, ever. Throughout our marriage, we’ve kept that rule as our own personalized sort of ‘kosher’ law, and our friends are all aware of it and respect the rule when they come over. Why would I go along with this rule, you might wonder, when I’m not a vegetarian myself? Easy. It’s a nice halfway-point towards my ideal. If I can’t go totally pescatarian like my wife, I can at least dramatically reduce my meat intake. I’ve learned to cook without meat, order meatless dishes for take-in, and just eat meat “outside the house” when the opportunity arises. It simplifies our domestic life as well: no need to worry about my accidentally cooking food with meat, or whether or not I can share my meals with my spouse and son. And because our guests respect the rule at parties and such… well, that’s ten fewer Mu Shu Beef entrees ordered from the local Chinese restaurant, which nudges our economy just a bit more away from meat. Hooray.

Still, though, I’ve become less satisfied with this halfway point. I still eat meat for lunch every weekday (whether it be at a restaurant, or Google feeding me lunch)… and the ethics of eating tortured animals has slowly been gnawing at me over the years.

So my latest experiment is to do what my friend Karl Fogel did: become a pescatarian who only eats untortured meat. This decision has no effect on my domestic life, where I eat 2/3rds of my meals. But it’s made eating food outside the home much more tricky. The fundamental problem is: how do you define animal cruelty? No doubt my vegan friends would describe any sort of meat consumption as inherently cruel, but I’ve already explained how I disagree with that. Well gee, you say, what about all those corporations advertising “free range” animal products? Aren’t I the target for their marketing?

Sure, but while the U.S. Government may have tough standards defining “organic”, they’re really flimsy on defining “free range”. The realm of animal cruelty is a big fuzzy space. If a chicken is allowed out of its tiny cage (just barely big enough to hold it) for 5 minutes a day, to walk around in a 3’x 3′ pen exposed to the sun, is it now a “free range” chicken that lays “free range” eggs? There’s barely any regulation on this stuff.

I was hoping that Whole Foods would be my refuge. I’ve heard all sorts of great rumors about how Whole Foods only sells free-range meat. But when I went to their meat counter last weekend, I found no labels on any products. The phrases “organic”, “local”, and “free range” were nowhere to be seen. When I asked the butchers behind the counter where the various meats came from (“which farm? how was it raised?”), they all hemmed and hawed and scratched their heads. (“I think some of it comes from um… Arizona? No, maybe Colorado?”) The only literature they were able to give me was a generic brochure talking about how the company had created a whole new foundation to increase animal compassion in farming techniques. Hooray and applause for Whole Foods, but… that still dosen’t tell me whether their hamburger is cruelty-free. Even the Information Desk was unable to help me.

When I got home and did some Googling, I found out that Whole Foods only buys meat from farms that meet some pretty hard-core requirements. The farms can’t use hormones or antibiotics on the animals, nor fed animal by-products. The whole system must be heavily documented and re-audited each year. The thing that stood out, though, was the requirement that “the time on a feedlot cannot be more than one-third of the animal’s life.” Hm. Two-thirds of a life grazing freely in a field, but one-third of one’s life in a feedlot? A feedlot is basically thousands of cows crammed into a giant lot of hard-packed dirt. It’s hard to move, there’s nothing to graze on, and the animals are force-fed grain to give their muscle tissue more fatty “marbling”, something they’re not evolved to eat at all. Once again, Whole Food’s definition of “compassion” is just one point along a fuzzy scale, and I’m not sure it’s compassionate enough for me. It makes my quest for humane meat all the more challenging. :-/

My wife imparted words of wisdom to me. She works Heifer International, a huge non-profit organization that teaches sustainable agriculture to impoverished communities around the planet. As you might expect, a number of their employees are either vegetarian, or at least extremely conscious about eating sustainable, ethical, locally-produced food. For most of her co-workers, the general rule of thumb is: if the cook can’t tell you exactly where the animal came from (i.e. “Joe Blow’s farm, 7 miles northwest of here”), they won’t eat it.

With that idea in mind, I will admit to eating meat twice in the last month. First, I bought some buffalo-burgers that came from a specially-advertised farm I could read about. Today, I went and ordered a pork burrito from Chipotle Yes, you read right: the scary burrito fast-food-chain that was once secretly owned by McDonalds. Despite my general revulsion to fast-food, their website actually tells you about their standards for raising pork, and when you walk into the restaurant, there are signs that say exactly which farm it came from!

Net result: I think I’ll be able to survive as a full-time pescatarian, provided I’m able to get a once-every-two-weeks “meat fix” from a source that I’ve ethically pre-screened. It’s going well so far. My drastic reduction in meat consumption also has me feeling lighter and healthier than I have in years. The planet is being slightly less ravaged by my decrease in consumption, and even when I do eat meat, I have no ethical qualms at all.

As a postscript and reward for reading this far: a fascinating book related to local food production is The Ominvore’s Dilemma. It really makes you aware of how we produce food, and why it matters what you eat.