Programmer Insecurity
I’ve got a lot to say today!
I want to chat about something that I’ve never noticed before, but probably should have. There’s always been a stereotype out there of programmers being nerdy, anti-social people (Q: How do you know when an engineer is outgoing? A: He looks at your shoes!). But my revelation of the week is that most programmers seem to be really insecure about their work. I mean: really, really insecure.
My buddy Fitz and I have long preached about best practices in open source software development — how one should be open and transparent with one’s work, accept code reviews, give constructive criticism, and generally communicate as actively as possible with peers. One of the main community “anti-patterns” we’ve talked about is people writing “code bombs”. That is, what do you do when somebody shows up to an open source project with a gigantic new feature that took months to write? Who has the time to review thousands of lines of code? What if there was a bad design decision made early in the process — does it even make sense to point it out? Dropping code-bombs on communities is rarely good for the project: the team is either forced to reject it outright, or accept it and deal with a giant opaque blob that is hard to understand, change, or maintain. It moves the project decidedly in one direction without much discussion or consensus.
And yet over and over, I’m gathering stories that point to the fact that programmers do not want to write code out in the open. Programmers don’t want their peers to see mistakes or failures. They want to work privately, in a cave, then spring “perfect” code on their community, as if no mistakes had ever been made. I don’t think it’s hubris so much as fear of embarrassment. Rather than think of programming as an inherently social activity, most coders seem to treat it as an arena for personal heroics, and will do anything to protect that myth. They’re fine with sharing code, as long as they present themselves as infallible, it seems. Maybe it’s just human nature.
Check out some of these stories I’ve collected:
- Requests at the Google I/O booth: A couple of weeks ago when my team was at the Google I/O conference, we ran a booth demonstrating our Open Source Project Hosting service. Over and over, we kept getting requests like this:
“Can you guys please give subversion on Google Code the ability to hide specific branches?”
“Can you guys make it possible to create open source projects that start out hidden to the world, then get ‘revealed’ when they’re ready?”
Translation: “I don’t want people to see my work-in-progress until it’s perfect.”
- Requests on the Google Code mailing list: Sometimes users need their googlecode.com svn repositories wiped clean. Legitimate reasons include the accidental commit of sensitive data, or the need to load code history in from a different svn repository. But most of the time we get (invalid) requests like this:
“Hi, I want to rewrite all my code from scratch, can you please wipe all the history?”
Translation: “I don’t want people to be able to find my old code, it’s too embarrassing.” Call it vanity, call it insecurity… the bottom line is that coders want prior mistakes or failures to be erased from history.
- Code-reviews taken as personal attacks. Fitz tells a funny anecdote about a friend of his who went from the open source world to a corporate software job. Vastly paraphrased:
During his first week, he started emailing friendly code reviews to each of his coworkers, receiving strange stares in turn. Eventually his boss called him into his office:
“You know, you really need to stop with the negative energy. Your peers say that you’re constantly criticizing everything they do.”
Moral: not only is code review not the norm in corporate environments, most programmers are unable to separate their fragile egos from the code they write. Repeat after me: you are not your code!
- Distributed version control — in a cave. A friend of mine works on several projects that use git or mercurial. He gave me this story recently. Basically, he was working with two groups on a project. One group published changes frequently…
“…and as a result, I was able to review consistently throughout the semester, offering design tweaks and code reviews regularly. And as a result of that, [their work] is now in the mainline, and mostly functional. The other group […] I haven’t heard a peep out of for 5 months. Despite many emails and IRC conversations inviting them to discuss their design and publish changes regularly, there is not a single line of code anywhere that I can see it. […] Last weekend, one of them walked up to me with a bug […] and I finally got to see the code to help them debug. I failed, because there are about 5000 lines of crappy code, and just reading through a single file I pointed out two or three major design flaws and a dozen wonky implementation issues. I had admonished them many times during these 5 months to publish their changes, so that we (the others) could take a look and offer feedback… but each time met with stony silence. I don’t know if they were afraid to publish it, or just don’t care. But either way, given the code I’ve seen, the net result is 5 wasted months.”
Before you scream; yes yes, I know that the potential for cave-hiding and writing code bombs is also possible with a centralized version control system like Subversion. But my friend has an interesting point:
“I think this failure is at least partially due to the fact that [DVCS] makes it so damn easy to wall yourself into a cave. Had we been using svn, I think the barrier to caving would have been too high, and I’d have seen the code.”
In other words, yes, this was fundamentally a social problem. A team was embarrassed to share code. But because they were using distributed version control, it gave them a sense of false security. “See, we’re committing changes to our repository every day… making progress!” If they had been using Subversion, it’s much less likely they would have sat on a 5000 line patch in their working copy for 5 months; they would have had to share the work much earlier. Moral: even though one shouldn’t depend on technical solutions to social problems, default tool behaviors matter a lot. This was my main theme way back when I wrote about the risks of distributed version control.
OK, so what’s the conclusion here? People are scared of sharing their unfinished work, plain and simple. I know this isn’t headline news to most people, but I really think I’ve been in deep denial about this. I’m so used to throwing my creative output up for constant criticism, that I simply expect everyone else to do it as well. I think of it as the norm, and I can’t comprehend why someone wouldn’t want to do that… and yet clearly, the growing popularity of distributed version control shows just how thrilled people are to hide their work from each other. It’s the classic “testimonial” for systems like git (taken from a blog comment):
“Don’t tell me I should cooperate with other people at the beginning and publish my modification as early as possible. I do cooperate with other people but I do want to do some work alone sometimes.”
Hm, okay. Please just don’t work alone for too long!
Sidetracking a wee bit, I think this is why I prefer Mercurial over Git, having done a bit of research and reading on both systems. Git leans much more heavily towards cave-hiding, and I don’t like that. For example, the ‘git rebase’ command is a way of effectively destroying an entire line of history: very powerful, sure, but it’s also a way of erasing your tracks. Rather than being forced to merge your branch into a parent line, just pretend that your branch was always based on the latest parent line! Another example: when it comes to pushing and pulling changesets, Mercurial’s default behavior is to exchange all history with the remote repository, while git’s default behavior is to only push or pull a single branch — presumably one that the user has deemed fit for sharing with the public. In other words, git defaults to all work being private cave-work, and is happy to destroy history. Mercurial shares everything by default, and cannot erase history.
I know this post has been long, but let me stand on my soapbox for a moment.
Be transparent. Share your work constantly. Solicit feedback. Appreciate critiques. Let other people point out your mistakes. You are not your code. Do not be afraid of day-to-day failures — learn from them. (As they say at Google, “don’t run from failure — fail often, fail quickly, and learn.”) Cherish your history, both the successes and mistakes. All of these behaviors are the way to get better at programming. If you don’t follow them, you’re cheating your own personal development.
Phew, I feel better now.
(This page has been translated into Spanish language by Maria Ramos.)
Yow, I’m all for early and often, but it does depend on the project. I remember presenting some early benchmarking results and getting feedback from the junior member of the team about how I should use different gnuplot commands and change the wording of the surrounding verbiage slightly. All this when I’d just thrown something together so that we could discuss the numbers and the implications of the numbers.
By the same token, I’m gonna check in my prototype code, but I don’t appreciate somebody critiquing my proof-of-concept as though it were the finished product. “Gee, why not make that green, and use a different font here?” “Uh, because the basic functionality isn’t there yet?”
I can deal with (and appreciate) “Your code sucks, and here’s why, and here’s how to make it better,” but “You should really do this, this, and this next,” is annoying when “this, this, and this” is the plan, but I’ve not done it yet ’cause I’m still working on it.
Great article, I really enjoyed it.
I used to work in a regular shop with lots of developers who were significantly better than I was, and could get feedback from quickly and easily.
Now I am working on a web application with a designer, but I’m the only programmer. Sometimes I get very nervous about what I’m doing and would love to get regular sanity checks, but I don’t see how it is feasible.
Any suggestions?
Thanks
Everything in this article rings true.
There’s a pervasive elitism at work in the programming community. Add anonymity to the mix, and everyone is suddenly elite.
Unlike most other professions where the apprentice is expected to make mistakes and learn from them, only absolute and immediate perfection is accepted (or at least that’s the assumption and thus is becomes a self fulfilling prophesy)
Online, the people with the most wisdom and experience do not share their knowledge in a friendly manner. They are the arrogant hard-ass black belt grandmaster who beats you with a stick for not getting it right the first time, as opposed to the wise Buddhist philosopher who teaches from the heart because he truly wants you to learn, improve and succeed.
You’re right. It comes down to insecurity. The true master wants his student to eventually surpass him. The insecure master seeks to undermine and inhibit his student for fear of becoming yesterday’s news.
Software developers are human, what an amazing discovery.
There is also the Alpha Male coder, the territorial one that thinks the code is only good when he did some, albeit minimal, changes to it.
You know, Google Code is a steaming pile of crap.
Can’t delete. Can’t select a branch to remove from public view. Confusing, limited interface.
Fail. Big time.
@sharkfish: nice try. 😉
There’s a reason that it’s nice to be able to make changes to a DVCS.
Sure you can look at the crap programmers who build terrible code for months and drop it as a bomb on the repository, but what about features that really can’t be brought in incrementally? What about forks that are made for a good reason–is there no value in keeping the source control connection between the two projects, so that they can trade fixes and features back and forth, even if they remain distinct?
What about the dozens of programmers who want their work to be tracked by source control, but never actually finish their work? Should that code be tracked in the main repository? Should those programmers need to beg commit permission before even starting on a feature? And what about their commits that will inevitably break trunk–that doesn’t exactly help them gain friends and influence people (or the project).
So you want them in their own (public) branch, as part of the main repository? Sounds like a recipe for a cluttered repository. Either way, it makes for a lot of programmers who won’t want to put in the effort to become “blessed” by the maintainers, and so therefore a lot of projects that won’t even get off the ground for lack of the necessary activation energy. I find myself on the outgoing end of the programmer spectrum, and yet haven’t ever asked for commit permission–even on projects that I’ve (successfully) submitted code to.
Sure it’s good to commit early and often, and code review is great. Just KNOWING my code will be reviewed makes me write better code. But if I’m thinking about taking a project in a new direction, I’m going to want to experiment without either a) Abandoning source control for my experiments, or b) Bothering the current project maintainers with my (possibly crazy) ideas.
I guess I have such a project in mind, though I haven’t started it yet. The nature of the project is “Change the underlying GUI implementation of a project from Windows-native to wxWidgets,” so it’s not something I would want to start randomly submitting snippets of to the main code line–or even try to convince them that they need a branch in their public source tree for, if they have no interest in such a port. On the other hand, before I start I’ll try to have a discussion with the current maintainers–and I’ll share any code that I can make work, if they’re interested. Because I’m not afraid of others seeing my code. But I am afraid of cluttering up their source control system, or breaking their current build for no good reason.
Nice post. I feel like such an asshole now! I think I should change my ways.
Let me ask this though, what if you have a unique idea and you want credit for the idea? Should code be shared early?
There are some serious coders out there that could accomplish something in two weeks that would take me a month.
Whats your take in that situation?
While I agree with your point, the guy that started a new job and immediately started mailing his new colleagues criticising their code needs to learn some people skills. What kind of reaction did he expect? You need to earn people’s respect before they will accept your criticism.
Nice article. Thanks for writing it.
About wanting to have some branches hidden in Subversion repository on Google Code:
I think it is not about “I don’t want people to see my work-in-progress until it’s perfect.†but rather “I don’t want people to see my work-in-progress until it’s ready.†Or at least it is IMHO *legitimate* concern.
One would want to have his/her work in progress (WIP) protected by version control system, but with ability to go back and forth to make first draft of series of commits introducing some feature into series of small commits, logical steps in introducing feature, without “oops, there was bug in second commit in series” etc. And I think one solution would to make use of distributed SCM and make changes available to some subgroup, without pushing them to “official” server.
———-
About using centralized SCM like Subversion offering protection from “hiding in cave”:
IMHO it is simply not true. The only thing that using centralized SCM does it making ‘cave-work’ submitted as huge single patch, instead of series of commits (patchbomb). One can always use some private SCM, e.g. SVK / SVN::Mirror / git-svn clone of repository do do “work in a cave”.
Distributed SCM, and especially free DSCM hosting sites offer another solution: you can work on your project “in a cave”, without ‘pushing’ into point-of-distribution official repository (although preferable sending some RFC and WIP patches to git mailing list for review), but with pushing changes into your public repository / your “fork”, of course mentining that in this repository given branches are in flux.
———-
About git-rebase:
First, Mercurial has Transplant extension, and Bazaar grafts plug-in, which are equivalent to git-rebase. Second, usually rebase (or some patch management interface like Mercurial queues, or StGIT or Guilt for Git) has its place either in early stages of branch development, or if you are contributor who sends his/her commits by emailing patches (great for code review, by the way). Third, rebase based workflow, as opposed to merge based workflow, has its advantages and has its disadvantages, like every technique.
———
About Mercurial exchanging all history, and Git exchanging single branch by default:
First, you have outdated information about Git. Currently preferred and “best practice” way of adding new repository to pull history from is to use “git remote add”, which sets up nickname (shortcut) used to pull _all_ [regular] branches. Thanks to globbing refspec fetch picks up new branches (you can use “git remote prune” to remove branches which got removed on remote).
Second, I wonder how many Mercurial repositories use multiple branches per single repository approach. Isn’t it usually singe branch per repository?
———–
Summary:
Learning to share your contributions early and often is as important as releasing (creating releases) early and often (release early, release often).
One reason for being ashamed:
http://www.codinghorror.com/blog/archives/000530.html
Sam,
The point for why I often refuse to reveal in an early stage what I am working on is that in that stage you’re rather vulnerable to discouraging — and people are not mean giving one discourage. Like the short-sighted ‘that’s impossible’, ‘alone, you’ll never make that’ etc.
If I knew a chance how to fix that, I’d gladly appreciate it.
As someone with both commercial and free code (not open source) I have mixed feelings about this article.
For the free and commercial source, I don’t mind getting patches and suggestions concerning the code, but I chose to control design and implementation. That’s just me and how I prefer to work after many years of coding. I take a lot of pride in my work since I do see it as an expression of myself. It also reflects past experience in business, I don’t want to be responsible for other peoples failures, only my own.
On the other hand, especially when you think your ideas might be the next Google, Facebook, Twitter, whatever, you want to play your cards close to your chest. You want to develop something and be first out the door with something novel. For myself, I share with my team, but not the world. I know I’ve written some crap code in the past, sometimes because speed was more important; “publish or perish, refactor after” (though this refrain can also support your point too).
You perspective is “open source, open mind, open attitude”, but there are those who have a different history, different experiences, different character who might be willing to share once they reach their an internal milestone. Writing code is a lot like writing a book, some authors want to present “a fait accompli”, while others will send chapters for review to their publisher.
I guess my point can be best expressed as “open choice”.
Agree in large part but I think the primary motivation for secrecy is the type of negative, geeky, point scoring criticism I sometimes see. Basically it is easier to criticise than to create and too many focus on the former rather than the latter. Making sure your code review is well thought through and positive even when disagreeing is the best way to encourage others to publish work in progress.
Whie I agree with the general thrust of your argument, I find it incredible that you would claim that any technology can cause or exacerbate what is clearly a social problem. Social problems such as this are not created by technology and, likewise, cannot be solved by technology.
To suggest or imply that people should not use Git because “Git leans much more heavily towards cave-hiding” is a dangerous red herring. The issue here is a dysfunctional culture, not a dysfunctional tool. You (and everyone else facing this sort of issue) would do far better to address this directly rather than project the cultural and social failure onto mere tool.
but doesnt google do the same.
they never release a project until its in a working condition?
the entire project is a code bomb in this case 🙂
well, in actually i need some help , i am new in programming , right now i am trying to design a programme over a problem ,
i am trying to make a programme which can find the all possible unique sequense combinations ,
as a example :
there is a block of few characters:
0 1 2 3
———–
0 | A T G C
1 | A C T
2 | G
1*2*2*3 = 12 ,
SO 12 IS THE NO WHICH SHOWS THAT FROM THIS BLOCK WE CAN OBTAIN 12 UNIQUE SEQUENSE COMBINATIONS.
SO THE OUT PUT SHOULD BE :
1: ATGC
2: ATGT
3: ATGG
4: ATCC
5: ATCT
6: ATCG
7: AAGC
8: AAGT
9: AAGG
10: AACC
11: AACT
12: AACG
i am clueless so please if any body can suggest, give any logic over this pls.