a Mercurial “super client”

This entry was posted by on Tuesday, 14 October, 2008 at

One of the cool trends I’ve seen is the use of distributed version control systems as “super clients” against Subversion. You suck down the entire history of a Subversion repository into a local, private repository, do all of your commits locally, make branches, experiment all you want, then “push” back up to Subversion again. On the internet, nobody knows you’ve been using DVCS (or that you’re a dog.) What’s particularly cool about these bridging tools is that they allow users to try out DVCS before deciding to officially convert a whole project over. Or, if a project happens to be using Subversion but you still want most of the power of a DVCS for local work, it’s a perfect solution.

For all the blabbing I’ve done about distributed version control systems, I’m still a big fan of Mercurial. Of all the DVCSes, I think it’s the easiest to learn for svn users. It has a small, tight set of commands, and the community which runs the project is polite and sane.

In any case, there have been a collection of Mercurial-Subversion bridges available for the last couple of years, but they’ve all been deficient in various ways: either not capturing svn history entirely, or being unable to push back to svn correctly (or only very awkwardly). So I’ve pretty much stayed away. But today I want to plug a new bridge written by a friend of mine (Augie Fackler) who finally did it Right: he wrote a bridge called hgsubversion which (1) uses the actual Subversion API to pull history down (which is faster, more accurate, and long-term sustainable), and (2) actually knows how to push changes back to Subversion correctly. I want the world to be aware of this tool, because I think it’s the first Mercurial-Subversion bridge which deserves to be promoted into the popular ranks with tools like git-svn.

The tool is still young and not generally installable by the public (i.e. you’re not going to find any magic .rpm, .dpkg, .zip or .dmg for it yet)… but here are my cliff notes if you want to start playing with it.

Requirements

  • The latest (unreleased) Mercurial
  • Local Subversion libraries, at least 1.5, with swig-python bindings built
  • A Subversion server that is 1.4 or later

To get the latest Mercurial:

$ hg clone http://selenic.com/repo/hg hg-latest
$ cd hg-latest
$ make
$ sudo make install

To get the latest Subversion python bindings:

$ # if you don't have a binary package for svn-1.5-python-bindings already,
$ # this is a summary of subversion/bindings/swig/INSTALL instructions:
$ svn checkout http://svn.collab.net/repos/svn/tags/1.5.3 svn
$ cd svn
$ ./autogen.sh && ./configure
$ make
$ sudo make install
$ make swig-py # make sure you have swig 1.3 installed already
$ make check-swig-py
$ sudo make install-swig-py

To get hgsubversion:

$ hg clone http://bitbucket.org/durin42/hgsubversion/ ~/hgsubversion
$ cat >> ~/.hgrc
[extensions]
rebase=
svn=/home/user/hgsubversion
^D

To make sure you’re ready to go, do a final sanity check:


$ python -c "import svn.core; print svn.core.SVN_VER_MINOR"
5
$ # if you get something less than 5, you may have conflicting
$ # versions installed, and may need to set PYTHONPATH

Now you can clone your favorite svn repository, and use it locally:


$ hg svnclone http://svn.example.com/repos hg-repos
converting r1 by joe
A trunk/
committed as 24dfb7b51d606a921333e2b8f19a9a6aa5661a69 on branch default
[...]
converting r100 by jane
M trunk/goo
M trunk/moo
committed as 54dfb7b51d6d6a931333e2b8f19a9a6005661a62 on branch default
$

The tool currently assumes a ‘standard’ svn layout of /trunk, /branches, /tags, and then tries to pull them into sane mercurial equivalents. After you’ve made a bunch of local commits, you can push the changes back to subversion:


$ # First merge the latest public svn changes into your repository:
$ hg svn pull
converting r101 by pinky
A trunk/awesomefile
committed as e85afd44dc83d5df2599157096a95b0868de6955 on branch default
$ hg up -C
3 files updated, 0 files merged, 1 files removed, 0 files unresolved
$ # We now have two hg HEADs, because the public svn changes are
$ # considered a different line of development from my own.
$ # For now, rebasing is preferred to merging:
$ hg up -C 9985f017b3ab
3 files updated, 0 files merged, 1 files removed, 0 files unresolved
$ hg svn rebase
saving bundle to /home/user/project/.hg/strip-backup/c4b9dfce6b09-temp
adding branch
adding changesets
adding manifests
adding file changes
added 4 changesets with 4 changes to 4 files
rebase completed
$ # At the moment, each changeset is pushed separately;
$ # changeset flattening not yet implemented
$ hg svn push

30 Comments to a Mercurial “super client”

  1. Karl Fogel says:

    October 14th, 2008 at 10:35 am

    Great post!

    Hmm, the fact that hgsubversion has to make assumptions about the locations of trunk / tags / branches — assumptions that may not be matched by every repository out there — just makes me think even more that Subversion is going to have to make tags and branches first-class objects somehow, rather than just conventions.

  2. Jeremy Whitlock says:

    October 14th, 2008 at 4:48 pm

    Great post. Gives me something new to try. I also like Karl’s idea of making branches/tags first class objects in Subversion. I guess we’ll see what happens.

  3. Bryan O'Sullivan says:

    October 15th, 2008 at 12:28 am

    Wow, great writeup, Ben. Thanks for plugging this and giving some very useful details on how to actually use it.

  4. Ori Peleg says:

    October 15th, 2008 at 3:25 am

    Very nice! I think convenient svn integration is a killer feature for “alternative” SCMs.
    I have to point out bzr-svn, the official Bazaar-Subversion integration (included by default in the Windows installer too) – it works great and is in heavy use.

  5. Dirkjan Ochtman says:

    October 15th, 2008 at 4:12 am

    Karl: right you are. The branches/tags-as-paths architecture SVN uses is great for people who are new to VCS, but it’s hell for converting to any other VCS. Committing to tags, partial branching, people have all sorts of weird crap going on. It’s flexible, but also makes for a kind of lock-in.

  6. Jason Allen says:

    October 15th, 2008 at 7:08 am

    @Karl – it’d be awesome if subversion did treat tags & branches as first class objects. Could it realistically be done?

  7. Augie Fackler says:

    October 15th, 2008 at 7:43 am

    Thanks for the plug Ben! It’s great to have some notes on use from someone else – I’m sure there are bad assumptions in the code that only work because of my particular workflow.

  8. David Allouche says:

    October 15th, 2008 at 12:05 pm

    Obligatory Bazaar plug.

    http://bazaar-vcs.org/BzrForeignBranches/Subversion

    Bazaar had bzr-svn for a long time. I believe it was the first to offer a transparent, read-write bridge between a DVCS and Subversion.

    The current development version is making some great progress towards relaxing requirements on the the consistent use of a specific branching scheme.

    Why talking about Mercurial and git, and not at all about Bazaar?

  9. Ben Collins-Sussman says:

    October 15th, 2008 at 2:05 pm

    I talk about mercurial because I use it, like it, and want others to try it; I mentioned git only because it’s got the most mindshare of any DVCS at the moment. I left out bzr only because I haven’t tried it yet and don’t know much about it. I also failed to mention darcs, arch, monotone, codeville, and svk. :-)

  10. David Allouche says:

    October 15th, 2008 at 2:47 pm

    I take issue with how your comment could be interpreted as putting bzr in the same league as darcs, arch, monotone cdv and svk.

    Git, hg and bzr have much more in common with one another than with the other ones. In terms of both technology and adoption.

  11. n8 says:

    October 17th, 2008 at 3:30 pm

    Hey, this is great! Everything is working, but I’m not sure exactly what’s going on in the last block of commands. Are the hg svn pull, and then the two updates, necessary? Or can I just skip to the rebase?

  12. Travis Clne says:

    October 17th, 2008 at 8:26 pm

    Looks good. Might have kept me around longer but I just recently bailed from svn+hg recently. Git+svn lasted a little while but we moved to git completely for the team. Git’s really not much more complex. Local branching and rebasing are miserable to complex in hg. Mq is nice in its simplicity but imo git wins out.

  13. Andre Stechert says:

    October 19th, 2008 at 3:16 am

    Many of us use mercurial because that’s what the projects we depend on use.

  14. Arne Babenhauserheide says:

    October 27th, 2008 at 8:55 am

    Thank you for your great post!

    I’m very interested in completely switching to Mercurial for working on svn projects, and hgsubversion sounds just about right.

    Besides: I found your post via the Mercurial wiki.

  15. Bastian says:

    November 7th, 2008 at 1:50 pm

    FYI: This extension can now be used from within MercurialEclipse (r884 and higher).

  16. Manuel says:

    December 3rd, 2008 at 6:45 pm

    Now (Dec. 2008), hgsubversion only needs the latest stable release of mercurial (not older than 1.1). No need to check out from hg-latest or crew.

    The only problem that remains: the swig-svn bindings (bridging the gap between the svn C-libraries and Python) are not part of the prebuild mercurial binary installations (at least thats the case for win32 binaries). Thus hgsubversion will not work with them for now. You have to build mercurial your self, which turns out to be a bit painfull (again for win32).

    Buildinstructions for Mercurial on Windows:
    http://www.selenic.com/mercurial/wiki/index.cgi/WindowsInstall

    svn-swig-python bindungs:
    http://subversion.tigris.org/servlets/ProjectDocumentList?folderID=91&expandFolder=91&folderID=260

    Thanks for your great article

  17. Arne Babenhauserheide says:

    December 11th, 2008 at 4:37 am

    I just tested it with the sourceforge.net svn repository for Phex, and for some reason hgsubversion only imports the code since Mar 13 16:55:25 2006.

    -> repo: https://phex.svn.sourceforge.net/svnroot/phex/phex

    I just wanted to let you know about that.

    I can use it anyway for coding, but I can’t use it as a full-fledged mirror of the repository.

    Best wishes,
    Arne

  18. Arne Babenhauserheide says:

    December 29th, 2008 at 4:26 am

    hgsubversion now works with the current release of Mercurial which includes the rebase extension (hg >=1.1).

  19. Lubo says:

    April 20th, 2009 at 7:49 am

    Great!
    I hope the working won’t stop on this plugin, this is exactly what I miss till now :-)

    Again thanks,
    L.

  20. Pablo says:

    April 29th, 2009 at 3:13 pm

    Hey, I’m getting an output I can’t figure out:

    $ hg clone svn://10.20.109.140/development/components/webflow
    Assuming destination webflow-hg
    abort: could not import module encoding!

    Any clues?

  21. Pablo says:

    April 29th, 2009 at 3:17 pm

  22. Pablo says:

    April 29th, 2009 at 3:21 pm

    Still don’t know how to work around this… is mercurial-1.3 strictly required for using hgsubversion?

  23. Craig McQueen says:

    May 5th, 2009 at 7:54 pm

    I too am stuck on the error message:
    abort: could not import module encoding!

    I’m trying it with Mercurial 1.1.2 on Ubuntu 9.04 (Jaunty). I have no idea how to progress from here.

  24. Ben Collins-Sussman says:

    May 6th, 2009 at 9:30 am

    Guys, I’m not really a good source of tech support here. You should direct questions to the hgsubversion mailing list: http://groups.google.com/group/hgsubversion

  25. Augie Fackler says:

    May 6th, 2009 at 5:37 pm

    I’m posting this here mostly in hopes of reinforcing Ben’s comment – *please* go to the Google Group with questions/problems. hgsubversion has been growing fairly fast, and the state of things has changed greatly since this blog post. We’re a friendly bunch, so no real need to be wary.

  26. panzi says:

    July 9th, 2009 at 5:19 pm

    Is it possible to exclude a specific svn directory? Because blender has a gigantic /lib directory with *binary* files (DLLs etc.): https://svn.blender.org/svnroot/bf-blender/trunk/

  27. Augie Fackler says:

    July 14th, 2009 at 8:19 am

    Panzi, you’re asking your question on a blog post about hgsubversion – I’d recommend going to the google group (http://groups.google.com/group/hgsubversion) and asking there. We’d be more than happy to help you out.

  28. Juliano says:

    October 7th, 2009 at 7:20 pm

    The pre text is really badly formatted… linebreaks were lost.

  29. Ben Collins-Sussman says:

    October 7th, 2009 at 7:28 pm

    Hm, the new theme messes up “pre” tags, but “code” tag seems to fix it. All set now. Thanks, Juliano.

  30. hgsubversion on windows « monkey island says:

    November 4th, 2009 at 1:40 pm

    [...] heavily borrow from excellent entry by Ben Collins-Sussman and from exceptionally diligent post by Dave Cameron. I want to focus on the smallest possible [...]