tinderbox requirements

Last summer I spent some time tinderboxing, looking at the various tinderbox options, and thinking about tinderboxing in general. It was good, and I think good for the project. For various reasons, though, it died when I went back to work.

This week, there has been a great thread on gnome-love about tinderboxing.. Lots of people interested, and already a tinderbox which uses mozilla tinderbox and one which uses a distributed hack of jhbuild. And someone is working on configuring one based on buildbot. This is all totally awesome.

I contributed to the discussion a bit by weighing in on the pros/cons of some of the existing systems, but what I should have done was write out the basic requirements, so that other people can learn from the mistakes I made last summer. Without further ado:

Hard Requirements For a New Tinderbox Tool/Setup

  • Supports Big List O’Modules: Support building a long chain of modules- jhbuild is well north of 100 modules now for a default gnome build. This may seem obvious, but it is relevant- at least some build tools are basically optimized for one module/one checkout scenarios, and as a result, some off-the-shelf stuff doesn’t work for us.
  • Many build types/sources: Must support building modules from a variety of source code repositories and (ideally) a variety of configuration tools. We have cvs, svn, and git in jhbuild right now, plus auto* and some custom stuff, and presumably bzr at some point soon, so if the tool don’t handle all of these, it either needs to be easily extensible or dumped.
  • Reporting: Reports results over the web in a way that is reasonably easy for non-experts to interpret, with links to full logs for experts. This reporting must be per-module, not just for the whole run (otherwise digging through the 20+M of logs for each build run would suck.) ‘jhbuild tinderbox”s output is a pretty reasonable start at this.
  • Tests: Supports ‘make test’, and at least hypothetically supports LDTP/Dogtail. [Both of these mean it must be possible to run the test under xvfb or similar, by the way.] The catch: the tests must be run and reported on, but can’t block the build of things later in the dependency tree- if we have to wait for all tests to succeed, we’ll never have a green run of the tinderbox.
  • Documented: How the tool was made to work with GNOME must be written down so that others can maintain it and use it. We’ve had too many tools like this that were made to run once by a mad genius who then left the project, or lost interest, and so when it broke we just had to discontinue the service.

Bonus These bits aren’t necessary, but it is really, really nice if they work.

  • Dependencies: a really nice feature of jhbuild is that a failure in, say, glib restarts you from scratch, but a failure in, say, gok, allows other modules to be built/tested. This isn’t a hard requirement but it definitely makes things better.
  • Notification: would be nice if, when failures occur, lists/IRC/etc. get notified of the failure, instead of people having to poll a web page.
  • Minimal Duplication of Build Information: Does not add Yet Another location that must be updated when modules are added/removed- i.e., must draw build information in some reasonably low-cost way from the canonical jhbuild moduleset. If this requirement isn’t met, better have someone damn dedicated to keeping it up to date.
  • Distributed: Should be able to have a central server that collects information from build tools installedon multiple distros/OSes/platforms/etc. This is in wishlist because we are so bad at keeping things building just on x86 linux that getting it running on just that (no server or whatever) would be a huge win.
  • Easy to set up/maintain: no one will take advantage of the distributed feature if it sucks to get running, or if lots of manual work is needed to keep things running once the initial build works.
  • Builds Ekiga: This is the one thing I never did :) Was just too hard to get the dependencies right. If you’ve got this working, then lots of other things are probably working very well :)
  • Active Development Community: The best software is software with a community that gives you ‘free’ new features. This should be obvious but is easy to forget :)

Lots of other wishlist stuff is on the wiki. [Update: I put the requirements in the wiki as well, under Tinderbox Requirements Doc, so others can add/edit.]