Notes about distros, QA, etc.

Yesterday I flamed Ubuntu, I think with cause, for breaking X. Followups:
Points out of the comments in that thread yesterday:

  • Fedora now believes that they are going to be able to support (apparently already have supported) distro->distro upgrades, like Debian has done for years and Ubuntu has done since day 1. This is very big for Fedora. Along with a growing selection of packages in Extras, two of Debian/Ubuntu’s biggest selling points to the technical community are under siege. Yay for competition :)
  • One of the Fedora dudes clarified my understanding of the FC5/X7.1 situation- it was not nearly as broken as I’d thought. That said, part of what Fedora should be trying to do is build a culture around QA and quality- which means clear messaging about these sorts of things. So I’m glad that they were doing the right things for roughly the right reasons, but they need to get better about communicating those so that their culture grows up with them.
  • rpath points out that the obvious solution to problems like the one I had yesterday is to be able to rollback packages, and that conary (rpath’s system management tool) can do that. I continue to think that rpath is doing really interesting stuff; this would be one good demo of why. (Yes, I know red carpet and other tools have done rollback for a while, but conary’s implementation, from what I can grok of it, is nice and well-integrated.)
  • Lucas did a really interesting (and totally unscientific) ‘survey’ of Ubuntu and Debian users by way of CTCP in IRC, and discovered that something like 4% of Ubuntu users were using edgy, while 76% of Debian users were using unstable. My hunch is that this says more about Ubuntu’s incredible success in getting newbies into #ubuntu than it does anything else, but the core question (‘what percentage of our users are using and testing our development branch? what steps are we taking to raise that percentage?’) is a really interesting one which every free software project should ask itself. (NB that GNOME is failing here, and has been since Ximian stopped funding packaging of unstable builds years ago. Ubuntu’s unstable builds have been a huge pickup in that respect.)
  • Shockingly, no real response from Ubuntu that I can see anywhere (planet, bugsquad list, forums), other than the fast fix. Remember that much of this is about expectations- I expect a lot from Ubuntu, so when they fuck up, (1) I get very very pissed, because I trusted them and (2) I expect openness about why it happened this time and how they are going to prevent it from happening next time, so that I can again trust them.

Some things I personally should have explained better:

  • I am not switching distros. All things considered, at this time Ubuntu still offers the best mix of maintainability and support for my needs, especially now that I know not to actually trust their support packages. Silly and naive of me to have done so earlier, though, and led to (frankly) great anger.
  • QA for this sort of thing (big, big bug in package everyone uses) is not really very hard to do. Ubuntu has been leading the way for quite some time in open source distro QA implementation, pushing packages early and quickly in their unstable branch so that their stable release is both well tested and fairly up-to-date- exactly what every other distro should be doing, and some do to various degrees. But Ubuntu have not quite pushed hard on the last mile for stable- getting users to test proposed updates before they ship. I only discovered yesterday that they have a ‘proposed updates’ channel for apt. Quite simply, if I (who am completely obsessive about open source QA) don’t know about your proposed updates channel, you haven’t pimped it enough. Every distro should have a proposed updates channel, like Ubuntu does, and pimp it heavily to their skilled users, which AFAICT Ubuntu does not.  Skilled users who are not running unstable should, in response, consider it nearly a moral obligation to use the proposed updates channel on any non-mission-critical boxes. That combo, used effectively, should have caught this before it went out. If Ubuntu is post-morteming this (which they should be) that would be the big question to ask- why did the community not catch this for us?
  • It is worth remembering that every significant community linux distro has a community of thousands who will gladly test anything you throw at them, so distros must actively encourage and take advantage of that. Any distro which doesn’t (and many don’t) is throwing away free time and free money. (Relatedly, I firmly believe that as a result of the opportunity for free QA, most open distros should in practice be more stable than their ‘enterprise’ alternatives, which have smaller user bases who would rather pay for someone else to do the work. That in practice enterprise distros tend to be more stable points to inefficiencies in how open distro QA is done, IMHO, not just the obvious points about business models.)