Yesterday I flamed Ubuntu, I think with cause, for breaking X. Followups:
Points out of the comments in that thread yesterday:
- Fedora now believes that they are going to be able to support (apparently already have supported) distro->distro upgrades, like Debian has done for years and Ubuntu has done since day 1. This is very big for Fedora. Along with a growing selection of packages in Extras, two of Debian/Ubuntu’s biggest selling points to the technical community are under siege. Yay for competition :)
- One of the Fedora dudes clarified my understanding of the FC5/X7.1 situation- it was not nearly as broken as I’d thought. That said, part of what Fedora should be trying to do is build a culture around QA and quality- which means clear messaging about these sorts of things. So I’m glad that they were doing the right things for roughly the right reasons, but they need to get better about communicating those so that their culture grows up with them.
- rpath points out that the obvious solution to problems like the one I had yesterday is to be able to rollback packages, and that conary (rpath’s system management tool) can do that. I continue to think that rpath is doing really interesting stuff; this would be one good demo of why. (Yes, I know red carpet and other tools have done rollback for a while, but conary’s implementation, from what I can grok of it, is nice and well-integrated.)
- Lucas did a really interesting (and totally unscientific) ‘survey’ of Ubuntu and Debian users by way of CTCP in IRC, and discovered that something like 4% of Ubuntu users were using edgy, while 76% of Debian users were using unstable. My hunch is that this says more about Ubuntu’s incredible success in getting newbies into #ubuntu than it does anything else, but the core question (‘what percentage of our users are using and testing our development branch? what steps are we taking to raise that percentage?’) is a really interesting one which every free software project should ask itself. (NB that GNOME is failing here, and has been since Ximian stopped funding packaging of unstable builds years ago. Ubuntu’s unstable builds have been a huge pickup in that respect.)
- Shockingly, no real response from Ubuntu that I can see anywhere (planet, bugsquad list, forums), other than the fast fix. Remember that much of this is about expectations- I expect a lot from Ubuntu, so when they fuck up, (1) I get very very pissed, because I trusted them and (2) I expect openness about why it happened this time and how they are going to prevent it from happening next time, so that I can again trust them.
Some things I personally should have explained better:
- I am not switching distros. All things considered, at this time Ubuntu still offers the best mix of maintainability and support for my needs, especially now that I know not to actually trust their support packages. Silly and naive of me to have done so earlier, though, and led to (frankly) great anger.
- QA for this sort of thing (big, big bug in package everyone uses) is not really very hard to do. Ubuntu has been leading the way for quite some time in open source distro QA implementation, pushing packages early and quickly in their unstable branch so that their stable release is both well tested and fairly up-to-date- exactly what every other distro should be doing, and some do to various degrees. But Ubuntu have not quite pushed hard on the last mile for stable- getting users to test proposed updates before they ship. I only discovered yesterday that they have a ‘proposed updates’ channel for apt. Quite simply, if I (who am completely obsessive about open source QA) don’t know about your proposed updates channel, you haven’t pimped it enough. Every distro should have a proposed updates channel, like Ubuntu does, and pimp it heavily to their skilled users, which AFAICT Ubuntu does not. Skilled users who are not running unstable should, in response, consider it nearly a moral obligation to use the proposed updates channel on any non-mission-critical boxes. That combo, used effectively, should have caught this before it went out. If Ubuntu is post-morteming this (which they should be) that would be the big question to ask- why did the community not catch this for us?
- It is worth remembering that every significant community linux distro has a community of thousands who will gladly test anything you throw at them, so distros must actively encourage and take advantage of that. Any distro which doesn’t (and many don’t) is throwing away free time and free money. (Relatedly, I firmly believe that as a result of the opportunity for free QA, most open distros should in practice be more stable than their ‘enterprise’ alternatives, which have smaller user bases who would rather pay for someone else to do the work. That in practice enterprise distros tend to be more stable points to inefficiencies in how open distro QA is done, IMHO, not just the obvious points about business models.)
starting off of the recent much publicised Xorg update breakage (http://osnews.com/comment.php?news_id=15587) in Ubuntu http://tieguy.org/blog/2006/08/22/still-learning-what-long-term-support-means/) that ended up being a very interesting discussion (http://tieguy.org/blog/2006/08/23/notes-about-distros-qa-etc/) touching various aspects of how you focus on QA (http://tieguy.org/blog/2006/08/24/more-on-qa-ubuntu-trust-etc/). He points out that * Fedora is now support upgrades between distributions – I believe he is talking about live upgrades using yum
I’ve bumped into a similar QA problem with Fedora Core some time ago. Some (but not all) of the relevant X packages were upgraded and available for download and install. I did so. When I rebooted, I was left with a system that was constantly trying to do something with X but couldn’t. I had no GUI and whatever this process was trying to do, I was being interrupted while typing on one of the virtual terminals.
I was able to gracefully shutdown and wait for a few hours while the package repo got whatever packages I needed. Later, I upgraded again and picked up the remaining packages. They installed, I had a bootable system with a GUI, and things returned to as they had been before.
So, I’d suggest that if an update requires more than one package, make sure all of its other packages are available before allowing a user to update any of the set.
Have you considered writing an article for LWN on this topic?
Enterprise distros are normally stabler. Definitely, RHEL is a lot stabler than Fedora.
Fedora has more users but it follows upstream very closely so new bugs are often introduced. RHEL basically only cares about stability so it’s obviously going to be more stable than Fedora. RHEL doesn’t really add features and it removes a bunch of features that are considered unstable.
I used to run Debian unstable on my desktop around six years ago. At the time I wasn’t a complete newbie to Linux but still very fresh. However when it came to installing Debian for the first time I chose Unstable over Testing or Stable. In terms of running recent software Stable was years out of date so I didn’t want to use that, and I had heard reports that Testing tended to be less stable because of how packages were put into Testing (x days without an update then gets put into testing). In the three years that I used unstable I think I only ran into 3 decently major problems, all which were solved within a day or two’s time.
Several years later I’m using Ubuntu on my laptop (previously it ran FC). Some of the reasons I chose Ubuntu over something like Debian is because Ubuntu has much better hardware detection and setup than Debian and I don’t want to spend a day after installing just trying to get X to work properly. Furthermore because of the 6 month release cycle, things are still very up to date on Dapper, I don’t need to use Edgy. I think a more useful comparison would be between those using Debian Experimental to those using Ubuntu Edgy.
> something like 4% of Ubuntu users were using edgy, while 76% of Debian users were using unstable.
I don’t think these percentages mean what it’s implied that they mean. Edgy has only just come out of being in huge flux, with packages merging in from unstable and the distribution as a whole being broken (and expectedly so) for significant periods of time. In contrast, there’s a more reasonable expectation that Debian/unstable will behave more often than not, and it’s been in existence and in working order for a far longer period.
There’s also a matter of motivation: until recently edgy was little more than a clone of dapper, and dapper is still pretty modern; there’s approximately no “I’d use dapper, but I need version $foo of $bar, so I have to use edgy” use case right now, but there are plenty of equivalent cases in Debian. If the impression people have is that edgy is like dapper but broken pretty often and with no significant new features, it’s unsurprising that they haven’t upgraded yet.
Debian unstable is a must for having a decent workstation, with updated software.
Ubuntu on the other hand has very recent software versions on the stable branch. The unstable branch of Ubuntu is like “experimental” on Debian, and no so many people runs that branch.
I used to use debian experimental on my workstation and unstable on my laptop (I depend more on the laptop, so I needed more secure release).
Right now I use Dapper on both, but with the hoary->breezy and breeze->Dapper transitions I updated the workstation to the experimental branch of Ubuntu when 2-3 months left for release.
In all this time I have had much less problems using the experimental branch of ubuntu (3 month away of release) than using debian unstable before.
“Shockingly, no real response from Ubuntu that I can see anywhere” … did you check http://www.ubuntu.com/ ? it has info since yesterday already …
Luis,
since yesterday evening, UTC time, the xorg-server upgrade issue is displayed quite proeminently on http://www.ubuntu.com website (its the first item displayed on it, actually).
We have worked full time on the last two days on this issue, to make sure:
1. Affected users are able to completely recover from it
2. Possibly affected users can check (and update) their computers if they were affected
3. Community, media, and everyone else interested on the subject is made aware of the problem
An update was quickly released, mirrors were redirected to the master archive, to make sure no new ocurrences of this happen due to outdated mirrors, instructions for fixing and checking if you have the bug were written, published on the Ubuntu website, media channels were contacted, etc.
and most important:
4. Make sure such an unfortunate event *NEVER* happens in the future
The Ubuntu development team has had a few brainstorming meetings since, to analize the situation, and take it as a (painful) chance for us to learn with our errors, and more important, measures to make sure something like this never happens again.
Official statements (unlike this one) will be made by Canonical, today, and in the next few days.
> That combo, used effectively, should have caught this before it went out.
Note that the X server update did not go through the public dapper-proposed channel, but was pushed into dapper directly from a non-public dapper upload queue, at least AFAIK. I think this policy (besides a lot of other stuff pertaining to stable updates) is being revisited now.
[…] James: I’ve not considered an LWN article on distro QA because for quite some time (really since around when I left Novell) I’ve been pondering writing the definitive serious white paper on the subject. As dobey is about to find out, writing anything of that length is hard :) We’ll see if these blog posts coalesce my thinking enough to get something LWN-length out, though. […]
Did anybody else read that as “something like 4% of Ubuntu users were edgy, while 76% of Debian users were unstable”?
As a Debian user, I fear it may even be true…
[…] Quotando Luis Villa : It is worth remembering that every significant community linux distro has a community of thousands who will gladly test anything you throw at them, so distros must actively encourage and take advantage of that. Any distro which doesn’t (and many don’t) is throwing away free time and free money. (Relatedly, I firmly believe that as a result of the opportunity for free QA, most open distros should in practice be more stable than their ‘enterprise’ alternatives, which have smaller user bases who would rather pay for someone else to do the work. That in practice enterprise distros tend to be more stable points to inefficiencies in how open distro QA is done, IMHO, not just the obvious points about business models.) […]
[…] Quotando Luis Villa : It is worth remembering that every significant community linux distro has a community of thousands who will gladly test anything you throw at them, so distros must actively encourage and take advantage of that. Any distro which doesn’t (and many don’t) is throwing away free time and free money. (Relatedly, I firmly believe that as a result of the opportunity for free QA, most open distros should in practice be more stable than their ‘enterprise’ alternatives, which have smaller user bases who would rather pay for someone else to do the work. That in practice enterprise distros tend to be more stable points to inefficiencies in how open distro QA is done, IMHO, not just the obvious points about business models.) […]