More on QA, Ubuntu, trust, etc.

Posted on by Luis Villa

Heard from Ubuntu this morning (in comments and via email, neither official) so I figured I owed an update, having slammed them fairly thoroughly here :) So some notes from email and comments:

  • I didn’t see an official Ubuntu announce about this because I didn’t look in the most obvious place of all- ubuntu.com. Looking there points you to this message, with more details in a subsequent linked page. Good on Ubuntu for discussing the issue in the most highly visible place they can, and promising (albeit all the way at the bottom of the second page) that they are investigating the problem. Given that one of the most valuable things any distro has (especially Ubuntu) is the trust of its users, I would probably have given the ‘we are researching the problem’ statement much more prominence, but it is there, at least.
  • To be very clear: I don’t expect Ubuntu to have researched the cause of the procedural problem and fixed it in 48 hours. That would be nice but unreasonable. I just expect them to very publicly say what they are doing about the problem, in terms of research, etc.
  • To also be clear: I’m surprised I’ve seen nothing on planet ubuntu (not planet gnome), because I assume that at least some developers blog about what they are thinking about/working on, and if no developer blogs about this Very Big Fuckup, then… that ain’t good :)
  • The negatives: apparently the problem was there for 17 hours. Not a good sign, but again, that is partially because I have high standards for Ubuntu.
  • Apparently the reason I didn’t know about dapper-proposed is that it isn’t fully deployed yet. That is mixed news, I guess- good that there is a reason I didn’t know about it; bad that something like dapper-proposed was not fully tested and in place before the LTS release. (Note here that again I’m holding Ubuntu to a very high standard; as far as I know no other distro has such a queue for their long-term distros yet either. Of course, every distro should. If I’m wrong, and other distros do have it, I’d love to know- please let me know in comments.)
  • James: I’ve not considered an LWN article on distro QA because for quite some time (really since around when I left Novell) I’ve been pondering writing the definitive serious white paper on the subject. As dobey is about to find out, writing anything of that length is hard :) We’ll see if these blog posts coalesce my thinking enough to get something LWN-length out, though.
  • error27, others who have discussed enterprise distros: Enterprise distros have substantial resources directed at identifying stable upstream versions, and stabilizing them even more. So of course we should expect that at this point enterprise distros are very stable; more so than their more bleeding-edge community counterparts. However, traditional enterprise distros can only be resourced from within the company that produces them, and their users are explicitly paying not to worry about it- the payment is mostly in lieu of other forms of contribution. In contrast, a community distro like Fedora or Ubuntu should be virtually unlimited in terms of the amount of testing, feedback, triage, etc., that it receives from community members. Given that, if coordination and communication problems are solved, community distros should be of at least equal quality to enterprise distros. (It should be of no surprise, given that coordination/communication problems are perhaps the biggest stumbling block to this, that I think everyone needs a bugmaster.)
  • Go read the comments in last night’s post for more comments on the Edgy/Unstable differences. They are all dead on; no need for me to repeat them, except to say that obviously there are a lot of various layers to the disparity. Still, the basic question stands: how do you get more people onto unstable, and get them contributing?

I swear I’ll write something about law school soon :)

Posted in QA

Published by Luis Villa

27 thoughts on “More on QA, Ubuntu, trust, etc.”

  1. http://tieguy.org/blog/2006/08/22/still-learning-what-long-term-support-means/) that ended up being a very interesting discussion (http://tieguy.org/blog/2006/08/23/notes-about-distros-qa-etc/) touching various aspects of how you focus on QA (http://tieguy.org/blog/2006/08/24/more-on-qa-ubuntu-trust-etc/). He points out that * Fedora is now support upgrades between distributions – I believe he is talking about live upgrades using yum rather than using the Anaconda installer (which has always been supported). I havent heard of anything public

  2. Who are we talking to? The recent X.org breakage highlighted the variation in communication expectations and needs between different audiences. Luis Villa emphasised the importance of blogs belonging to prominent project members. Other members of the Ubuntu community may look to the ubuntu-users mailing list or the forums. More than likely, the majority of “ordinary” Ubuntu users would seek information on Ubuntu.com.

  6. Fedora has the updates-testing repo available, this has been in place basically since day one and works beautifully for QA purposes, you are also as a QA tester encouraged to sign up for the fedora-test mailing list. Fedora contray to popular belief is a longterm supported system, after Red Hat EOLs a given Fedora release it is handed over to Fedora Legacy and can as such in theory be supported indefinitely. Though the actual Legacy entension is somewhere around 2 years. Meaning a total of about 4 releases of support for a given release (more than Dappers LTS promise for desktop deployment).

  8. Mark: I’d forgotten about Debian’s queue, though of course it has been around for years. (5 or so, IIRC?)

    David: The line between Fedora and Fedora Legacy is a blurry one. Suffice to say that Fedora does send out messages about stopping support for FC releases (even if Legacy will pick them up) fairly quickly, relative to other distros. I did not know about updates-testing- that’s great. I’m glad to have learned quite a bit about Fedora in the past few days and will have to give it a more serious look for my next box.

    Robert: I’m not sure who/what this ubuntu-demon is, but ‘random animated head without a name meandering about doing a lot of engineering work to fix a problem that should not exist’ was not what I was looking for when I was looking for information on planet ubuntu :) [To be specific, X server failover sounds offhand like a dumb idea. It’s like kernel failover. X should never break (and in practice, very rarely does!), so spending lots of engineering coping with what happens when it does break is wasteful. It fixes the symptom, not the core illness.]

  9. apt-listbugs is pretty nice when running debian… too bad there’s not a version integrated with launchpad (somehow).

  10. Indeed Luis the line is so blurry that the next version of Fedora should ship with the Legacy repo enabled by default so that once it reaches EOL it will just continue to work, no interaction required by the user what so ever. Neat right?

    The only problem with Legacy currently is a slight lack of manpower, Red Hat hired the Legacy lead developer Jesse Keating to act as the Fedora Release Engineer and I don’t know if we really have a person willing to take on the task without getting paid.. Luckily work it progressing nicely regardless so I’m sure it will work itself out, there is after all a great interest in the project. Hopefully some company will hire us some help if we can make them see it as the good investment would be.

    As Legacy gets integrated with the distro the EOL date becaomes somewhat irrelevant, if a given release is solid and depended upon there would be no problem extending it’s lifetime provided we get help. How many companies who depend on Legacy or want to couldn’t give up half a days work of one guy in their IT department to help that project? – given the cost saving compared to a lenghty upgrade it would probably make it worthwhile. If that happened there would be no reason we couldn’t support Fedora for years and years. Help make it happen please.

  11. Ubuntu Demon was probably the one who saved me, being the only information source I could find at the time.
    The notice on ubuntu.com WASN’T there then, it only got there later.

    Another thing: Ubuntees are very bad at describing their changes. The most recent changelog entry for the upgrade that fixed things again was actually the changelog entry for the change that broke things. Better than nothing, I s’pose, which is how package upgrade changes are usually described.

    I learned about the problem when a friend of mine called that he had this nvidia problem on his machine after turning it on again when he returned from work. I did remember this little X upgrade in the morning (and he had also upgraded in the morning and then shut his machine off). After we had established that the usual workarounds didn’t work (nv, ChipID) and that starting a second X server on my machine didn’t work either, we had an inkling and I looked around on Planet Ubuntu and ubuntu.com.

    Ubuntu Demon had pointers to the bug report and various forum threads about it and everything.
    Apparently they had discovered the problem and after some deliberation removed the patch(es), bumped the version number, started a new build + mirroring, etc … and then gone to sleep, without making sure that there was official mention or anything on their main page.

    Very good communication from him :) and very bad from Canonical :(

  12. Re. planet, maybe Ubuntu devs prefer to spend their time working rather than blogging :) Many open source projects don’t have devs blogging at all.

  13. Matt: I was wondering how long until that point was made, but was in a hurry this morning and didn’t want to elaborate. (For what it is worth, I expected it would be Dave Camp who would make the point.)

    I think that at this point anyone paid to develop in an open source community should consider communication with the community about ongoing process part of the job responsibility, especially in a community where a planet exists, greatly increasing the probability that the community will read any particular developer’s blog. The amount of time used to say ‘fuck, we really blew that, and it pisses me off, and we’re trying hard to figure out how we make sure it doesn’t happen again’ is very small; the benefit gained in the community is very large in contrast. Or to put it another way: if I’m Mark, and the people I pay to be a critical part of my community are not participating in my community even in time of crisis, when the community is most demanding of information, then I’m pissed.

    I agree that in general, daily blog posts of huge duration detailing the author’s favorite boing boing posts or even a blow-by-blow of the day’s work are not necessarily useful, and I certainly think volunteers can do whatever the fuck they want with their time. But a certain minimum of communication is necessary for a healthy community and long-term project success, and blogs are a great, low-effort way to do that, so if I were investing, that’s one of the things I would look for.

  14. Hi Luis:

    I don’t now much about the Ubuntu setup wrt payment etc, since I don’t really travel in those circles, but it seems to me that as far as people being paid having a responsibility to communicate, they’ve fulfilled that by posting an official announcement on their front page.

    I know the GNOMErs are very blog-friendly, but it just doesn’t always work that way – it’s a personal and cultural thing that doesn’t exist everywhere. I’m quite involved in the Blender project, I have a blog that I post to from time to time, and I agree about the value of them. But there’s really only one other Blender developer that I know of that has a blog, and even then it’s not updated frequently at all. I certainly don’t post to mine that often, and rarely about the things I do in Blender dev either. Apart from that, none of the other Blender developers really blog, and that’s just fine – communication happens in other various ways, it’s going extremely well and I think is a good example of a very successful open source project. Anyway, I’m just saying that as long as they’re communicating, that’s the main thing, and that if so, it’s not fair to condemn them for not using a particular medium.

    cheers

  15. Mike: I have no idea whether or not Mark and Canonical think that Ubuntu employees should blog, and I assume none of them have it in their job description (except maybe Jono.) But I do think that Mark and Canonical should think that Ubuntu employees should blog as part of their job- reasonable amounts of blogging by your employees is an investment in your community.

    There are many ways and levels of communicating. Ubuntu.com is one way (and a good and necessary one) but there are also other ways of speaking with customers, or in this case, with community members who are at least nominally peers. That communication is best when it is informal and human- think of the way you’d speak to your friends when you have a bad day at work, versus how you’d speak to your boss or your customers. Ubuntu can’t put ‘shit, we fucked up’ on ubuntu.com, but it can on the planet, and that framing makes the whole project look more human and more accessible- which is an investment in Ubuntu’s biggest competitive advantage.

    (I would recommend, if you haven’t read it already, reading the Cluetrain Manifesto, and also adding Passionate Users and (maybe) Seth Godin to your blog list.)

  16. On blogging; a couple of things to keep in mind.

    The first is that the developers are all in one place at the moment for the Distro Team Sprint, and during such meetings it tends to be hard to blog due to the insanely busy nature of them. Actually being together has been helpful, as you can imagine there’s been a lot of post-mortem of the fuck up.

    And the second is that jobs could be on the line, so I suspect people are being overly cautious about saying anything for fear of getting fired or at least having recriminations.

    I would disagree that it’s a responsibility to blog; personally I find it one of the least efficient ways of talking about things, which is probably why I don’t blog that much!

  21. I lost my boobs

    [Ed.: left in by the editor because somehow I feel this is the most amusing blogspam I’ve ever seen. Link itself deleted.]

  26. […] real faces and real names is established, people get upset when the conventions are broken (quoth Luis Villa: I’m not sure who/what this ubuntu-demon is, but ‘random animated head without a name […]

Comments are closed.