productive testing tips

I was cited in an article on testing tips this past week. Here is some cut and paste from the email I sent to the author (Joe Barr) when he asked me for tips; my email goes into more detail than he was able to put in his article so I thought it might be worth posting.

  1. use a bugtracking system of some sort, and use as much metadata (including good titles) in that bugtracking system as you can. At first it is time-consuming, but over the long run, it’ll save you time by helping you avoid testing things or filing bugs twice. Relatedly, if we’re talking about things that can be done as a group, and not just as an individual, have a bugmaster- someone charged with organizing, sorting, and generally knowing what is going on with the bug tracker. That one person will help every other person who uses the bug tracker (both testers and developers) be more productive, which is invaluable.
  2. Where possible, use the very latest code. Don’t be afraid to rebuild things from CVS every night or every morning. The moreup-to-date your code is, the quicker you’ll catch things (again, helping everyone’s productivity) and the less time you’ll spend going back and forth with ‘is this in the latest version?’ Again, when speaking of a team, if someone can be charged with making this process as easy as possible, that one person has a productivity multiplier for the entire team- makes everyone more productive.
  3. If at all possible, write automated unit tests. The best way to be productive is to have the computer do the work for you. Again, this has up-front costs, but over the long run is a *huge* win. (If we’re talking about GUI software, try to focus on non-GUI code first when writing tests- GUI tests tend to be fragile, and you’re best off writing tests that will fail when things go badly wrong, not when a pixel shifted here or there.)
  4. dogfood, dogfood, dogfood. If at all possible, use the code you’re testing under real-life conditions to do everyday work. Real-life testing is always going to be more efficient than fake ‘do this, then do this, then do this’ checklist testing- that doesn’t catch edge cases, and it feels like work. If you’re using code for real work, you catch the edge cases that real users find, and you do it in the courseof doing something else- it doesn’t feel like work then. (Relatedly: if you dogfood and find a bug, make sure to do (1) file it immediately, and (3) write an automated test to duplicate the bug ASAP.)
  5. use automated crash reporting tools: every major OS now has ways to catch stack traces of catches and send them back to a bug database. Use those, and ship those- help the users help you.
  6. if you have an active volunteer community who are using nightly builds to do daily work, don’t spend your time testing things that they will inevitably test. For example, I’ve seen test plans that say things like ‘check to make sure it launches’. If it doesn’t run from the command line, a good community will let you know about it ASAP. If it crashes when you open the print dialog with 10,000 printers on your local network, well, most communities don’t run into that sort of thing- they have one printer. So spend your precious test cycles testing *that* kind of scenario, instead of testing basic stuff our community will catch like ‘does it start? does the file open dialog work?’

8 thoughts on “productive testing tips”

  1. Zapomniałeś zapłacić rachunek ? Billreminder ci o tym przypomni. Więcej oraz przykładowe screenshot’y na stronie Og Maciel’a tutaj.Jak efektywnie testować. Przynajmniej kilka rad dał Luis Villa. Więcej na jego blog’u.Zasięg użycia google earth jest coraz bardziej imponujący. Można np. zobaczyć jakie zniszczenia stworzyły kopalnie odkrywkowe. Więcej w artykule na SlashDot’cie.Linux domination ? Po IBM’ie, Monachium itp. teraz Citroen się zdecydował na

  2. “every major OS now has ways to catch stack traces of catches and send them back to a bug database”

    it would also help if the tools would automate the activation of debug flags or other necessary information; most of the stack traces i’ve submitted have come back with messages like the following:

    “Unfortunately the stacktrace does not have any hints about the real cause of the crash and does not help in debugging this issue.”

  3. “2. Where possible, use the very latest code”

    I think there is too much emphasis on the latest code. How about fixing issues in current releases instead of always pointing to new releases?

    I am talking from a users point of view where I always have to upgrade to the latest version just to get fixes that should go into the current version (Firefox, Gnome etc.)

  4. @stephen: the user would have to install additional debug *packages* to provide a better stacktrace. we hope that for 2.20, we will be able to use the google airbag code to workaround this problem.

  5. Chris: Whether you’ll be testing stable or unstable depends on a lot of things, including developer availability, project priorities, etc., which were beyond the scope of my comment. The point I was trying to get across was that you need to use the latest version of whatever you’re trying to test. That means that if your purpose is to test the stable branch, you need to be building out of stable branch revision control which the next tarballs will come from, rather than the last released set of tarballs. Hope that is more clear.

Comments are closed.