Larry has a great post about the myth of zero defects. As a release manager (or rather, bean counter), the issues discussed there are part of my daily life.
One interesting aspect of bug management that may not be immediately obvious has to do with automation. Years ago, product teams at Microsoft didn’t have much of it. Many of the tools were in their infancy and we still did a lot of manual testing. Since then, automation has become an increasingly important part of the testing diet, which is great for many reasons.
On paper, you’d think it to be reasonable that if you have more automation, you’ll find bugs faster, be able to verify fixes faster and thus save testers the time they used to spend doing those things manually. So therefore you can get to that zero bug bounce faster and thus can do more writing of new code than you used to in the same time frame.
But there are many complexities – the two foremost on my mind are:
- The somewhat obvious point that writing the automation takes time.
- Automation is code. Code has bugs.
You have to schedule time to write the automation, of course, and that time replaces a chunk of the time testing used to spend manually testing. But you also have to account for time to fix the bugs in the automation just like you account for time to fix the bugs in the product code. This is something we didn’t used to have to deal with years ago, and so we’ve been learning how to adjust our planning to handle this. This is the kind of thing I deal with all the time as a release manager.
The increase in automation is a fantastic thing for us – we don’t just release a product, we release a test product. That test product is used to verify the product operates as expected as the environment changes (OS service packs, etc) and as the product changes with service packs. But overall it’s been an eye-opening experience for me to see how the improvements in engineering excellence have impacted our scheduling (in both directions).