A Blog's Life: A Windows Vista Test Manager Addresses Bug Count
You can be forgiven for having missed the fanfare around the kick-off of Windows Test Manager Paul Donnelly's blog, entitled Customer feedback makes for a better Windows ... because there wasn't any. Paul's initial post is unassuming but provocative, as he responds to a Longhorn Blogs article by Robert McLaws entitled Windows Vista Bug Reports: An Analysis, where Robert analyzes the team's incorporation of customer feedback into the Windows Vista development process.
In "About all those bugs, Robert...," as Paul puts it, it's fair to take into consideration two points that Paul makes in an attempt to clarify the team's handling of customer feedback. First, Paul states,
About the increasing quantity of bug reports – what is not factored in is the quantity of people we are adding to our programs. We are constantly adding new people to our programs to get more feedback. That coupled with the sheer interest generated as we approach the final stages tends to account for the uptick.
Thus, it's fair to say that the ratio of bugs per customer is steadily dropping as we incorporate more customers into the testing process while continuing to resolve issues. This should be interpreted as a good thing, in that each individual customer is discovering fewer and fewer issues. It shows that we're improving the product with each successive version and that our customers are playing a vital role in that.
Second, and arguably of even greater significance, is Paul's comment about the gravity of the issues being reported of late:
What is significant is that the severity [of] reported bugs is dropping. We’ve moved from seeing the serious issues you see in a Beta 1 to a very usable system that generates more ‘this doesn’t work quite right’ issues at Beta 2.
Again in this case, this should be construed as a very positive trend and evidence of the impact our customer-engagement programs such as TAP (Technology Adoption Program) and CPP (Customer Preview Program) continue to have on Windows Vista.
The funny thing is that this is to my knowledge the first time that one of our customers has gone so far as to calculate the number of issues identified by customers over time and its correlated effect on product quality. While you should not expect us at MS to publish such information publicly, you can nevertheless take this as evidence that our engineering teams are well aware of the effect customer feedback has on identifying and resolving bugs in Windows Vista.