Looking Back on Testing


It’s amazing how much can change in six years. Test has always been an area of constant innovation, creative problem solving, and process improvement. As a result of this continuous improvement we have dramatically changed our work lives as testers.

When I started my first contract job for Microsoft nearly six years ago I was responsible for running daily BVT’s (build verification tests) for SMTP in MCIS. I had four test machines, two x86’s and two Alpha’s. I had images to build x86 machines, but I had to build the Alpha’s from scratch every day. Someone told me I was lucky to only test on two platforms as there had previously been four (MIPS and PPC support had been recently cut). I was extremely busy, and being new to the industry the learning curve was huge, but I was really, really excited.

My day consisted of setting up four test machines, installing the product, and running the automation. Of course there were side projects that came up, failures to investigate, developers to follow up with, and bugs to log. However, the bulk of my time was spent on BVT’s, and I spent a lot more time preparing to test than actually testing. Running the automation, mostly perl scripts, didn’t take that long. Aside from BVT’s we had a few tools or scripts we used for isolated tasks, but most of the setup and testing was still done manually back then.

Needless to say a lot has changed. Now we have several automated tools to build topologies, and Windows 2000/2003 can be set up in less than an hour. I used to build four machines a day, now I could build hundreds in an hour or two (if I had that many machines). Now in Exchange nearly every tester writes code, and we develop test automation in step with product code. Test passes keep getting faster, we do more testing with fewer resources (aka people), we connect to test machines with Terminal Server and now I enjoy many days in a row without entering the lab.

Still, there is a lot that hasn’t changed. We still work hard, in an environment that is constantly changing, and we still reorg fairly often (I’ve had eight managers in 3.75 years since I was hired full time). We still own product quality and fight for our bugs, and the rush of finding a great bug remains the same. If someone needs to know how a feature really works, they still ask test. The friendly dev-test rivalry will always be part of life. Devs may think they have the upper hand, but as a tester you know the truth. You’re reminded every time a dev you’ve been working with sees you in the hall, stops in their tracks and says “Oh god” as you walk in their direction.

Carol Swales

Comments (1)
  1. ChrisS says:

    As an ISV and Microsoft partner I’d love to get more feedback on the types of scenarios they test and what they think is important to test. I know a number of other Exchange ISVs who have similar desires.

    For example, one ISV I know does a lot of testing around calendaring, and we do a lot of testing around SMTP transport sinks.

    Knowing broadly what a regression test looks like for MS would greatly help us to identify some of the corner cases and test against those. The net result would be better software for our customers and fewer PSS incidents related to 3rd party software (not that we’ve had any of those, but theoretically…) :)

Comments are closed.

Skip to main content