Product Management and QA: Constraining the Test Cases, Part 2/2

  • Article

DJ Dunkerley provided Part I/II earlier this week. Here's Part II/II:

____________

With CATS, what you really want to do is eliminate redundancy by only testing each parameter with every other parameter once. There is no algorithm (that I know of) that can generate the test cases for you, you need to sit down with colored pencils and a paper and draw up columns of letters that represent the parameters you need to tweak to generate the test cases. For the example above, you need to have 8 columns of five parameters, or 8 columns of parameters ABCDE. Let's generate the first test run, which will be all the parameter that will be tested with parameter A in the first column:
1. AAAAAAAA
2. ABCDEDCB
3. ACBCCBDC
4. ADEBCBBD
5. AEDEDEEE

Now we will generate all the test runs for parameter B in the first column (use a different color pencil or it gets confusing).

1. BABDCEDB
2. BBABDCBE
3. BCDCEDAE
4. BDCEABCE
5. BEEABAED

Move to C. Change color of your pencil.

1. CDACCDEC
2. CECEEABA
3. CABABBAB
4. CBDBACCD
5. CCEDDEDE

Oh, so boring and tedious. But not as boring as testing 390,000 test cases. Change the color of your pencil.

1. DADDACAB
2. DBBBBECA
3. DCCCDBDC
4. DDEECABE
5. DEABEDED

The last run. I promise.

1. EAEDDCEB
2. EBDAEDDE
3. ECAEAEBC
4. EDBBCBAA
5. EECCBACD

Whew, that was a lot of work, and I'm not sure I did it right. What we should have is just 25 test cases, with every parameter tested with every other parameter. That's right, you have even less test cases than if you were to test the parameters in isolation with one another. Another beautiful thing about pairwise testing is that the maximum number of test cases is X times Y where X is the column with the highest number of parameters and Y is the column with the second-highest number of test cases. It doesn't matter how many columns you have.

I first encountered this theory back in 1998 when I was the tester for the Prescript team. It allowed me to generate all the test cases for GUI testing and be very confident that I had covered all reasonable possibilities. Interestingly, when I tried to get developers to develop a program for me that would generate pairwise test cases for me, nobody was interested.

I guess this tells you what developers think about testing. Anyways, if you are in Q & A and you know about this theory, it can totally save your rear end (it did save mine), as it makes testing manageable (which is the name of the game). Quite frankly, it can make testing almost easy (in the sense you'll find the bugs really fast with these test runs).

Mind you, companies don't seem to hire QA people who have good theory knowledge of QA, rather they seem to leaning towards cheap talent that know some apps that purport to make testing easy (like the Mercury suite). Pity.

_____________

DJ you bring up an interesting system. So a challenge to the audience... do you have suggestions, comments or your own stories?

Thank you,
Stephen Ibaraki