Test metrics series1 – Test efficiency vs. Test effectiveness

Introduction:

The theory of measurement is undisputed as measurement is crucial to the tracking and progress of all projects. Regardless of the measurement scale, when the data are gathered we need to analyze them to extract meaningful information of data, more importantly what needs to be measured?

Measure and metric

 A measure is a single quantitative property of an entity. This is a basic building block for any measurement program. In contrast metric is a derived quantity, which cannot be directly observed. In simpler terms measure is a number directly obtained by simple observations, where as a metric is derived from a measure using standard formulas and calculations.

All measurement will have a standard unit of measure (UOM) like seconds, meter, kilograms etc. In software, no of defects, KLOC (Kilo lines of code) are some examples.

There are two types of metrics, objective and subjective, objective metrics can be quantized and is readily available, but subjective metrics relies on opinions, gut feelings, personal attitudes etc. An example will be CSAT (customer satisfaction survey used as part of every engagement), though the former is more reliable, than the later, the reliability of subjective metrics can be improved by having checklists and guidelines. For e.g.: survey question need to have, probe areas, facets, scale definitions before an option can be chosen.

Test efficiency and Test effectiveness

I wanted to write about two vital metrics, which always comes to my mind, when I think about metrics. But I have seen that these two metrics are being used interchangeably in the software community, and both metrics have been given many different definitions. When there is no clear cut definition for a measure or a metric, then the objective of the program doesn’t make sense, there will be little effort to capture these and the test reports won’t capture data, facilitating the computation of the metric. Below we will be looking at how these two metrics have been perceived, differently, I would also like to share a set of checklist questions ^{Appendix A} which will help determining and eventually defining Test efficiency and Test effectiveness metrics as per our requirement. In short we need to know “What we want” ,” What we are going to do with it” , “How do we want it”, “How to get it”

Let us talk about the meaning of efficiency, most of the definitions state efficiency as the percentage of ratio of output to input of any system using unit measures (delivered /supplied). But I agree to the following definition.

Noun: The ratio of the effective or useful output to the total input in any system

Therefore, efficiency is an attribute which means to maximize the useful output for a given input reducing wastage or losses. Efficiency cannot be more than a 100%, in a sense that a 100% efficient system will have zero losses. Having discussed so much, now let us see what test efficiency mean.

Test efficiency (TE)

A Consultant as part of any engagement would want to deliver, the solution on time, within budget, having an acceptable level of quality, quantified by the customer. To achieve this goal, the team should work with efficiency, which is to constantly show progress to the effort put in. Test efficiency is a quality attribute of the test team, to carry out all testing activities in an efficient manner saving cost and time. To mention few of test efficiency focus areas,

1. Resources
2. Tools
3. People
4. Process
5. Time

Let us look at the broader picture of the metric, test efficiency is not only about test execution alone, but all or most of test activities, like test planning, comprehension, test cases creation, review, execution, defect tracking and closure. TE is not just one single derivation but a number of calculations at each phase and activity of testing. What activities or phases one is interested in, what is particularly measured, depends on lot of other things, the type of project, complexity, availability of resources, the situation, customer requirement etc.

So TE similar to Test effectiveness (TEF) can be calculated for specific set of test activities. For e.g.: Test preparation efficiency will be the time taken for ‘X’ number of test cases to be prepared, reviewed and reworked to finalize them. There is a catch here, the quality standards of the test cases should be predefined by using defining standards, as I have tried to state a few of them here.

1. The Test cases are complete with respect to Use Cases on which they are based.
2. A tester should be able to execute this test case using only this test case and any directly referenced items given the proper software and hardware configuration.
3. Test data must be specific. For example, don’t say “select any menu option to navigate out of current page.” Say “Click the Back button.” Don’t leave any test data to the imagination of the tester.
4. Usually, each test step should contain a single action. E.g. “Save” and “Search” functionalities should be split into 2 steps
5. Test Case names follow the agreed upon naming convention
6. No grammatical mistakes

Let us assume that there are two test teams with equal number of resources with comparable skill set, functional expertise etc working on the same product, and If test team ‘A’ has prepared 500 Test cases with agreed quality, for the product in 10 days and for the same product if test team ‘B’ prepares 500 Test cases in 8 days, then which team can we say is efficient? Definitely team B is efficient, but there is no guarantee that Test team ‘B’ is effective, we are not sure how many defect can team ‘B’ uncover as compared to team ‘A’. That is our next topic…

Test effectiveness (TEF)

Efficacy in contrast to efficiency is focused to just produce the desired result or effect or achievement as such and not the resources or time spent.

Noun: Effectiveness means the capability of producing an effect .

Let us look at what Test effectiveness is, Test effectiveness of a technique or a system or a team is the ability to find defects and isolate them, from a product or deliverable. Test effectiveness is to ensure quality and close the two quality gaps, namely producer’s quality gap and customer’s quality gap. As definition of quality goes, quality is both process and product quality which is meeting customer requirements and conformance to product specification. These metrics should be quantified, as they closely relate to quality, and for many people the term quality is relative.

Can we take a simple example of calculating the test effectiveness during each testing phase?

Be happy if some product has these many defects as given below J , nevertheless I will rather turn suspicious.

The table shows the defect origin, on the X, where a defect was injected, where he belongs to? and on the Y, where the defect was detected.

Let us calculate the test effectiveness of the Integration testing activity:

Total number of defects of all origin found during Integration testing activity = 17

Total number of defects existing while entering in IT = (6+15+28) – (35) = 14

Total number of defects injected in the current stage = 14

Effectiveness of IT test phase = Total defects found in this phase/(No of defects existing + injected)

Effectiveness = 17/(14+14 ) * 100 = 60.7%