Earlier this year, we started publishing a new set of metrics on our portal – An evaluation of our protection performance and capabilities. These metrics show month over month how we do in three areas: coverage, quality, and customer experience in protecting our customers.
And, since we started to publish the results on this page, I've had many great conversations with customers and partners alike, discussing what the results mean for their organization and their protections. In this post, I want to cover some of the most common taxonomy questions I was asked during those conversations and also discuss the results for September 2013.
First, let's dive into what the terms we use really mean:
- Coverage – the infection metric
This is how we measure threat misses and infections. If we block a threat, that means we've protected our customers as expected and that's a win. Misses and infections show up as a red dot and the bar chart in red.
Misses are threats we had early warning detections on (non-blocking detection), but by the time we determined it to be a threat, the threat had either disappeared or changed into a different file on the computer.
Infections are threats we detected and then had to remediate (instead of a block). We call these active because, according to our telemetry, they appeared to have some active running component when we detected them. On the positive side, our real-time protection detected and worked to remove the active threat. We continue to work on methods to determine the ways in which threats become active, for example, through vulnerability exploits, through another program that drops the malware, or through credential-based attacks so that we can further address these active threats and provide actionable information to customers about how to protect themselves.
Here's why that's important. Many threats, like Conficker, show up as active because the threat uses passwords or exploits that were effective in compromising the system for a very brief moment in time. For example, 85% of Conficker infections on Windows 7 happen through credential-based attacks (read more about this Conficker case in SIRv12). When we detect a Conficker infection that was delivered this way (which happens immediately), we identify it as active because it was written by a system process compromised through a credential-based attack.
- Quality – Incorrect detections
Incorrect detections happen when antimalware products incorrectly flag and misclassify a file as malware or unwanted software. The yellow dot and the other bar chart represent incorrect detections. In any given month, only an extremely small number of programs are incorrectly detected. In most months in 2013, for example, only 1 in a million customers experienced an incorrect detection - the percent of customers with incorrect detections was less than three zeros to the right of the decimal (<0.0001%).
- Customer experience
With this criteria, we measure the performance implications of antimalware on the day-to-day activities that a person might perform – such as opening an application, browsing the web, downloading files, and playing games and multimedia. Latency perceptible by a human tends to land within the 50 to 100 millisecond range. In most months, most activities stay under 100 milliseconds latency. This is the second graphic on our results page and it shows the customer experience when running the latest version of Windows Defender on the latest version of Windows 8. September's measurement reflects Windows 8.1.
To sum it up, the two graphics on our results page highlight the findings for coverage, quality, and customer experience (in terms of system performance). The first graphic shows protection coverage and quality for Microsoft's real-time protection products that cover home, small business, and enterprise, which represent approximately 150 million endpoints. The second graphic shows the performance implications when running the latest version of Windows Defender on the latest version of Windows 8. There is a great whitepaper that provides additional insights at this link.
And finally, let's talk about the September 2013 results:
- Coverage and top infections – September 2013
In September, 0.17% of our customers encountered a miss (0.03%) or an infection (0.14%). This infection number was uncharacteristically high because of the resurgence of an old threat we currently call Sefnit. 44% of the active detections for the month were related to this Sefnit family. That's a very large percentage – on normal months, no one family represents more than 6% of active infections. As we investigated the threat, we noticed that the distributors of Sefnit were using some sneaky techniques to infect computers, including using installer programs that install legitimate software but occasionally install legitimate software with bonus material (Sefnit). Sefnit distributors are also modifying the appearance of components, such as sometimes using an obfuscator and then sometimes not.
- Incorrect detections – September 2013
This month, only 0.00025% customers were impacted due to incorrect detections. This percentage was slightly above average. The driver for the slightly above average impact was due to an incorrect detection on a 2009 version of the Microsoft Malicious Software Removal Tool.
- Customer experience – September 2013
We consistently provide great performance for our customers using Microsoft antimalware products. In September 2013, the results have been consistent with the 50 to 100 milliseconds range.
Our goal is to provide great antimalware solutions for our consumer and business customers. I hope this blog demonstrates how committed we are in raising the bar for ourselves and others in the industry for doing so. We're monitoring our results, performance, and progress closely, prioritizing for real threats that might affect our customers and applying lessons learned to make our products even better. Plus, we support antimalware partners in order to build a strong and diverse ecosystem to fight malware – the true adversary.
Holly Stewart, Senior Program Management Lead, MMPC