The Debate on Security Metrics

  • Article

Recently I was sitting on a panel which was pretty heterogeneous: There was a representative from IBM (actually from former ISS), customers, a representative from the Open Source community (who actually, during his presentation always said how bad our security is) – well, and me.

In order to have some fun, the moderator wanted to bring some fire in the discussion and said: We often hear people saying that Open Source is more secure than your software model, what do you have to say on this? Well, there were so many different themes on the table which were – in my opinion – more interesting to discuss than a debate on Open Source vs. Microsoft, I actually did not want to go down that road. So, I asked the moderator back: Could you please elaborate a little bit what you mean by "more secure".

To cut this story short, we actually had a very good discussion on how security can be achieved, what is necessary and a little bit on metrics.

Why am I raising this? Well I read a blog post this morning on our Security Development Lifecycle blog called How Secure is Secure? Where Eric Bidstrup actually raises a few very good points:

  • He differentiates between Security Functional Requirements and Security Engineering Quality Requirements. If is obvious that the primary focus of SDL is the second. However, if you do good Threat Analysis, you will tackle quite a bunch of the first as well.
  • And then he points out an interesting figure: Microsoft has been releasing security bulletins since 1999. Based on some informal analysis that members of our organization have done, we believe well over 50% of *all* security bulletins have resulted from implementation vulnerabilities and by some estimates as high as 70-80%. (Some cases are questionable and we debate if they are truly "implementation issues" vs. "design issues" – hence this metric isn't precise, but still useful).

So, we could raise the debate again on the value of the "number of vulnerabilities"-metric again but I actually would rather like you to read the post.

Roger