Creating Risk Management Metrics that Matter
As a security team, you are what you measure. The problem is that too many security teams are tracking vulnerabilities, not measuring risk. This post examines how vital it is for security teams to establish risk-based metrics, offering examples of both the right and wrong measures to use. The paper then looks at the key steps to building risk management into operations, so security teams can better ensure that efforts, workflows, and investments are aligned with what matters most: strengthening security.
The Promise/Demand for Risk Management
To be effective, security teams need to move from a focus on vulnerabilities to a focus on risk management. Why is this distinction so vital? It’s essential for security teams to understand the spectrum of risk, based both on the likelihood of a vulnerability being exploited, and the potential damage that may result.
When you adopt this risk management approach, you focus on the issues that pose the biggest danger to the business. Only by doing so can security teams ensure that they’re focusing their limited resources on the efforts that matter most. That way, they can effectively track and make progress towards the ultimate goal: reducing the likelihood of a security incident that would hurt the business.
Before we keep going, here’s a few key definitions:
- Risk. Risk is a calculation of the probable frequency and magnitude of a future loss.
- Threat. Threat is a negative event that results in the loss, damage or exposure of an asset.
- Vulnerability. Vulnerabilities are what make a negative event possible or more significant.
For more information on these terms, here are two great resources:
- The Open Group has written an extensive standard, “Risk Taxonomy (O-RT), Version 2.0.”
- Daniel Miessler has written an informative, high-level tutorial, “The Difference Between Threats, Threat Actors, Vulnerabilities, and Risks”
The Problem: Security Can’t Go It Alone
So the security team has started taking a risk management approach and all’s going to be rosy, right? Not exactly. Once security practitioners embrace risk management, in many ways the hard work is only just beginning. The problem is that security teams can’t do it alone. When it comes to establishing a strong security posture, a lot of individuals from different organizations need to be involved. Once security is working towards managing risk, the rest of the organization needs to start singing from the same song book.
Security needs to rely on a number of other individuals from a range of teams, and all too often they have limited success in getting them to do what’s required. Part of this is self-inflicted. As Kenna Security has discussed in other white papers, security teams can be their own worst enemy. Too often, they resort to dumping huge lists of vulnerabilities on unsuspecting developers and administrators. Not surprisingly, the results are mixed at best. So how do security practitioners get better at building support for risk management across remediation teams?
How to Realize the Promise: Operationalizing Risk Management
For security practitioners and the business to succeed in reducing risk, risk management needs to be incorporated into operations across the organization. When security starts to be part of core operations—rather than an ad hoc afterthought for those outside of security—the critical efforts that need to happen, do happen, and ultimately strides are made in strengthening security.
To begin to operationalize risk management, organizations need to execute two key efforts:
- Measurement. Before an organization can operationalize risk management, practitioners need to make sure they’re measuring actual risk. All key stakeholders should buy off on what is being measured and ensure actual risk reduction is being encouraged.
- Integration. Once an organization is reporting on risk, it is critical to ingrain risk management in core policies and procedures.
The following sections offer key insights into each of these areas.
Step 1: Selecting the Right Metrics for Measuring Risk
I often meet with security practitioners, including folks from large and small businesses, and from a wide range of industries. Across this diverse range of businesses, I see some common missteps. Chief among these is that security teams are measuring the wrong things.
More often than not, teams are taking a vulnerability counting approach. A security analyst will run a report and find that many vulnerabilities have been unaddressed for longer than 90 days. He or she will then prioritize remediation efforts based on this aging data—for example, they may direct the team to focus on the vulnerabilities that have been unaddressed the longest.
Contrast this approach with a risk-based strategy. If you take a risk-based approach, you may realize that those older vulnerabilities don’t pose as much risk, but that three vulnerabilities discovered yesterday pose both a great likelihood of being exploited and significant potential damage to the business. With this insight, the need to prioritize these three vulnerabilities is clear.
These contrasting scenarios underscore the criticality of tracking and reporting on the right metrics. Metrics are vital in incentivizing and guiding behavior, and play a key role in measuring success, tracking progress, getting executive buy-in, and investing in new solutions and approaches.
Ultimately, the reason we preach the importance of risk management versus vulnerability counts is this: It can be far better to address one high-risk vulnerability than 100 low-risk vulnerabilities when accounting for real likelihoods and impact. The key is to establish metrics that truly measure risk in an objective, meaningful way, so you can make these kinds of calculations with clarity.
Security teams need to establish metrics that focus on risk, both in terms of the likelihood of a vulnerability being exploited as well as the business impact that exploit would have. Metrics should track and incent progress in reducing risk across these two attributes. These metrics therefore need to factor in associated assets, including an asset’s importance and impact to the business, processes associated with an asset, what mitigating controls may be in place for an asset, and the most common forms of attack that would target the asset.
While specific metrics that are optimal will vary somewhat depending on the nature of the specific business and technological environment, there are some common dos and don’ts when it comes to metrics. Following are some details on what makes a good or not-so-good risk-management metric.
Metrics to Avoid
Following are a few metrics to avoid:
- Total open vulnerabilities
- Average vulnerability age
- Total vulnerabilities open longer than X days
Why Calculating Averages Can Be a Fool’s Errand
Another common mistake I run into is a reliance on averages. The problem is that averages tend to be heavily skewed by outliers. Take average time to remediate vulnerabilities as one example. A team may have addressed a few vulnerabilities very quickly or let a few languish for a significant length of time. Either way, the metric can be heavily skewed by these exceptions. For the most part, you’re much better off assessing these types of metrics by using median calculations.
Solid Metrics to Consider
Organizations that employ a risk-based approach can consider tracking a number of key metrics:
- Remediation rate of high-risk vulnerabilities, and the number of these high-risk vulnerabilities in a specific environment
- Median time to remediate a high-risk issue
- Median time to discover a high-risk issue
- Number of high-risk assets (Note: This is VERY different than tracking high-risk vulnerabilities)
By and large, if you’re tracking these metrics and seeing progress, you’ll be making real improvements in overall security.
Step 2: Integrate Risk Management into Operational Processes
Don’t Reinvent Wheels—They Will Come Off
When it comes to operationalizing risk management, don’t start by trying to create new operational processes. Instead, focus on inserting risk management into existing processes whenever possible.
Too often, security teams have created out-of-band tools and procedures—and the results suffer. Under any circumstances, it’s going to be challenging to get remediation teams to focus on security activities. Creating these unique tools and workflows significantly exacerbates this challenge.
Every day, each staff member has to balance and prioritize among a host of competing efforts and activities. The last thing you want to do is add barriers to their taking care of the security efforts that matter. But that’s what happens when creating out-of-band security processes.
To significantly enhance your odds of success, leverage existing teams’ processes wherever possible. Look to bake risk management into existing tools and workflows that staff members are using every day. In effect, you’re starting with what everyone is doing today, and applying a risk-based lens to it. For example, find out how to feed your prioritized risks into the development team’s bug tracking. Weave remediation efforts into the operations team’s existing service desk, ticketing systems, change management platforms, and workflows.
Leverage Compliance Efforts and Investments
Getting buy-in and collaborating with the organization’s compliance teams and efforts is another great way to operationalize risk management. In most organizations, the reality is that significant, and ongoing efforts and investments have been made in addressing compliance mandates.
By getting buy-in from auditing staff, you can establish a strong partner in risk management. For example, if the team is preparing for an upcoming audit, look at the efforts underway and find out how risk management can be tied into these efforts. This can be a great way to leverage the urgency around audit deadlines to accelerate risk management efforts, while at the same time enhancing operational efficiency for everyone. Ultimately, you can address both near-term compliance requirements and enhance security in the long term. Instead of scrambling to comply with a checkbox-focused audit, it then becomes about managing risk in an effective manner.
The Payoff of Operationalizing Risk Management
When security teams adopt risk management and then get it encoded into their organization’s core operational processes and systems, good things start to happen for these groups:
- Security teams. Security staff starts measuring real risk, and understands how best to reduce it. Ultimately, they become more effective at meeting their charters: improving security.
- Remediation teams. Remediation teams can become much more productive. They continue to work with the tools and processes they are comfortable with and know—instead of dealing with the time and effort of dropping other tasks, getting trained on a new platform, and so on. They aren’t stuck feeling like they’re doing busy work for the security folks, but rather they get visibility into risks facing the business, and how they can play a part in reducing them. With a risk-based approach, remediation teams get small, meaningful chunks of work, say the top three things they can do to significantly strengthen security, versus running on the vulnerability treadmill and not making any real progress.
- Executive teams. When security teams start tracking and reporting on real risk, executive teams can gain a much better understanding of the company’s security posture, how it’s changing, and, most importantly, which efforts and investments need to be made to improve it. Ultimately, the business profits from stronger security and better productivity across a number of different business groups.
How Kenna Can Help
Too often, security teams’ efforts are being stymied by their ongoing focus on vulnerabilities. By taking a risk-based management approach, they can ensure they’re focused on the activities that matter. Further, they can start partnering more effectively with remediation teams, so these different groups better support security objectives. However, in order to adopt this risk management approach, many security teams need to add some new tools to their arsenal. Vulnerability scanners are only one small piece of the solution.
Kenna helps teams leverage more intelligence to move beyond identifying vulnerabilities, and start measuring and managing risk. Kenna helps security teams get clear on where the biggest threats are and how they can be addressed. This information enables security teams to provide prioritized lists of efforts and clear solutions, so remediation teams won’t have to sift through giant reports or waste time trying to figure out what the fix is. With this information, security teams can become more effective at collaborating, coordinating efforts, and strengthening security.