Kenna Security is now part of Cisco

|Learn more
Contact Us
Talk to an Expert
Request a demo

7 Questions to Ask Every Vulnerability Management Vendor

Jun 15, 2021
Kenna Security

Share with Your Network

Vulnerability management is a wicked problem. And not in an endearing New England kind of way. “Wicked” is actually a recognized term describing how a problem shifts under your feet as you’re trying to solve it. In fact, the mere act of trying to solve it shifts the problem itself. 

This makes vulnerability management a moving target. And when traditional scanners and CVSS-based prioritization inflate the criticality of large sums of vulns (while understating the criticality of others), you’re faced with tackling an even more difficult task that doesn’t actually get you closer to lowering your risk. 

But there’s good news. The number of vulnerabilities that actually pose a real potential threat to your organization is much smaller than you think. Roughly 2-5% of your observed vulnerabilities are actually exploited. So the real challenge then becomes targeting these high-risk vulns without missing anything vital and without expending too much energy and resources to do so. 

This is why the solutions you choose to help you accomplish this is key. And with this narrowed problem scope, more sophisticated vulnerability management approaches become possible. 

We get it. This can be murky water to wade through. Separating sales pitches and marketing from an actual offering and technology can be exhausting. Research it long enough, and scanners or vulnerability prioritization tools all start sounding alike. 

Fortunately, we have some tips to help you understand how to measure and test a vendor’s performance and effectiveness. 

Remediation strategies: Understanding coverage and efficiency

Before you measure vendors, you must be able to measure yourself. Understanding effective remediation strategies (particularly your remediation strategy) is key. As pioneers in the risk-based vulnerability management (RBVM) space, we’ve evolved our success metrics to focus on coverage and efficiency (sometimes known as precision and recall). 

In a nutshell, coverage refers to the percentage of known risk being remediated. Of those remediated within the net you cast, efficiency represents the number that posed a real threat. For more on this key concept, check out Coverage and Efficiency of Vulnerability Remediation

Can you look at your own vulnerability management program and figure out the coverage and efficiency target you have? How efficient you are in achieving that coverage and can you get more efficient? 

7 questions every CISO needs to ask vendors

As you look to lower your risk in as few moves as possible, we’ve compiled some key questions to ask vulnerability management vendors. These should help you narrow your consideration set:

  • What percentage of the National Vulnerability Database (NVD) do you have signatures for? First and foremost, be sure to ask a vendor this: Out of the 150,000+ vulns found in NVD, how many would they capture in an authenticated network scan today if they triggered every signature? If they don’t know, they don’t know the base rate for their scanner. That’s a red flag.
  • What is your historical false positive rate for authenticated network scans? As good as scanners have gotten, network scans uncover plenty of false positives. A reputable vendor should be measuring these and able to disclose this information. 
  • What percentage of vulnerabilities are returned high or critical? Every vendor should be working off some kind of model, and even better if it’s a calibrated model. In your exploratory conversations with a potential vendor, gain an understanding of what their model looks like (calibrated or not). You should be able to ask them if you returned every CVE you have a signature for, what percentage of them are high or critical? 
  • On average, how long does it take from CVE release to signature generated? Time-based components impact vendors and customers. And since the NVD itself has been slowing down due to the sheer number of vulns it houses, understanding a vendor’s median delay is important to adjust operating assumptions.
  • What is the efficiency of your model at 20% coverage? 50%? Consistently drill down into what efficiency looks like at different coverage levels so you can make an informed decision about risk tolerance. Every RBVM vendor should be able to provide the efficiency they’ve cut over the past year. If a vendor doesn’t disclose those numbers, either their efficiency isn’t improving or they’re just not thinking about it. Either way, it’s not a good sign.
  • What is your distribution of the volume of success exploitations? How many years of outcome data do you use? How often is it updated? When you’re considering a data stream that will help you make decisions about risk, vulnerability management, and resource allocation, these are valid questions to pose. A reliable RBVM vendor should be able to answer them by quantifying exploitation data. 
  • Is your model calibrated? What percentage of vulnerabilities scored 50/100 (or 5/10) are observed to be exploited? No model is perfect (ours included), but vendors should constantly question the calibration of their model. If a model is calibrated, vendors should be able to deliver retroactive data about how it’s performed over time. An uncalibrated and biased model has downstream consequences for your operations. If you’re stuck using one of those solutions, you’ll need to know about these ahead of time to tweak your remediation SLAs accordingly. 

Make measured, data-driven risk decisions

Vulnerability and risk management boils down to making decisions about prioritization, resource allocation, risk tolerance, etc. Yet as scanners increasingly become a commodity offering and fewer vulnerability data tools require authenticated scanning, the need for data-driven and measured decisions becomes paramount. 

That’s not just the approach you want to take with RBVM — it should be the approach you take in choosing an RBVM vendor.  Your search for the vulnerability management vendors you want to partner with should be thorough and exhaustive. Settling for anything less may leave you in the dark when it really matters. 

For more details on how to measure vendor performance, including a deeper dive into coverage and efficiency, a breakdown and comparison of popular RBVM platforms, and all the charts you could ever hope for, check out this webinar hosted by data science wunderkind Michael Roytman: 7 Questions Every CISO Should Ask Vulnerability Management Vendors.

Share with Your Network

Read the Latest Content

Threat Intelligence

18+ Threat Intel Feeds Power Modern Vulnerability Management

One question we often get from customers or prospective clients is why we use 18+ threat feeds and exploit intelligence feeds? Seems excessive, right? Why not just 4 or 5 feeds? Or 10? Why do we use more than 18 threat feeds and exploit intelligence feeds to power the Kenna.VM modern vulnerability management platform?  The…

READ MORE
Data Science

Ask Us About Our Data Science

What’s in a buzzword, like data science? A lot of resentment, for sure, but also a chance to explain.  A buzzword is a word or phrase, new or already existing, that becomes very popular for a period of time. Buzzwords often derive from technical terms. Yet through fashionable use, the original technical meaning disappears, and…

READ MORE
Risk-Based Vulnerability Management

What is Modern Vulnerability Management?

Modern vulnerability management is an orderly, systematic, and data-driven approach to enterprise vulnerability management. It leverages full visibility into a technology stack to target the riskiest vulnerabilities, enabling companies to adhere to designated SLA’s, respond to threats rapidly, and have meaningful discussions about organizational risk tolerance. Got that? Let’s unpack it.  To understand what modern…

READ MORE
Sign up to get the latest updates
FacebookLinkedInTwitterYouTube

© 2021 Kenna Security. All Rights Reserved. Privacy Policy.