One-fifth of the most-used Docker containers have at least one critical vulnerability
When CVE-2019-5021 was released on May 8, it made me wonder how widespread the issue of vulnerabilities in popular containers is. Businesses have increasingly come to rely on containers as an agile development tool, but because they are inert when not in use, security vendors have found them difficult to scan.
Earlier this year, an amazing new open source security tool called Trivy was released, which enabled container scanning in a DevOps environment. I built a tool that would use Trivy to check as many containers as possible to determine how vulnerable containers really are.
As a result of my research, I am happy to publicly launch VulnerableContainers.org, which pulls the top 1000 most popular containers from Docker Hub based on the number of downloads, and scans them for vulnerabilities.
It took nearly 10 hours to pull and scan the top 1000 containers as my tool is currently written, but I am working on rewriting this to include a caching layer and more logic that will allow me to run this script more efficiently.
From the first scan, I was amazed at the number of frequently used containers, some with over 100,000,000 pulls, that had hundreds of open CVEs. Many of the containers had at least one CVE labeled with a CVSS score that identified them as high risk. While this should be alarming, there are several factors that impact the risk a vulnerability poses to an organization that is not reflected in CVSS score, like the age of the vulnerability and how often attackers are using an exploit associated with it.
I added the Kenna Risk Score to this research, which correlates variables such as exploitability and attacker activity, to provide a more accurate picture of risk. Kenna’s analysis can cut both ways. Because it is designed to help overwhelmed cybersecurity teams predict and prioritize the risks they face, sometimes Kenna highlights threats as more seriously than the CVSS score suggests, and sometimes it downgrades threats. For context, the Kenna Risk Score is a number that spans from 0 to 1,000. Scores below 330 can be considered low risk, scores between 330 and 660 are considered medium risk and should be addressed promptly; scores above 660 are considered high risk and should be immediately remediated.
Our data suggests that many of the most popular containers are home to highly critical vulnerabilities. Over 60 percent of the top Docker files held a vulnerability that had a Kenna Risk Score above 330; and over 20 percent of the files contained at least one vulnerability that would be considered high risk under Kenna’s scoring model and should be addressed as soon as possible.
The oldest container on the list, abh1nav/cassandra, had over 1.5 million pulls. It was last updated in May of 2015 and has over 431 open vulnerabilities, the most critical having a Kenna Score of 720. Keyvanfatehi/sinopia has the distinction of having the most open CVEs with 2,004 and has over 1.7 million pulls.
After I completed the scanning, I did some math. The results looked like this:
- Average CVE’s per container: 176
- Median CVE’s per container: 37
- Average Kenna Score: 400
- This score is well within our median-risk range
- Average last update day: 10/23/2018.
- Median last update day: 4/17/2019.
- These containers have over 20,982,707,439(20.9 Billion) pulls
- The average container has 21,003,569 pulls
- The median container has 4,073,960 pulls
The results show that the Docker files typically containing the most vulnerabilities appear to be abandoned or are otherwise EOL’ed (end of life). If you are using these Docker files, this data should cause you to be concerned, especially when using these files on critical assets, but you shouldn’t necessarily boycott these files. The typical enterprise IT system houses millions of vulnerabilities. It’s not humanly possible to protect everything, but it’s very important to patch the vulnerabilities that matter.
Developers can protect their products by acknowledging the risks and taking basic precautions when running Docker files. Don’t blindly trust that these containers are secure, just because it is on Docker. Don’t run containers out of the box, as is. Set new containers to update when they run. Rebuild containers every few months. Build your own image from the docker file, and use smart access control.
I wanted to spend a little bit of time laying out all the tools I `glued` together here to make this possible.
Trivy: Trivy is at the heart of the project and an amazing open source container scanner that I have been impressed with since its release and really made this project possible.
Docker Hub API: The Docker Hub API has been really helpful. It is where I pulled the information for the last time a container was updated and how many pulls it has.
Kenna Risk Scoring: The Kenna Security Platform leverages 15+ exploit intelligence feeds, a knowledge base of 3+ billion (and growing) managed vulnerabilities, global attack telemetry, and predictive modeling to provide granular risk scoring and vulnerability prioritization. The Kenna model algorithmically determines risk scores for each unique vulnerability, and with asset criticality scores, determines an actionable risk score for each asset and group of assets that ranges from zero (no risk) to 1000 (highest risk).
For this analysis, we assume that each container has an internal asset score of 10 (the most critical), and is based on an analysis of the most critical CVE associated with each container to provide the Kenna Risk Score.
For more information on Kenna Risk Scoring, please visit: https://www.kennasecurity.com/the-science-behind-kenna/
CSV-to-HTML-Table: Derek Eder from Chicago released a simple CSV-to-HTML-Table template a few years ago that I love to use to display data of this type. He also runs ChiHackNight which I need to check out soon.
AWS System: I am running vulnerable containers on a T2.xlarge AWS image with 2TB of storage to hold all of the scanning data and the containers, which took up 550GBs alone.
Released Code (Soon): I am refactoring this over the next few days to increase the speed with a caching layer and make it clear which containers have vulnerabilities and which were not scanned so I am holding off on releasing this POC code as of right now.
Notes & What’s Next:
Set up the system to run this script nightly automatically.
Write logic that will break out containers that truly have zero CVEs and containers that Trivy can not scan yet.
Improve Coverage & Speed.
Build a “Test This Container” page.
Please feel free to reach out to me on twitter (@JGamblin) if you have any questions.