While that is a great start, it’s important to remember that security is an ongoing process, not a one-time event. Those scans that passed in development pipelines only mean your applications were secured at the time you deployed them. What about a day, month, or year later?

In reality, not every application is built and deployed daily, and that can mean your perfectly secure Kubernetes cluster is actually full of misconfigurations and CVEs. That's why it's crucial to establish a robust security posture that includes ongoing vulnerability assessments.

See how your base images perform today

Securing each layer of your Kubernetes infrastructure

At Mondoo, we understand that securing each layer of your Kubernetes infrastructure, including the application images, is essential for a comprehensive security approach. That’s why we’ve developed a full-stack security approach to help you secure your infrastructure from development to retirement.

Our policy-as-code security approach and out-of-the-box security policies make it easy to ensure that your infrastructure is secure at every stage of development. We also provide you with the ability to scan images locally or in your CI/CD pipeline to identify common misconfigurations and outdated packages with CVEs. This allows you to address issues before they ever make it to production.

But we don’t stop there. Once workloads have been deployed, our Kubernetes Operator continuously scans the images as new CVE data is published. This allows you to identify new CVEs in previously deployed applications and rebuild containers to fix critical vulnerabilities.

The alarming reality of accumulating CVEs in container images

What happens to the security of the application containers running in production when teams deploy weekly, monthly, or even just once a year? At Mondoo, we were curious about this question too, so we tasked our security research team with investigating the issue.

Their findings were alarming. Container images accumulate a surprisingly large number of CVEs over time, which varies greatly depending on the kind of base image used. Traditional Linux distributions, such as debian and ubuntu, amass large numbers of CVEs over short periods of time, while container-optimized operating systems, such as alpine, see significantly fewer CVEs (but still critical CVEs after just a few months). Even container images for programming languages like Ruby or Python, which are based on Debian, have large numbers of CVEs over time.

These findings underscore the importance of ongoing vulnerability management for Kubernetes clusters. By continuously monitoring container images and remediating vulnerabilities, you can mitigate the risks associated with outdated or vulnerable images.

Tracking CVEs Over Time: A Deep Dive into Container Image Scanning on Docker Hub

At Mondoo, we wanted to get a better sense of just how many CVEs were accumulating in container images over time, so we pulled container image tags for some of the most popular base images on Docker Hub. We scanned each of those container image tags and recorded the results, cleaning them up to remove test releases and one-off tags.

However, we ran into a roadblock with Docker Hub’s API. Once a tag is overwritten, we can no longer poll each image that was released for that tag, which makes it impossible to track the security of common tags like ‘latest’ or ‘stable’. To work around this issue, we rely on version tags for projects over time. Fortunately, most projects are released on a fairly regular basis, which gives us good visibility into the security of the container.

Mitigating CVE risks in container images

After deployment, how long does it take for a CVE to become an issue? The answer is not straightforward and it depends on several factors. Our research found that many of the popular Docker Hub images had multiple CVEs in their current releases, while others only had CVEs in older releases that were several weeks old. As a user, you can reduce the number of CVEs over time in your application images by changing these base images. Let's take a closer look at the key takeaways from our research and explore how you can mitigate CVE risks.

The importance of choosing the right container image

One of the biggest factors that determines the number of CVEs that arise in container images over time is the type of container image used. Traditional base OS images such as Debian, amazonlinux, or Ubuntu tend to have more CVEs in a shorter period due to the number of packages they include. In contrast, smaller, container-optimized images like alpine have significantly fewer CVEs, with just 38% of those found in Debian, 23% in ubuntu, and only 8% in amazonlinux after 3 months. By choosing container-optimized images you can reduce the need to constantly rebuild and redeploy applications to keep CVEs out of your production infrastructure, making it an easy and effective security win.

Container Optimized Application Variants

In addition to using container optimized images for your own applications, it’s also important to choose these images for open source projects running in your infrastructure. Projects like nginx and ruby ship different variants of their application containers, and deploying the container optimized variants is a great way to reduce the number of vulnerabilities in these 3rd party applications.

The differences in Ruby variants was perhaps the most shocking finding. A standard ruby 2.6.6 container from 2021 has a whopping 401 CVEs, while the alpine variant of the same container has only 23.

Choosing slim container images for legacy or commercial applications

In situations where switching to fully stripped-down, container-optimized images is not possible, it is recommended to use slim container images as the base for your containers. Slim images are standard Linux distribution images, such as Debian or Ubuntu, but with fewer extra packages that can introduce potential vulnerabilities. While these container images require more effort to test and find missing packages for reinstallation, they result in more secure applications that don’t need to be rebuilt as often. This is particularly useful for commercial applications or legacy apps that were never meant to be containerized.

Updating base images: more than just pulling the latest

Unfortunately, relying solely on the latest base image doesn’t always resolve CVEs in your application containers. Many popular base images, such as debian and alpine, are constantly being rebuilt and retagged when new CVEs come out, While others like oraclelinux only get rebuilt when a new OS is released. As a result, applications built with the latest Oracle Linux base container may contain dozens of CVEs, including vulnerabilities in critical libraries such as OpenSSL. Even if you don’t directly use these containers, you may be using language containers such as openjdk or python that are built using these unpatched base images. To address this issue, it’s important to switch to base images that are continuously updated, like debian or alpine, and always update the base OS packages as part of your container build process. A simple `yum upgrade -y` or `apt-get update; apt-get upgrade -y` in your container image pipeline goes a long way towards securing your environment.

This mysql image includes Debian or Oracle based images. The latest Debian images have 1 CVE while the Oracle-based images have 25 CVEs out of the box.

Harness the power of CVE scanning for container security

After conducting extensive research, we have discovered that introducing CVE scanning at stages of the container lifecycle is crucial. Ensuring that only secure containers are promoted to production and that those images remain secure by scanning running container images in Kubernetes clusters is equally essential. Proper image scanning and alerting can enable teams to ensure that the workloads they deploy are secure every day.

If you’re looking for a comprehensive solution to scan your containers, sign up for a free Mondoo account today. Our platform enables you to scan docker images, private registries, running containers, or images in Kubernetes clusters. With Mondoo you can find images with critical CVEs, rebuild, and remediate vulnerabilities that let attackers in.