When a CVE affects a container in production, how soon do we find out, and how long does it take to ship a fix? Scan each image before deployment and keep scanning it while it’s running. New vulnerabilities can come to light weeks after a release.
A hardened base image leaves fewer packages to track and patch. When a fix is ready, a separate release path and CI tests that use production startup settings help us get the patched image out sooner.
1. Start with a Red Hat Hardened Image
When Red Hat offers an image that fits your service, it’s a good default. Red Hat keeps the base small by leaving out tools such as shells and package managers, and tracks upstream vulnerabilities so it can ship fixes for the packages it includes. You get fewer packages to watch and a maintained base to rebuild from when a patch is available. Red Hat Hardened Images overview
Pick an image that has what your service needs to run. Keep build tools and other packages you only need during the build in a separate stage. That leaves fewer packages to track and patch in production.
Check the base image’s software bill of materials (SBOM) and CVE report. Then scan the finished application image for vulnerabilities in the dependencies you added. Red Hat Hardened Images catalog · Red Hat image documentation
2. Scan images before and after deployment
Before each production deploy, scan the exact image you’re about to ship. A new feature or a dependency update can bring in a vulnerable package. Check the new image and the one it’ll replace against the same vulnerability data so you can see what the change introduced. If the change brings in a known vulnerability, stop the rollout.
An image can pass its scan on Monday. By Friday, a vulnerability in one of its packages may be public. Keep a record of the image digest each service is running and rescan those images automatically as new vulnerability data comes in. Send new findings to the team that owns the service so they can decide what to patch first.
3. Give security patches their own release path
A patched image in the registry does little for the service that’s still running the vulnerable one. Make it possible to ship a security fix while the next feature release is still in progress.
- Make sure someone owns the first step. They check which deployed images are affected and decide what to fix first based on severity, how easily the flaw can be exploited, and how exposed the service is.
- Rebuild with the patched base image or dependency. Scan the new image and run the automated tests.
- Deploy the exact image that passed the tests, using its digest. Confirm it’s running in production and have a rollback ready in case the update breaks the service.
- Track the time from confirming that a CVE affects the service to deploying the fix. When a patch takes longer than expected, look at where it waited.
4. Use production startup settings in CI
A patched image can build successfully and still fail at startup. A library might be missing, or the user running the service might lose access to a file it needs.
Start the release image and its dependencies on an isolated network in CI. Use the image digest, entrypoint, arguments, runtime user, and other startup settings from the production deployment definition. Supply test credentials and endpoints through the same settings the service uses in production.
Once the service reports that it’s ready, call its API and exercise paths that depend on TLS or a database connection. Save the container logs before shutting everything down so you can diagnose failures. Run the same checks for every patch and promote the tested image by digest once they pass.
Testcontainers can start the containers, wait for readiness, and clean up afterward. If you build your own framework, it can use the Docker API to configure and run the containers directly. Testcontainers · docker-java API documentation
Working through a similar challenge?
Tell me what you’re up against. I can help.