Secure All Software [#14] - Signing Containers

Container image tags are overly trusted and rarely validated. Cryptographic signing is available. Here are a few ways to do it.

Happy Friday, everyone 🎉. A long and eventful week is wrapping up. I dealt with some interesting issues, one related to Kubernetes and how to ensure the authenticity of the images running within.

Why Docker image tags are bad

When we build containers, we primarily control what's going on. Once pushed to the registry, it is considered stable and safe.

Docker tags are not as stable as we like to think they are.

Dependency integrity is a - more or less - solved problem for operating systems and library dependency managers. These usually use cryptographic hashes to verify what you wanted to download and that what you downloaded is the same.

In the container universe, we would rely on tags for years to decide what is installed in our clusters and systems. Ignoring that these tags can be recreated and something else could be installed without us knowing. Even when building our own images, integrity is not always guaranteed by just using a tag.

How do you improve this situation and mitigate the risk?

There are two ways to improve on this. First, we can rely on the sha256 of the image instead of the tag. The tag is for humans to read, but machines are okay with reading and comparing long hash strings. The sha256 of an image changes as soon as some of the content inside changes.

Whether for your in-house built images, or external ones, I recommend using the hash and not the tag. If you build on top of public images, rely on the SHA and not the tag to be in control. It makes updating things more inconvenient but guarantees you the authenticity of the image you build on top of.

On top of relying on the hash, signatures have been established to use cryptographic public/private-key methods to guarantee the authenticity of an image. After building an image and before sending it to the remote registry, the author would create a signature with the private key. The signature and the image are then pushed to the registry. It guarantees that what is downloaded is the same as what was uploaded.

At the moment, there are two different methods for signing images.

Out of the box, Docker has rolled out Docker Content Trust (DCT). It is s a built-in mechanism in Docker and allows for the verification of images. To use this in Kubernetes, you need to deploy a custom admission controller or use connaisseur.

The second option is cosign by sigstore. The basic concept is the same. However,, you can rely on cloud-native key management systems (KMS) like Hashicorp Vault, AWS KMS, and others.

Cosign offers a policy controller that can be deployed in Kubernetes. Connaisseur also supports cosign. Cosign can also be used to sign any other build and deployment artifacts.

Unfortunately, most publishers don't use either DCT or cosign to sign their images.

The way, for now, is to mirror the images you require, inspect them, and sign the ones you trust.

Enabling container signing for your images is easy and straightforward.

In the future, this will become more widespread and a standard.

What do you think? Let me know at andy@occamslabs.com.

Links

🔐 15 Kubectl Plugins for Security Engineers

Kubernetes has a good plugin ecosystem. Here are some useful tools day to day work with kubernetes. Especially interesting is the Cilium Plugin, which replaces the current container networking of k8s with a eBPF-based Networking layer for better observability and security.

☠️ Destroyed by Breach

Adrian Sanabria is curating a list of companies that went out of business due to “cyber security attacks”. A very interesting read. This could be eye-opening for CTOs and company owners to understand the impact of a breach.

👥 K9 Security - AWS IAM is a mess

Whenever dealing with AWS IAM, it’s a tricky walk. The system has great power and allows for super fine-grained access permissions. On the other hand, it is a complex beast to tame. My friend Stephen Kuenzli is building a tool to help to handle this complexity.

Thank you

Thank you for reading along. If you have feedback or questions, message me: andy@occamslabs.com. I would appreciate it if you shared it with people that can benefit from the newsletter.