You are doing it wrong! Kubernetes Image Tags
You are doing it wrong! Kubernetes Image Tags
9 out of 10 deployments are doing it wrong, can you guess why?
Thank you for being here. Please subscribe if you’ve found your way over by some miracle but are not yet subscribed.
Link-List
Methods to Backdoor an AWS Account
Six methods to get a backdoor into your cloud environment. From the most obvious way of having a pair of access keys and secrets to a more sophisticated approach of hiding the backdoor in the EC2 metadata service.
Building Smaller Rootless Non-Shell Docker Images
We are all building containers all the time. Mostly, we are focused on getting the applications running to enable business for the company. Too often, we don’t try to build the smallest, most secure image possible.
How Malicious Code Can Sneak into Your GitHub Actions Workflows
Tags are a double-edged sword. While very useful for communicating between humans, they allow for hiding bad intentions after you think you have done your due diligence.
You are using the latest tag!
(scroll down for a 5-sentence summary)
You are using the latest tag. We have all been there, and this is the most convenient way. There is no need to update anything. Just restart the pod. Building and pushing the image is also a breeze, with absolutely zero logic needed. This might be okay for a continuously rolled-out development cluster. You shouldn’t use this anywhere else, though. How do you know exactly what is deployed at the moment? How do you do a rollback? Essentially, you are not in control; you pull whatever is there at any time. There is a time and a place for using this, and your deployments aren’t one of them. Good luck, you will need it.
So, you are using image tags like v1.2.3. This is great for humans to know what we are talking about. And it is better than using the latest tag. But it's still wrong. Tags are mere bookmarks of changes at a certain point in time. Anyone with access can move around them, and they are not immutable. With proper tags, you can have better conversations about deployment issues. You can do more efficient rollbacks for most parts. You still can not trust that what you are deploying right now is the same as when you deployed it the first time. This is a big risk, especially for third-party services.
What is the correct way to deploy container images in Kubernetes?
You have to use the SHA256 of the images for enhanced security and accuracy. The latest tags, or any other tag, are mutable. They are a bookmark to a certain SHA256. The only way to be certain about what you are deploying is by using the cryptographic hash. The SHA256 is the cryptographic digest of the image with all layers included.
Bonus points if the image is signed by docker content trust or cosign.
The easiest way to get the hash of any image:
docker inspect --format='{{index .RepoDigests 0}}' $IMAGE
Should you use the hashes also for deployments of third-party services? This is particularly important. You don’t control the third-party repository, and you want to maximize the control you have. Changing the deployed version to the deployed hash will give you confidence.
Summary
When doing Kubernetes image deployments, using tags like "latest" or version numbers such as "v1.2.3" can pose problems related to version control and security. To address this, consider using the SHA256 hash of images for deployment. This method ensures immutability and accuracy, and I recommend signing images using tools like Docker Content Trust or Cosign to enhance security. This approach is especially valuable when dealing with third-party services, granting you better control and confidence in your Kubernetes deployments.
Working Together?
Thank you for reading along. If you have feedback or questions, message me any time andy@occamslabs.com. If you want to work together, here are a few different ways I can help you:
Security audit of your systems
Improving the security of your current systems
Designing secure systems from the ground up
