How Long Should Developers Wait for CI/CD Pipelines?

Continuous integration and deployment (CI/CD) has revolutionized software development over the past two decades. Despite its transformative benefits, CI/CD can become a source of frustration due to increased build times as more checks and automation are added. The process, once designed to streamline development, can slow to a crawl, with build times ballooning to 30 minutes or more.

Balancing Speed and Thoroughness

As teams incorporate various tests, validations, and security scans, the CI/CD pipelines grow increasingly complex and prone to failure. Murphy’s Law often ensures that the longest-running task is most likely to fail, further exacerbating delays.

Job Classification

To optimize CI/CD efficiency, it’s essential to categorize jobs:

• Non-negotiable: These jobs must run before code merges or deployments. They are critical for ensuring the integrity and security of the build.

• Optional: While beneficial, these jobs should not block the pipeline if they fail or are skipped. Their results can be logged and addressed later.

Parallelization and Efficiency

Running jobs in parallel can significantly reduce build times. If jobs depend on each other, breaking up these dependencies can help. Additionally, consider the setup and startup costs of each job; sometimes, bundling tasks into a single job can be more efficient.

Frequency of Jobs

Not all CI/CD jobs need to run on every push. Tasks like long-running fuzzing or vulnerability scans can be scheduled daily, separate from the main development workflow. This approach minimizes interruptions and reduces the likelihood of false positives that can frustrate the development team.

Conclusion

CI/CD automation is a powerful tool that, when used wisely, can greatly enhance development efficiency. However, overloading the pipeline with checks can slow progress. Teams can strike a balance that maximizes the benefits of CI/CD while minimizing its drawbacks by categorizing jobs, parallelizing tasks, and adjusting the frequency of long-running jobs. Use CI/CD wisely, and it becomes a near-magical tool for development; misuse it, and it becomes a handbrake on progress.

Interesting Links

WTF is ASPM?

James is shedding light on yet another security acronym. Application Security Posture Management (ASPM) is the latest Gartner-fueled buzzword to take over cybersecurity, but no one knows exactly what it is.

The Monsters in Your Build Cache

GitHub Actions cache poisoning can compromise build security, as detailed in Adnan Khan's blog. Attackers exploit GitHub's cache mechanism, injecting malicious code by predicting or forcing cache keys, leading to unauthorized access and lateral movement within workflows.

Building a (SOC 2 Compliant) GitOps CI/CD Pipeline with GitHub Actions

Mathieu shared their blueprint for making a simple and developer-friendly GitOps-based CI/CD pipeline built on GitHub Actions, designed for SOC 2 compliance. It is simple, has great developer experience, and is SOC 2 compliant.

Share

Thank you for reading along. If you have feedback or questions, message me anytime at andy@occamslabs.com. If you want to be featured in the next issue or have suggestions, send them my way.

Most data breaches happen because of accidentally leaked or stolen credentials, such as API keys or passwords.

According to IBM, the average cost per breach is over 4 million dollars

This can ruin a business in the blink of an eye. 

One of the most recent data breaches: 

Okta Data Breach (October 19, 2023): A service account stored within the system allowed unauthorized access to Okta’s customer support system.

The solution to this? Do not use long-lived access credentials.  This applies to humans and machines. 

I am running an experiment offering anyone 30 minutes of free security consultation. Just sign up for a spot or share with a friend.

For humans make sure sensitive actions or logins have a short-lived session time or require re-entering passwords and second-factor authentication.

Machines use short-lived access tokens or, even better, a form of zero trust based on machine identity. 

On a k8s cluster, you can use ServiceAccount tokens to identify your workloads. On most cloud providers, your systems can have an identity tied to your chosen role.

You can use systems like OIDC to establish trust between service providers and various SaaS.

In a past project, we combined Kubernetes Service Accounts and AWS Roles to get to almost zero long-lived access credentials. (Luckily, this application did not have a lot of external third parties)

In the end, we integrated:

• GitHub

• GitLab

• AWS access to internal components

• HashiCorp Vault as Credentials proxy/ temporary credentials 

Ultimately, this reduced the inventory of secrets that needed to be maintained tremendously, allowing us to rotate these more efficiently and regularly. 

Developers were also happy with it, as they didn't need to request secrets be stored for them in production vaults and could fully own their code.

Interesting Links

Harden EKS

Runs check against your EKS clusters to ensure they follow the EKS best practices.

Open-Source Security Automation

Security orchestration, automation, and response (SOAR) allows you to react to detected security events and automate much of the response. Similar to Zappier or IFTT, it is a no-code/low-code environment that enables security teams to do more automation and less manual work.

The Business of Protecting Individuals

This article takes a deep dive into why building a successful consumer-focused security company is challenging. In the end, it boils down to awareness, perceived threat, and the likelihood of actually being attacked or hacked.

Share

Working Together?

Thank you for reading along. If you have feedback or questions, message me anytime at andy@occamslabs.com. If you want to be featured in the next issue or have some suggestions send them my way.

While the post gave me the feeling of an old man yelling at clouds, there was something to it that made me think.

Many of these abstractions, libraries, frameworks, and tools allowed us to build more and cheaper. They helped us avoid constantly reinventing the wheel and instead have stable and well-engineered building blocks.
This made the landscape much more complex and difficult to understand. Many different domains and tools are mingled into a single project—have you seen your average NodeJS project? Expecting every engineer to be on top of all this is unreasonable, and the need to augment your working process is vital.

From a security perspective, this looks vastly different. Using generated code from AI without checking it can be dangerous. Even code on Stack Overflow has many vulnerabilities. This is being battled by AI to fix detected vulnerabilities automatically.

Will this lead to engineers becoming dumber over time? Or do they need to be better integrated into the company's business, security, and other aspects to orchestrate the development and generation of new products and software?

Interesting Links

Found Means Fixed

GitHub's latest innovation, code scanning autofix, is now in public beta for GitHub Advanced Security customers and promises a monumental shift in handling code vulnerabilities. Leveraging the power of GitHub Copilot and CodeQL, it offers an automated solution that addresses over two-thirds of alerts with minimal manual intervention. This tool speeds up the remediation process and significantly cuts down on security debt by enabling developers to fix vulnerabilities as they occur.

Lean Software Development

The article delves into the complexities and pitfalls of modern software development, highlighting how excessive reliance on external libraries and bloated code bases compromise software security and efficiency. He critiques the industry's approach to software development, emphasizing the risks posed by the sheer volume of code and dependencies. The text advocates for a shift towards leaner, more secure software practices and discusses regulatory efforts aimed at improving software quality.

Node.js Secure Coding: Defending Against Command Injection Vulnerabilities

My friend and long-time personal hero published a book on writing NodeJS Defending Against Command Injection Vulnerabilities. This book is innovative in teaching secure coding, using real-world CVE vulnerabilities in popular open-source npm packages. Through hands-on exercises and code review, you'll learn how to avoid common security pitfalls and adopt a security-first mindset. It's worth getting when you are a backend JS developer or security Engineer.

Kubernetes 1.30 a security overview

Kubernetes 1.30 introduces significant security enhancements and improvements in the developer experience. This update focuses on strengthening secrets management, node and cluster management, and data security. It features advancements like improved handling of secret images, reduction of secret-based service account tokens, and bound service account token improvements. Additionally, Kubernetes 1.30 offers new measures to safeguard data integrity within volumes and enhances container isolation and authorization controls. This release represents a comprehensive effort to bolster the security and usability of Kubernetes, reflecting the community's commitment to advancing the platform's capabilities.

Share

Working Together?

Thank you for reading along. If you have feedback or questions, message me anytime at andy@occamslabs.com. If you want to work together, here are a few different ways

I can help you:

• Security audit of your systems

• Improving the security of your current systems

• Designing secure systems from the ground up

We have built dogmatic approaches, software, and processes around the topic of software dependencies.

We built whole companies and open-source automation chains around this problem. To handle what is essential dependency creep and over-use of libraries. Some applications install hundreds to thousands of libraries in various versions. Many times, only a very small amount of the library's functionality is used. The rest is just stuff that isn’t needed. But now, we must manage, update, and review if a scanner finds something. 

How did we end up here?

Access to libraries and other people's codes has never been easier.  There are multiple projects solving the same problem using a different approach or style. Something for everyone and anyone's needs.

It has never been easier to build and release new software products to the world.  This is fantastic and enables people to create better lives for themselves and their families. Creating and maintaining open-source libraries has also become part of cultivating your resume as a developer. 

But not all is great!

Very often, a small functionality is required to solve a problem. A quick search reveals a library. We add the library to our dependency management. The code snippet in the README.md solved the problem, and we moved on. Now, we have an extra universe of functionality, dependencies, and potential problems.

What could have been a different way?

Design and create the functionality yourself? Get inspired by the open-source project or piece of code on StackOverflow on how this could be implemented and write it ourselves. Is it worth it?

It depends on the dependency.

Software dependencies can help to create new products and functionality quickly and easily. But they come with many unknown and hidden problems we have to account for.

Being more in control of your code and product reduces the risks of dependencies disappearing, being compromised by malicious actors, or being exposed to security vulnerabilities in your dependency tree.

Dependencies do have their benefits.

Using popular open-source libraries helps to have potentially more secure code. Recreating your own authentication or cryptography systems might not be a great idea. With many eyes on the codebase, problems are quickly identified and fixed. The best is to operate in between. 

Choose dependencies carefully and review how much extra is coming in.  Consider building smaller functions yourself and base them on existing libraries. For libraries that provide more functionality or are core to the product, make sure to vet them correctly and follow updates closely. Wrap them where possible to have better interfaces to control them.

Reducing the dependencies makes it much easier to keep them up to date and ultimately reduces the risk of supply chain attacks.

Link Of The Week

GitHubs approach to delivering availability, security, and accessibility

Being a massive development organization, GitHub developed internal tooling to stay on top of fundamental and important traits of any application. This blog post outlines a good way to automate the metadata to be on top of the overall picture.

Share

Working Together?

Thank you for reading along. If you have feedback or questions, message me anytime at andy@occamslabs.com. If you want to work together, here are a few different ways

I can help you:

• Security audit of your systems

• Improving the security of your current systems

• Designing secure systems from the ground up

Thank you for being here. Please subscribe if you’ve found your way over but are not yet subscribed.

Subscribe now

Link-List

Software Supply Chain Attacks rise 742%

The 8th Sonatype report on Software Supply Chain Security is an interesting read. Most vulnerabilities come through transitive dependencies. Dependencies are a big risk landscape that is slowly being improved. The attackers have almost endless opportunities to be a step ahead. Could a vetted Open Source mirror be a solution to this?

Supply-chain Levels for Software Artifacts, or SLSA

Brought up in a recent chat on LinkedIn again. It is a great specification on how to prevent and detect attacks. While it’s a great specification, it can get complex fast. Implementation and maintenance need some manpower. As always, with security, it comes in layers, and the work never stops.

When MFA isn't actually MFA

Some retool employees recently got tricked into giving up their second factor to an attacker. A very well-executed phishing campaign tricked employees into clicking on a link. The attacker then would call them to continue the scam. It shows that even if you have multiple layers of protection in place, you are not 100% safe.

Caught Developers

(scroll down for a summary)

The Attack

Phishing attacks are getting more sophisticated by the day. Their target audience is changing every day. In the past, high-profile individuals were targeted using CXO attacks.
These days, the scope has broadened and is targeting the whole organization.

It is not just a rushed email from your CEO with links to a website that tries to steal your login credentials or make you download malicious software. 

Attackers are getting way more creative to target people and who they are targeting.

Very recently, I encountered an interesting approach to phish a Developer.

The Developer was doing freelance work on the side, which was nothing uncommon. They got a request on a platform to help someone with a problem in a code base. After reviewing and accepting the job, they got access to the repository with instructions on getting it going and what error they would experience.

The developer went through the Readme, cloned the repo, installed the dependencies, and ran the setup commands. Nothing out of the ordinary and a very common approach. The developer ran into an error, tried a few things to fix it, and eventually dug deeper.

That's when they discovered some obfuscated code that would run upon setup. If the developer didn’t use the latest version of the Language, they probably wouldn’t have noticed.

As they dug deeper into the codebase, they discovered it contained Malware. It would connect to a remote command and control (C2) server and upload the clipboard, browser data, and a lot more. I will go into more detail later.

The developer immediately took the machine off the network, informed the security team and changed all of their passwords. The blast radius on this was quite big. All credentials this person had access to needed to be rotated. All direct and adjacent systems must be checked for access, abnormal behavior, or anything unusual. It took the team significant time and work to cover all grounds. Which is quite normal with the complexity of things we tend to build.

The Reveal

The malicious actor found a very good way to target developers effectively. So effectively that they are operating at a 90% + success ratio. The attackers can make sure the code is run in the best possible environment without needing to scout. Essentially, they can write a full profile of what is required, and their victims come their way 🤯.

There is no need for fancy zero-day exploits or relying on other weaknesses within the target system. They can even get full root access by asking for it. Everything happens voluntarily and in plain sight.

The Shield

How do you protect yourself, your team, and your company from this kind of attack?

Education is the most effective and best prevention. Security always starts with the people. Technology is there to assist and aid them. A person not educated in or willing to follow security principles will not be saved by tooling (for the most part). If you and your teams don’t know how to identify and spot these kinds of threats, it is hard to protect yourself. 

How could this have been avoided? 

First, if something happens, ensure people can bring up when they think they got targeted or even hit by an attack. Do not push them into keeping it a secret by punishing them for wrongdoing. Be kind and empathic; don’t patronize. IT Security is intimidating and scary. Most people do-not like to make mistakes. Even fewer want to have to report that they made a mistake.

Have clear policies in place. What is allowed and what is not. Is it okay to do freelance work on the side? Can they do it on their workplace computer, or is this a strict no-go? All of this depends on the company, business, the risk appetite, and the culture. There is no one-size-fits-all set of policies.

It also helps to have some (technical) guardrails in place that protect, prevent, or reduce the impact:

• Ensure no admin access to the computers without passwords

• Use managed browser profiles to ensure passwords are not stored etc.

• Use single sign-on wherever possible, ensure everyone is using a password manager, and the second factor is not managed in the same password manager.

• Do not have permanent access to any live system, especially not with privileged accounts.

• Keep any application secret in a dedicated secret store like HashiCorp Vault etc. 

• Have a good inventory of your secrets. This allows for faster rotation when required.

• If possible, have some intrusion detection and mobile device management systems in place

The Malware

What did the malware try to do?

In this case, it was a 3 part malware. The first part was a JavaScript script hiding in a test runner. It was obfuscated/minimized and hard to reverse engineer. From what we identified, it tried to contact a command and control server to download further instructions and invoke the second and third parts of the malware written in Python.

The second script would:

• Grab the contents of the clipboard.

• Get the browser history, sessions, and stored data (form and passwords).

• Look for password files, SSH keys, keyrings, and all kinds of files.

And upload all these files to a C2 server.

The third script would establish itself within the system for later use. All 3 scripts talked to different C2 servers.

The codebase was based on an open-source tooling that was modified. Variations of this attack can be found on GitHub in plain sight.

The TL;DR

A developer got tricked into cloning some code that included malware to steal credentials, crypto wallets, and more. They did the right thing to take the machine offline and inform their security team. The most important thing to prevent attacks like this to educate and teach your team, set out policies to prevent them, and put the right tooling into place to do damage control.

Working Together?

Thank you for reading along. If you have feedback or questions, message me any time andy@occamslabs.com. If you want to work together, here are a few different ways I can help you:

• Security audit of your systems

• Improving the security of your current systems

• Designing secure systems from the ground up

  

Share Secure All Software

Thank you for being here. Please subscribe if you’ve found your way over by some miracle but are not yet subscribed.

Subscribe now

Link-List

Methods to Backdoor an AWS Account

Six methods to get a backdoor into your cloud environment. From the most obvious way of having a pair of access keys and secrets to a more sophisticated approach of hiding the backdoor in the EC2 metadata service.

Building Smaller Rootless Non-Shell Docker Images

We are all building containers all the time. Mostly, we are focused on getting the applications running to enable business for the company. Too often, we don’t try to build the smallest, most secure image possible.

How Malicious Code Can Sneak into Your GitHub Actions Workflows

Tags are a double-edged sword. While very useful for communicating between humans, they allow for hiding bad intentions after you think you have done your due diligence.

You are using the latest tag!

(scroll down for a 5-sentence summary)

You are using the latest tag. We have all been there, and this is the most convenient way. There is no need to update anything. Just restart the pod. Building and pushing the image is also a breeze, with absolutely zero logic needed. This might be okay for a continuously rolled-out development cluster. You shouldn’t use this anywhere else, though. How do you know exactly what is deployed at the moment? How do you do a rollback? Essentially, you are not in control; you pull whatever is there at any time. There is a time and a place for using this, and your deployments aren’t one of them. Good luck, you will need it.

So, you are using image tags like v1.2.3. This is great for humans to know what we are talking about. And it is better than using the latest tag. But it's still wrong. Tags are mere bookmarks of changes at a certain point in time. Anyone with access can move around them, and they are not immutable. With proper tags, you can have better conversations about deployment issues. You can do more efficient rollbacks for most parts. You still can not trust that what you are deploying right now is the same as when you deployed it the first time. This is a big risk, especially for third-party services.

What is the correct way to deploy container images in Kubernetes?

You have to use the SHA256 of the images for enhanced security and accuracy. The latest tags, or any other tag, are mutable. They are a bookmark to a certain SHA256. The only way to be certain about what you are deploying is by using the cryptographic hash. The SHA256 is the cryptographic digest of the image with all layers included. 

Bonus points if the image is signed by docker content trust or cosign.

The easiest way to get the hash of any image: 

docker inspect --format='{{index .RepoDigests 0}}' $IMAGE

Should you use the hashes also for deployments of third-party services? This is particularly important. You don’t control the third-party repository, and you want to maximize the control you have. Changing the deployed version to the deployed hash will give you confidence.

Summary

When doing Kubernetes image deployments, using tags like "latest" or version numbers such as "v1.2.3" can pose problems related to version control and security. To address this, consider using the SHA256 hash of images for deployment. This method ensures immutability and accuracy, and I recommend signing images using tools like Docker Content Trust or Cosign to enhance security. This approach is especially valuable when dealing with third-party services, granting you better control and confidence in your Kubernetes deployments.

Working Together?

Thank you for reading along. If you have feedback or questions, message me any time andy@occamslabs.com. If you want to work together, here are a few different ways I can help you:

• Security audit of your systems

• Improving the security of your current systems

• Designing secure systems from the ground up

Share Secure All Software

Thank you for being here. If you’ve found your way over by some miracle but are not yet subscribed, please subscribe.

Subscribe now

Links

Kubernetes Security Ultimate Checklist: Cloud Native Security Basics Part VI

This blog post provides a good overview checklist of security best practices for Kubernetes. It covers many topics: authentication, authorization, networking, secrets management, and auditing.

The Emergent Cloud Security Toolchain for CI/CD

I recently found this talk by James Wickett again in some notes. Although it is from 2018, most of it holds up very well. The principle of making security part of your toolchain in every step still hasn’t seen the wide adoption I was hoping for.

The Difference Between SCA and Supply Chain Security

Software Composition Analysis (SCA) identifies and assesses the security risks of third-party components used in software. Supply Chain Security (SCS) is a broader concept encompassing all aspects of securing the software supply chain, from development to deployment. A great article by the fantastic Tanya Janca.

Personal Privacy & Security for CISOs

This one is a bit out of the usual kind of links I share. However I felt important to put the spotlight on personal security. This is not only relevant for CISOs, but also for anyone working online.

Share

How to Secure K8s without a cruft of tools?

Authentication and Authorization

In the world of Kubernetes, authentication and authorization form the cornerstone of security. Authentication verifies the identities of users and components, while authorization determines their level of access.

Authentication Mechanisms

Exploring authentication mechanisms such as client certificates and service accounts is vital. These mechanisms establish trust and ensure only authorized entities interact with your cluster.

Role-Based Access Control (RBAC)

RBAC allows you to define fine-grained access policies. Learn how to assign roles and permissions to users and services, mitigating the risk of unauthorized actions.

Network Policies

Kubernetes network policies offer a powerful way to control traffic flow between pods and nodes. By implementing proper network policies, you can curtail unnecessary communication and enhance the security of your clusters.

Image Security

Securing container images is non-negotiable. Learn why using trusted, signed images is essential and how image scanning tools can help you identify vulnerabilities before deployment.

Best Practices for Kubernetes Security

Secure Cluster Setup

A solid security foundation begins with a secure cluster setup. We'll guide you through configuring your Kubernetes securely to prevent initial vulnerabilities. Making the control pane private is a very important first step. Kubernetes APIs and etcd should never be publicly exposed to the internet. They should be in a private network only accessible using a VPN.

Node Security

Your worker nodes are the heart of your cluster. Ensure regular updates to minimize attack surfaces and leverage tools like kube-bench. The nodes should not be directly exposed to the internet, each application is made available through a load balancer to hide the IP addresses of the nodes. Ideally, all traffic is encrypted using TLS/HTTPS.

Use minimal operating systems. It is even better if the file system is read-only as much as possible to reduce potential malware installed. Implement Node Hardening and further reduce any attack surface. This is extra important when managing and operating your kubernetes worker nodes. Regularly update the operation systems and rotate old nodes.

Pod Security

Each pod has its own security context. Use security policies to ensure containers run with the right permissions and restrictions.

Avoid running as a root user or any other highly privileged user. Ensure any secrets used by the pods are limited in scope and impact. Avoid hardcoding sensitive information like passwords and API tokens directly in your pod configurations. Instead, use Kubernetes Secrets to store and manage such information securely.

Configure your pods to use read-only filesystems whenever possible. This prevents potential attackers from modifying container files, which can help mitigate certain security breaches. Resource limits and requests can prevent resource exhaustion and abuse. Define appropriate resource limits for your containers to avoid consuming excessive CPU, memory, or other resources.

Network policies control communication between pods and are instrumental in segmenting your application. By allowing only necessary communication and denying the rest, you reduce the attack surface and limit the lateral movement of threats. Configure your pods to use read-only filesystems whenever possible. This prevents potential attackers from modifying container files, which can help mitigate certain security breaches.

Secure Communication

Encrypted communication is crucial outside and within your cluster. Using TLS for communication between your services, nodes, and external partners is as important as ensuring secure communication from the outside in. Most Cloud Service Providers offer certificate services that can be used with kubernetes. Another free option is using Let's Encrypt with Certmanager.

Not only your workloads and users should be using secure communication. Also, your interactions with your cluster control pane must be secured anytime. Try to aim for 100% encryption in transit.

Monitoring and Auditing

Monitoring can be its own pandora’s box. Choose a system that works for you, can store data longer than 30 days, and has the ability to generate alerts based on logs. Define some key metrics that are the baseline. Then, define alerts that are outside of regular operations. For security monitoring, we are interested in error rates and abnormalities. Part of your monitoring should include logging. Building metrics based on logs gives you correlation superpowers.

Audit logging must be enabled for any Kubernetes System; most Cloud Providers have it tied up with their own in-house built logging solution. This is sometimes extra to your current logging stack and can be an issue. Your applications should also have audit logs, that clearly document who is doing what and where. All of these audit logs must be stored securely and tamperproof away from the rest of the systems.

Need help? Let’s talk: andy@occamslabs.com.

Share

Thank you

Thank you for reading along. If you have feedback or questions, message me andy@occamslabs.com. I would appreciate it if you shared it with people that can benefit from the newsletter.

Subscribe now

Share Secure All Software

Thank you for being here. If you’ve found your way over by some miracle but are not yet subscribed, let me help you with that.

Subscribe now

Kubernetes and the Supply Chain 

Running your workloads on Kubernetes(k8s) is becoming a de facto standard for many companies building software. The usual steps involve building software, running tests, building a container, hopefully doing some security scanning in the process, pushing to the registry, and deploying.

In general, there are two different risks with k8s images deployed:

• Vulnerabilities in Operating System packages and the software packages

• Tampered or deliberately deployed images.

Many of the workloads running on your cluster didn’t pass through your security tooling in your CI/CD pipelines. Tools directly installed using helm with publicly available container images.

These usually have a high degree of vulnerabilities that are hidden away from you.

Also, software that is not deployed all the time can amass vulnerabilities over time. While code doesn’t change, people keep finding vulnerabilities in existing software.

There are four stages of kubernetes supply chain security:

1. CI/CD: scan your images and dependencies

2. Container registry scanning

3. Workload scanning while running and deployed in the cluster

4. Admission control only allows images based on an allow list and valid signatures. 

CI/CD image scanning

Tools like Trivy can detect vulnerabilities in the container image. Next to the operating system issues, it can also detect problems with installed libraries of your software and secrets might have accidentally been added.

This will catch issues at the time of the build and prevent potentially shipping new vulnerabilities.

Container registry scanning

Registries like the ones offered by Google and AWS have the ability to scan existing images. This can make sense, but it usually is a dump yard that many people never look into.

If set up properly with alerting this can be useful. However, in my experience these are not on the radar of many teams.

Workload Scanning 

Scanning and alerting on your currently running images helps you understand your current problems. It also catches container images that did not pass through your regular CI/CD pipelines or your registry.

Tools like Trivy operator and Clair can support this. They are regularly scanning your running containers and will emit alerts for you. This adds extra load to your system but allows for the most accurate representation of vulnerabilities in your clusters.

Admission control

By controlling what can run on your cluster, you enhance an extra level of security.

Having an allowed list of images will reduce attack vectors by quite a bit. If, on top if this, you verify the signatures of the images you will get even more peace of mind. Docker Content Trust and Cosign by sigstore are currently the best supported. 

What do you think? Let me know at andy@occamslabs.com.

Share

Links

CNAPPGoat: The Multicloud Open-Source Tool for Deploying Vulnerable-by-Design Cloud Resources

A multi-level approach to get a better understanding of cloud security issues. It is also useful for testing your security tooling to spot various misconfigurations.

kubefuzz - admission controller fuzzing

Kubefuzz is a generative and mutative fuzzer for kubernetes admission controller chains. It can be used to uncover unexpected behavior in complex admission controller setups.

6 Best Practices to Ensure Kubernetes Security Meets Compliance Regulations

Security must be precise enough to meet compliance requirements without impeding DevOps and developer productivity. A fairly high level overview of some useful practices.

Krane - Kubernetes RBAC Analysis made Easy

Krane is a simple Kubernetes RBAC static analysis tool. It identifies potential security risks in K8s RBAC design and makes suggestions on how to mitigate them. Krane dashboard presents current RBAC security posture and lets you navigate through its definition

.

Thank you

Thank you for reading along. If you have feedback or questions, message me: andy@occamslabs.com. I would appreciate it if you shared it with people that can benefit from the newsletter.

Subscribe now

Share Secure All Software

Subscribe now

Vulnerability vs. Supply Chain Attack

What is the difference between them?

A vulnerability is usually an unintended software bug that opens an attack angle for a malicious actor. The good thing about an open-source project is, that the bigger the project, the more eyes are on it. The more likely these vulnerabilities will surface and be fixed fast. In smaller or dormant projects, this is usually not the case.

A supply chain attack is a deliberately created attack against a language ecosystem or certain users. 

Recently, very targeted attacks have been seen against the fintech sector. 

The attackers would use various different techniques to spread the malicious code. From simple typo squatting - where you publish a new package with a common spelling mistake - to sophisticated attacks - where social media profiles are created to build trust with the community and victims - the range is wide. 

How can you protect your applications from either?

As discussed last week, lockfiles are an essential part of detecting attacks. There is more that can be done. A variety of open-source and commercial projects review the required dependencies. Then they compare the versions with various data sources to identify if you have a vulnerability in your dependencies.

Make these tools part of your workflow. Add them:

• your Local development

• the CI/CD pipelines for every merge request,

• running in your live systems to detect new vulnerabilities that are actively in use.

These exact tools also help you identify known vulnerabilities in your dependencies.

The moment you enable tools like this, the chance is high to get a lot of findings as a result.

How to triage vulnerable dependencies

Scanning for vulnerabilities can lead to many results. How to tame the findings and get actionable?

Step one:

• Filter all findings by the highest severity 

• Review each finding with the questions in step two

Step two, review each finding:

• What is the potential impact

• Can it be used by external attackers?

• How complicated is it to be executed?

• Is the code in use? (not easy sometimes)

Step 3:

• Include the version that is secure

• Reduce the severity of the finding if needed.

• Add an expectation until when this should be fixed (internal SLA)

• Create a ticket with severity and questions/answers of the assessment

What do you think? Let me know at andy@occamslabs.com.

Share

LinkDump

Building a Detection-as-Code pipeline (part one, part two)

A two-part (part one, part two) in-depth walkthrough of how to build and integrate event detection from external secrets with GitHub automation by David French. Very interesting approach to combining various SaaS tools with some custom logic.

The Zero CVE Challenge: Can official Docker Hub images pass the test?

If you build your images on top of official docker hub images, are you getting security vulnerabilities with it that you could avoid? It seems that updating OS packages in popular Docker hub images reduces vulnerability counts, on average, by only 5.5%. It might be better to build your own images sometimes

GitHub Actions Goat

A deliberately Vulnerable GitHub Actions CI/CD Environment. This is a good resource to make yourself familiar with some of the biggest problems in CI/CD pipelines and how to spot them.

Thank you

Thank you for reading along. If you have feedback or questions, message me: andy@occamslabs.com. I would appreciate it if you shared it with people that can benefit from the newsletter.

Subscribe now

Share Secure All Software

Subscribe now

Supply and Lockfiles

A supply chain attack is when one or more of the dependencies in your application have been compromised, and some “bad code “is running on your systems and applications that is intended to harm you. These kinds of attacks are becoming more common and more sophisticated.

Most programming environments that offer a form of dependency manager have a locking mechanism in place. The manager allows to specify what kind of version should is used. And stores the version along with a checksum/cryptographic hash in a sometimes-called lockfile.

These lockfiles are essential to keeping applications safe from supply chain attacks. If the remote code changes but the same version number is used, you will get a checksum mismatch.

No matter if you update dependencies by hand or use tools like Dependabot or Renovate. You must review the changes to ensure the functionality is as you expect and the new code does not introduce any malware or obvious bad intentions. Simply updating and not checking is dangerous and circumvents the whole system.

How do you spot bad” intentions? Some ideas: 

• the change introduces a new dependency,

• changes to pre/post-install hooks of the dependency manager, 

• introduces cryptic code that is unclear

• downloads stuff from the internet all of a sudden."

How often should you update your dependencies?

There are two strong opposing approaches to updating software and dependencies. 

The first approach you constantly update everything to stay on the bleeding edge. The second one is you update only when there is a necessity. Otherwise, never touch it.

What is the best way? 

It depends on the environment and the company, how many people how many systems and moving parts they have. A good rule of thumb is to keep things up to date "enough," so it is easy to get it done during an emergency security update. You don't need to spend days or weeks fixing stuff.

What do you think? Let me know at andy@occamslabs.com.

Share

LinkDump

Open Source Supply Chain attack targeting banks

Malware authors pretended to be employees of banks. They even went that far to create fake LinkedIn profiles. The dependencies were hidden in plain sight using Azure’s CDN

kbom - A Kubernetes Bill of Materials

Allows you to create a JSON structured representation of what is running within your cluster. You can use this to build alerting into your security tooling when something changes that should not.

earlybird - a sensitive data scanner

American Express is maintaining an open-source scanner that is looking for clear text password violations, PII, outdated cryptography methods, key files, and more. Another piece of tooling that can be added to the CI/CD to help detect leaks early.

The Six Dumbest Ideas in Computer Security

A - quite on point - rant about computer security and some of the ideas driving it. It seems like every day, a new tool or toy is being released that claims to solve problem XYZ. After all, nobody got fired for hiring IBM or so.

Thank you

Thank you for reading along. If you have feedback or questions, message me: andy@occamslabs.com. I would appreciate it if you shared it with people that can benefit from the newsletter.

Subscribe now

Share Secure All Software

Welcome back! After some hiatus from writing and Twitter shutting down the old platform we will be back to regular schedule.

Thank you for continuing to read, let us dig right into it.

Who owns security?

Security used to be a thing you do at the end of a project or a release. Usually ceremonially crowned by the almighty pentest.

This has dramatically changed over the last few years. Today people talk a lot about security by design, shifting left, and other buzzy phrases.

Whose responsibility is security? Who owns it to design, build, and ship software securely? Unless you have a dedicated team, there is usually no clear ownership.

What if a security feature causes issues with the product design & usability?

How do you solve the issues of security negatively impacting onboarding and marketing numbers?

What if your security tooling is adding extra steps and delays for shipping code fast and furiously because you shifted all tooling as far left as possible?

Once security starts to impact the KPIs of individual business units and teams, they will start working around it and pushing it aside, and even silently discarding it.

I believe that security first and foremost starts with the people involved. And is everyone’s job to continuously work and improve on it.

Security in modern startups is a continuous conversation between design - engineering - marketing - business. There is no singular ownership.

Design wants secure features with great UI and UX. Engineering wants to ship code, fast easy, and secure. Marketing wants easy frictionless onboarding and uses security as a differentiator. Business wants to reduce business risk and build trust with clients and investors.

All parties have their business goals but also need to balance risk and reward. This means that leaders and teams must have an aligned security mindset and keep each other accountable.

When you build this kind of security culture, security tooling starts to become useful and not just in the way.

People won't be trying so hard to work around the tools and processes, but rather figure out how to improve them. They then can focus on their work and ship secure products with great features and high business impact.

Modern security is about shifting perspectives and defining ownership of security in daily processes.

Maybe Product Security is a new role in the future to bridge all of these in the future? What do you think, let me know at andy@occamslabs.com.

Share

Links

🔐 The Vault Policy Guide - link

Vault access control lists (ACL) can be confusing and are not always straightforward to handle. This deep-dive is valuable for anyone who wants to set up Vault, but also for folks who have it running and want to review their policies.

☸️ K8sGPT - Ultimate tool for kubernetes scanning - link

Kubernetes being pretty much everywhere security is a big problem here. Combine it with AI and it can be a great tool to tame the complicity it brings. While the tool is mainly designed to find configuration issues, it helps to detect security misconfigurations as well.

🥑 Google GUAC v0.1 Launched - link

Graph for Understanding Artifact Composition (GUAC) is a metadata analysis tool that aims to improve supply chain security. It looks like a promising project to combine many different sources and refine them into a simple-to-use API.

📝OWASP Top 10 for Large Language Model Applications - link

Artificial intelligence is here to stay. It will support our future work and might allow us to reach new levels of efficiency. With great power comes great responsibility and the first attacks have already been successful. The OWASP project released a first draft of the Top 10 security issues to watch out for.

🧠 Hallucinating AI as a security risk? - link

ChatGPT and other LLM systems sometimes make things up including names of opn source libraries . This could be used by an attacker to craft a vicious package and publish it under the made-up package name. While this is a bit hypothetical, it is not impossible at all.

Thank you

Thank you for reading along. If you have feedback or have some questions just send me a message: andy@occamslabs.com

Subscribe now

Share Secure All Software