Solving CI/CD Bottlenecks - Some Practical Tips
Tired of waiting for computers to do their stuff?
Let’s start this edition with a classic comic from xkcd: https://xkcd.com/303/
How Long Should Developers Wait for CI/CD Pipelines?
Continuous integration and deployment (CI/CD) has revolutionized software development over the past two decades. Despite its transformative benefits, CI/CD can become a source of frustration due to increased build times as more checks and automation are added. The process, once designed to streamline development, can slow to a crawl, with build times ballooning to 30 minutes or more.
Balancing Speed and Thoroughness
As teams incorporate various tests, validations, and security scans, the CI/CD pipelines grow increasingly complex and prone to failure. Murphy’s Law often ensures that the longest-running task is most likely to fail, further exacerbating delays.
Job Classification
To optimize CI/CD efficiency, it’s essential to categorize jobs:
Non-negotiable: These jobs must run before code merges or deployments. They are critical for ensuring the integrity and security of the build.
Optional: While beneficial, these jobs should not block the pipeline if they fail or are skipped. Their results can be logged and addressed later.
Parallelization and Efficiency
Running jobs in parallel can significantly reduce build times. If jobs depend on each other, breaking up these dependencies can help. Additionally, consider the setup and startup costs of each job; sometimes, bundling tasks into a single job can be more efficient.
Frequency of Jobs
Not all CI/CD jobs need to run on every push. Tasks like long-running fuzzing or vulnerability scans can be scheduled daily, separate from the main development workflow. This approach minimizes interruptions and reduces the likelihood of false positives that can frustrate the development team.
Conclusion
CI/CD automation is a powerful tool that, when used wisely, can greatly enhance development efficiency. However, overloading the pipeline with checks can slow progress. Teams can strike a balance that maximizes the benefits of CI/CD while minimizing its drawbacks by categorizing jobs, parallelizing tasks, and adjusting the frequency of long-running jobs. Use CI/CD wisely, and it becomes a near-magical tool for development; misuse it, and it becomes a handbrake on progress.
Interesting Links
James is shedding light on yet another security acronym. Application Security Posture Management (ASPM) is the latest Gartner-fueled buzzword to take over cybersecurity, but no one knows exactly what it is.
The Monsters in Your Build Cache
GitHub Actions cache poisoning can compromise build security, as detailed in Adnan Khan's blog. Attackers exploit GitHub's cache mechanism, injecting malicious code by predicting or forcing cache keys, leading to unauthorized access and lateral movement within workflows.
Building a (SOC 2 Compliant) GitOps CI/CD Pipeline with GitHub Actions
Mathieu shared their blueprint for making a simple and developer-friendly GitOps-based CI/CD pipeline built on GitHub Actions, designed for SOC 2 compliance. It is simple, has great developer experience, and is SOC 2 compliant.
Thank you for reading along. If you have feedback or questions, message me anytime at andy@occamslabs.com. If you want to be featured in the next issue or have suggestions, send them my way.