This is a summary of my research of modern CI/CD practices while working on Reliza Hub. This list is rather opinionated but I try to provide explanations why I hold specific opinions. Finally, I’m making this an ordered list, but it’s not actually sorted by importance.
So, let’s start:
Table of Contents
1. Separate CI and CD – Use asynchronous pipelines
Too many times I see convoluted multistage CI+CD pipelines, all in one, with bunch of tests and approvals. Such super-pipelines are hitting multiple stages and culminating in production deployments.
While this is still much better than manual no-CI/CD approach, it is not great either. Key problem here is presented by pipelines stuck in the middle waiting for approvals or tests to complete. This quickly creates unmanageable Backlog of Work-in-Progress pipeline runs. Different versions are now tied up together and it is not clear which knot to untie first.
Separating CI and CD means that you have small pipelines that build artifacts. Then you have other small pipelines that perform tests, and yet others that do deployments.
Sometimes you can still mix things. For example, you may mix some quick unit tests into your build CI. But you definitely shouldn’t include your production-grade load testing in there.
As a result of separation, you get multiple small testing, approval and deployment components. They need to run independently and asynchronously – meaning that you also need some system of record to store results and progress (such as Reliza Hub).
2. Your base CI should be your Dockerfile
That simple. With the invention of multi-stage docker builds it is almost always possible to have all CI logic for a component wrapped into its Dockerfile. Key benefit of this approach is consistent build environment everywhere. It will also allow you to switch between various CI tools easily and not get locked-in.
3. Build only once / Move away from GitFlow
One of the main ideas of successful CI/CD process is that you build artifact once and then move it across stages of tests, deployments and approvals. GitFlow breaks this concept since it specifically asks you to do code merges and rebuild artifacts between environments.
Therefore, it is time to admit that GitFlow is an anti-pattern these days. If you’re looking for an alternative, check Trunk Based Development.
4. Verify integrity – use signatures or digests
Too frequently security issues are being ignored, and one of the key ones in CI/CD is checking integrity of components. Best way to verify integrity is to use signed content, but sometimes this gets very complicated.
Second best way to verify integrity is by matching cryptographic digests (usually sha256) which is relatively simple and should always be done.
This point is especially relevant to containers, which have mutable tags. Sometimes developers may not realize that if they include something like redis:5.0.9 in their Dockerfile – this may actually mutate their builds over time. Preferable way would be to use sha256 digest, such as redis:5.0.9@sha256:08aab527ca57f536f2805e031535a6881bab63171146aa6414de69d54b14a84d.
5. Keep it DRY – use templating tools
Templating tools are a huge trend in DevOps these days. Not every ecosystem has good templating tools, but existing tools cover many frequent use cases already. Namely I’m talking about Terragrunt, Helm, Kustomize and others. We’ve even added some templating capabilities to Reliza CLI recently.
Templating tools are essential when dealing with multi-environment projects. Among other benefits, they allow for quick comparison of what is actually different between environments.
6. Do your best to not block developers and fail fast
CI/CD process needs to be built in such a way that developers are allowed to make commits as soon as their code is ready locally.
And if a pipeline fails, it should fail fast. Such failure should not then cause cascading problems across the organization. Instead, developer should get a fast feedback, correct the problem, make another commit and roll forward to successful state.
To achieve fail-fast pattern, use asynchronous execution of pipelines that I mentioned before. The other thing that is required is automated testing. Sufficient automated testing is needed to catch problematic code quickly and fail fast.
7. Track DevOps metrics
Keep metrics that are relevant to you – such as number of deployments, build times, test times, approval times, number of changes per instance, ratio of incidents per deployment and others.
Generally, metrics and KPIs would make a good subject for another post. The point I want to make now is that you need to have some metrics to track. Metrics allow you to quantify progress you are making. So you need metrics to see how you are progressing.
3 comments