Even though this article is about issues experienced with GitOps, I must start by saying that GitOps is certainly a big improvement relative to CIOps. If you’re not very familiar with GitOps, a lot of key details can be found in this post by Dmitri Lerko.
Let’s now jump strait into issues:
1. Default GitOps pattern is not fully auditable
Auditability is frequently named as one of the key advantages of GitOps. However contrary to popular belief, default GitOps process is not fully auditable. Key reason for that is force push. Force push essentially allows to remove any unwanted blocks of commits from Git history on the central repository.
To solve this issue, many organizations disable force push (either for the whole repository or for specific branches). However, for some auditing requirements this still may be not good enough. Since force push is a Git feature and may be turned back on at any time.
This leads to 2 more options to ensure proper auditability:
a) Frequent signed and dated back-ups of Git central repository. Yet, for the purposes of GitOps specifically, when several deployments may happen within seconds, the interval of such back-ups would not be enough to cover pace of changes.
b) Using production logs of Git central repository server as audit trail and sign, date, encrypt and stream those logs continuously to another location. While this is a fine solution, parsing these logs would be the complicated part.
Bottom line – it is possible to build fully auditable logic around GitOps, but it is certainly not there from day one. Let me repeat here that auditability requirements may differ. Particularly, what basic GitOps implementation offers would be perfectly reasonable for many organizations. But there are other organizations who need all those additional workarounds.
2. Business Approvals
Typical approval model used in GitOps is via Pull Requests (PRs). Developers make changes, create PR, then approver may accept such PR, which would be recorded and change would be deployed. Such PR-based approval model is based on a premise that everybody already uses Git and knows what PR is.
All this is great to the point when we need business approvals in the mix. Business people may very well not know Git, not know what PR is and not understand what is happening.
So if we have business approvals we have to go back and have to build a process over GitOps process, which shoots our lead times through the roof.
3. Versioning and Configuration Management
I recently blogged about the underlying issue of versioning in microservices. This issue is very much in a full bloom when we talk about GitOps.
Key problem with GitOps is that we need to supply versions of components into our YAML files under Git version control – and those versions would then be picked up by our CD system of choice (such as Flux or Argo).
If an organization allows to always pick latest version of every component and then resolve differences on production – this is not a real problem for them. But for many organizations this would not be an option. They need to be strict about what changes went in and what production version they have at what point (this partly reflects the business approval problem above).
Feature flags may help only somewhat here, since there must be a high level of tolerance to the fact that each feature flag implementation may actually contain breaking changes too.
Now, if our organization is strict about its versioning policy and may not use latest and greatest all the time, it becomes very hard to keep track of all the possible permutations of components. If we are doing GitOps, the question now becomes “how we assign components to environments?” Or in other words, “how we know what versions of what components we should actually commit to Git?”
Speaking about configuration management, we are facing issue that is very similar to versioning. If we need to deal with configuration changes, we need to know which ones we may safely submit via our GitOps tool of choice. And keeping track of those changes is very much non-obvious.
Summary
GitOps is a great approach and earned a right to be seen as a big improvement over previous generation of CIOps. However, it does not solve all the questions in the world of Continuous Delivery and there are at least 3 significant issues still present with GitOps. Namely: Auditability, Business Approvals and Versioning and Configuration Management.
2 comments