Surprisingly I’m recently not finding a strong majority of voices saying that Push-Based Continuous Delivery should never be used due to security concerns. So I feel there is a need to clarify risks more explicitly.
First of all here is what I mean by Push-Based CD. Simply, this is the approach, where SSH-key or some deployment token (i.e. Kubernetes token) is placed directly into CI tool and used to establish connection from CI tool to deployment server and force some action on that server.
Disclaimer: I would like to specifically mention that this post does not cover larger enterprises which may have well-established and satisfactory security practices around Push-Based CDs. Instead the post is catered towards smaller organizations where Pull-Based CDs unlike Push-Based provide natural controls.
So what could be wrong with Push-Based CD?
- Credential leak from the CI system. Anecdotally, this is pretty much the only attack vector that is well understood. The understanding here is that CI system would somehow be breached and credential would leak. While this is extremely significant, this is by far not the most likely event to happen. But there are other – more likely – attack scenarios as you will see below.
- A person with push access to source code may abuse the system. This can be either malicious or accidental, but essentially while CI systems do not normally expose credentials directly, it is possible to use such credentials to run arbitrary commands on deployment environment. It is also possible to modify CI script from some branch to explicitly egress credentials to anywhere. How to mitigate this: common way is to ensure that credential scope is only limited to specific deployment operation, implement code owners, branch restrictions and other methods ensuring that only certain people can commit code to certain branches and certain locations. Important: while these mitigation methods exist, they are increasingly complex and it is easy to make a mistake or miss an attack scenario – which in turn would result in serious security breach.
- Management APIs from deployment instance must be widely exposed. This includes things like wide exposure of SSH port, or exposing Kubernetes APIs to the Internet. Essentially, the implication here is that the attack surface is substantially increased.
- Credential leak on transport layer. When the credential is transferred to its storage on the CI system, there is a risk that this credential will be compromised. Generally if working with things like SSH credentials, the rule of thumb is not to transport them at all. Find more about SSH security specifically in my related talk.
- Increased pressure on book-keeping and maintenance. If the above is not enough, remember that any change in the deployment process would put more pressure on maintenance of existing CICD jobs and credentials. The implication is that a mistake or broken process may cause security breach.
The alternative is clear – use Pull-Based CD approach from within the instance itself. So essentially, CI job builds an artifact and pushes it to some storage. Then from the instance the artifact is pulled by CD job via some routing rules. Read more about modern CI/CD best practices in my other post.
To summarize, not so long ago it was well understood that any production-grade instance must be isolated from the Internet and use jump-boxes or bastions to connect to them. Push-Based CD was clearly not even considered as a viable option in such setup.
However, now it seems that there is proliferation of quick-start tutorials that ignore basic security practices without a good disclaimer. As large amount of people seem to only learn through these tutorials, it creates a dangerous precedent in the community, where too many think now it is suddenly ok to use Push-Based CD.