Remember “The Lean Startup” where they asked engineers to push a change to production on their first day of work? Then the quote to justify it was: “If our production process is so fragile that you can break it on your very first day of work, shame on us for making it so easy to do so”.
“The Lean Startup” was published almost 10 years ago, in 2011. Unfortunately, this simple idea described above still does not resonate with many organizations. On the contrary, putting more pressure on developers committing and pushing code is quite common.
In my experience, the #1 reason for the loss of Software Development productivity is this exact fear to push a change to the Version Control Repository.
Bureaucracy Always Wants To Slow Down Git Push
Developers are frequently told all sorts of things before they push changes: that they need to branch first, or they need to merge to several branches, or they need pull requests first, or they must test locally first, or they need peer reviews, and on and on it goes. Frequently it turns into a strict ever-changing checklist of what a developer needs to do before pushing a simple change. Organization bureaucracy is thus trying to protect itself from negative consequences of botched releases.
But the truth is that if developers are afraid of pushing changes all sorts of bad things will naturally happen:
- Developers would hoard changes to push at the very last moment, creating a snowball effect, increasing the batch size and making it very hard to make sense of what is pushed.
- Conflicts are now everywhere, which leads to proliferation of merges, which leads to lots of bugs and yet makes already large snowball of changes even bigger.
- Every process is synchronized, and everybody spends a lot of time waiting for something – for approvals, for branches being cut, for changes being pushed, for meetings being held – which results in a plain downtime for a lot of people across the organization, while they sit waiting or redoing their work due to merge-conflicts.
- Finally, since pushes happen at the last possible moment, deadlines will be missed, releases truly will be botched and the customers won’t be happy.
Making It Right
So as DevOps / GoalOps people, what can we do to make it right? First and foremost, let developers push their changes whenever they feel changes are ready. For this to work correctly, CI process on push must become asynchronous.
This means that the CI process itself must be decoupled from downstream approvals and deployments. With such approach, CI is guaranteed to run on every push and provide needed level of feedback to developers. At the same time untested and unapproved CI artifacts should not clog any downstream CI/CD components. Meaning that developers may do their work freely, while downstream teams may do their work freely on other artifacts at the very same time. Sure enough, they need to communicate, but the process must prevent teams from blocking each other.
A reasonable developer or a development team getting continuous feedback from such CI process would gradually improve and over time balance the amount of code and testing needed for builds to roll accurately. This is what we call asynchronous process, where each team receives needed feedback quickly while at the same time not clogging processes for other teams.
Surely, we need the right tooling to make this work. Having one-fits-all pipeline in Jenkins would unlikely support asynchronous workflow. Instead, another approach would be more suitable: asynchronous CI goes first, then asynchronous Release Assembly, and then asynchronous CD – an approach of this kind we are building with Reliza Hub. The key is decoupling release phases and removing hard dependencies between teams for the sake of faster feedback.
Summary
If we can remove fear from git push – this is one thing that can increase Software Development productivity dramatically. Getting this right is also an indicator of maturity of GoalOps culture. Both technical and business people will appreciate that.
2 comments
Comments are closed.