On-demand builds on GitHub Actions with Reliza

10 days ago I published an article how we integrate Reliza Hub with GitHub Actions and ArgoCD for continuous delivery to Kubernetes. Today I am going to improve the approach described there with a new feature we just added to Reliza Hub. The feature is on-demand builds for GitHub Actions that would run on new releases and approvals.

As with the original article we will be discussing Continuous Integration and Continuous Deployment for the Mafia Game project. By the way, if you want to play Classic Mafia with me, I’m usually playing every Saturday with Vancouver Mafia Club.

So the point we want to improve from the original article is the GitHub Actions workflow to poll Reliza Hub for new production versions. This workflow was running on a cron every 5 minutes and if there was a new version, it would roll out an update. The improvement we want to make – instead of using cron there, we want to have on demand GitHub Actions run every time someone approves a Mafia Game product release. Below I will show step-by-step guide how to achieve that.

GitHub Actions Side

In the GitHub, proceed to Reliza Hub app at https://github.com/apps/reliza-hub.

Reliza Hub Application on GitHub

Click Configure. Then choose your user or organization where you want to install the application.

Install Reliza Hub Menu on GitHub
Install Reliza Hub Menu on GitHub

In my case, I will be selecting my user taleodor and only one repository – taleodor/mafia-deployment – where my GitHub Actions workflows are deployed.

Install Reliza Hub on GitHub - Select Repositories
Install Reliza Hub on GitHub – Select Repositories

Now I just click Install which brings me to the App Installation page shown below.

Reliza Hub App Installation Page on GitHub
Reliza Hub App Installation Page on GitHub

One thing we need to do here is note the installation id which is shown in the browser address bar. In my case that is 1234567 as shown on the image above. Kidding – I’ve hidden the real id, but you get the idea 🙂 .

We are now done with GitHub part and moving on to modify our workflow code to listen to trigger build events from Reliza.

GitHub Actions Workflow Code Side

Remember that we had cron doing updates which had this code at the top:

on:
  schedule:
    - cron:  '*/5 * * * *'

Meaning that our action would be running every 5 minutes.

All we need to do here is change it to action listening code as following:

on:
  repository_dispatch:
    types: [reliza-build-event]

This is all we needed to do in the workflow code side, and now we are moving to the Reliza Hub SaaS side.

Reliza Hub SaaS Side

First of all we need to ensure that we already have integrated repository registered on Reliza Hub. This is indeed the case for the Mafia Game project, but if not, we need to go to VCS menu and click on the plus-circle icon to add it there.

Mafia Game VCS Repositories in Reliza Hub

Now after we ensured we have our repository, remember that our Mafia Game project consists of 2 major parts: UI and back-end. Each of them should trigger deployment build on approval. So we need to add integration to each of those projects. I will demonstrate what needs to be done on the mafia-vue (UI) project. Doing so for the back-end would be essentially the same – so I will save some space by not mentioning it explicitly.

First we need to go to our project section and click on the tool icon to open Project Settings as shown below.

Mafia-vue Project - Project Settings via Tool icon
Mafia-vue Project – Project Settings via Tool icon

Then in the GitHub App Installation Id we enter our Installation Id from GitHub noted above. We also set checkbox called Trigger GitHub Action On Approval. Note that when this checkbox is checked, every new release for the project and every following approval will trigger build on GitHub Actions.

Finally we click edit icon near Not Set and select our VCS Repository. How it all looks is shown below.

Setting GitHub Actions Integration for the mafia-vue project in Reliza Hub
Setting GitHub Actions Integration for the mafia-vue project in Reliza Hub

I then click Save GitHub Integration button. And then click Trigger GitHub Build to see if our integration works as expected. If everything was configured correctly, you would see GitHub Actions build started by reliza-hub bot with the event called reliza-build-event in your GitHub Actions build page as shown below.

GitHub Actions workflow started by reliza-hub bot
GitHub Actions workflow started by reliza-hub bot

How it Works on Releases and Approvals End-to-End

Now, when everything is configured let us push a simple change to the Mafia Project. In this case, I want to add a short page about classic mafia clubs. I prepare my change and push it to the UI repository.

CI process using GitHub Actions in Mafia Vue repository picks up the change and starts the build.

When it is done, I am getting slack notification that the build is ready.

Slack notification that mafia-vue build is ready
Slack notification that mafia-vue build is ready

Following that our new GitHub Actions integration will trigger deployment build. This deployment build will pull up new project definition for testmafia.brolia.com server and commit it to the mafia-deployment project. This will in turn trigger new deployment for testmafia itself via ArgoCD. Finally, new Mafia product will be created. I will get notified of all that activity via slack:

Slack notifications for Mafia Deployments
Slack notifications for Mafia Game Deployments

Now, I would test the changes in testmafia.brolia.com and once happy set approval to my Product release.

Set PM Approval for Mafia Game Product Release in Reliza Hub
Set PM Approval for Mafia Game Product Release in Reliza Hub

Setting approvals triggers another build on GitHub Actions, which would this time promote our new release to the mafia.brolia.com production instance. Here is how it looks on Reliza Hub after the update:

Mafia Game Production instance in Reliza Hub
Mafia Game Production instance in Reliza Hub

Summary

We were able to replace deployment job working on cron schedule to on-demand version which is triggered by new releases and approvals. All that by extending GitHub Actions functionality with on-demand builds using new Reliza Hub integration capabilities.

Note that current integration is static in terms that it does not send any special parameters to GitHub Actions.

I am now thinking about creating a service with parametrized on-demand builds on GitHub Actions managed from Reliza Hub. You would be able to send custom parameters from there to GitHub Actions builds and trigger them. Would it be useful for you?

Storing Arbitrary Values in Java Keystore

Java Keystore is a nice tool, but it has a very limited number of inputs it supports natively. Mainly those are pem certificates and corresponding keys.

Fortunately, there is a way to store arbitrary data using keytool’s -importpass command and base64 encoding.

Here is how to achieve that. Let’s imagine we have some secret.bin file, we would like to store as a secret. Below is a shell scripting sequence using keytool that would create our keystore (remember to set desired alias and storepass):

base64 -w 0 secret.bin | keytool -importpass -keystore test.jks -alias key1 -storepass mypass

Now, we can read it in Java with code like the following (note, that I’m omitting proper error handling and stream closures – you should use try-with-resources pattern):

char[] pass = "mypass".toCharArray();
KeyStore ks = KeyStore.getInstance("JKS");
ks.load(new FileInputStream("/path/to/test.jks"), pass);
Key mykey = ks.getKey("key1", pass);
byte[] keybytes = mykey.getEncoded();
byte[] decodedKey = Base64.getDecoder().decode(keybytes);

This is it, now decodedKey variable contains your secret binary and you can use it further.

You Can Now Book 30-min Reliza Hub Demo With Me

Use this link to book no-strings-attached 30-minute Reliza Hub demo with me.

I’ll use the first 15 minutes to give you brief overview of the platform and its capabilities. Feel free to use the remaining time to discuss DevOps or Software Productivity issues in your organization or ask any questions you may have.

It will be my pleasure to meet you!

Wrote a Medium article how we are using Reliza Hub with GitHub Actions and ArgoCD

This is about using Reliza Hub for Kubernetes CICD pipeline. Key problem we are trying to help with is managing different versions and permutations of versions of microservices.

The article can be found here: https://medium.com/@taleodor/building-kubernetes-cicd-pipeline-with-github-actions-argocd-and-reliza-hub-e7120b9be870

Take The Fear Out Of Git Push

(Git) Push To Reset The World
Photo by Tom Grünbauer on Unsplash

Remember “The Lean Startup” where they asked engineers to push a change to production on their first day of work? Then the quote to justify it was: “If our production process is so fragile that you can break it on your very first day of work, shame on us for making it so easy to do so”.

“The Lean Startup” was published almost 10 years ago, in 2011. Unfortunately, this simple idea described above still does not resonate with many organizations. On the contrary, putting more pressure on developers committing and pushing code is quite common.

In my experience, the #1 reason for the loss of Software Development productivity is this exact fear to push a change to the Version Control Repository.

Bureaucracy Always Wants To Slow Down Git Push

Developers are frequently told all sorts of things before they push changes: that they need to branch first, or they need to merge to several branches, or they need pull requests first, or they must test locally first, or they need peer reviews, and on and on it goes. Frequently it turns into a strict ever-changing checklist of what a developer needs to do before pushing a simple change. Organization bureaucracy is thus trying to protect itself from negative consequences of botched releases.

But the truth is that if developers are afraid of pushing changes all sorts of bad things will naturally happen:

  1. Developers would hoard changes to push at the very last moment, creating a snowball effect, increasing the batch size and making it very hard to make sense of what is pushed.
  2. Conflicts are now everywhere, which leads to proliferation of merges, which leads to lots of bugs and yet makes already large snowball of changes even bigger.
  3. Every process is synchronized, and everybody spends a lot of time waiting for something – for approvals, for branches being cut, for changes being pushed, for meetings being held – which results in a plain downtime for a lot of people across the organization, while they sit waiting or redoing their work due to merge-conflicts.
  4. Finally, since pushes happen at the last possible moment, deadlines will be missed, releases truly will be botched and the customers won’t be happy.

Making It Right

So as DevOps / GoalOps people, what can we do to make it right? First and foremost, let developers push their changes whenever they feel changes are ready. For this to work correctly, CI process on push must become asynchronous.

This means that the CI process itself must be decoupled from downstream approvals and deployments. With such approach, CI is guaranteed to run on every push and provide needed level of feedback to developers. At the same time untested and unapproved CI artifacts should not clog any downstream CI/CD components. Meaning that developers may do their work freely, while downstream teams may do their work freely on other artifacts at the very same time. Sure enough, they need to communicate, but the process must prevent teams from blocking each other.

A reasonable developer or a development team getting continuous feedback from such CI process would gradually improve and over time balance the amount of code and testing needed for builds to roll accurately. This is what we call asynchronous process, where each team receives needed feedback quickly while at the same time not clogging processes for other teams.

Surely, we need the right tooling to make this work. Having one-fits-all pipeline in Jenkins would unlikely support asynchronous workflow. Instead, another approach would be more suitable: asynchronous CI goes first, then asynchronous Release Assembly, and then asynchronous CD – an approach of this kind we are building with Reliza Hub. The key is decoupling release phases and removing hard dependencies between teams for the sake of faster feedback.

Summary

If we can remove fear from git push – this is one thing that can increase Software Development productivity dramatically. Getting this right is also an indicator of maturity of GoalOps culture. Both technical and business people will appreciate that.

From DevOps to GoalOps

By now you are probably tired of all the different ways how we may call Software Operations. ITIL, Agile, DevOps, DevSecOps – and then quite a few more recent ones like NoOps, AllOps, AIOps, MLOps, GitOps, DataOps… I know I missed a few.

So why on earth we need “GoalOps”?

Surprisingly, modern software development deviated quite a bit from the spirit of “The Goal” written by Eliyahu M. Goldratt. Yes, we have whole DevOps movement in technology popularized by Gene Kim’s “The Phoenix Project” and most recently “The Unicorn Project”. And on the other side – business side – we have Agile and its various methodologies.

However, it turned out that DevOps appeared to be too geeky, while Agile appeared to look too much like some marketing trick. Key problem here is the huge rift between technical people and business people – the fact that they don’t understand each. To they point they tend to use different terms for the same thing.

I recently heard quite a bit from product marketing people that they don’t really understand their tech teams, and it is like they go in different directions. I frequently heard even stronger words towards “business people” coming from the tech teams. As a result we get poor products and broken organizations.

The time has now come to do things differently. It is time to show to both business and technical people that they are part of the same organization and they need to work towards a common Goal.

For this to happen – their optics must be aligned, their culture must be aligned, their perception of success must be aligned and The Common Goal must be defined clearly.

That is why we need GoalOps – a process of business and technology parts of organization working together towards The Common Goal. A new prospective that can finally be understood and agreed by both sides.

Software development, Robotics, HealthTech, AI – all those are just too important to let the old rift between Technology and Business continue. That is why starting from the Goal, then applying it to modern realities and joining forces across whole organization is so important. That is why GoalOps is the future.

Approvals for any CI CD in Reliza Hub – Demo Video

As we are working with Reliza Hub to make it useful to both Technical and Business teams, we have just launched a new feature: approvals for any CI/CD DevOps or DevSecOps pipeline out there. Here is a demo video that I recorded.

Would very much appreciate any feedback on YouTube:

Reliza Hub Approvals For Any CI CD pipeline – Feature Demo

3 Problems With GitOps

GitOps commits
Photo by Yancy Min on Unsplash

Even though this article is about issues experienced with GitOps, I must start by saying that GitOps is certainly a big improvement relative to CIOps. If you’re not very familiar with GitOps, a lot of key details can be found in this post by Dmitri Lerko.

Let’s now jump strait into issues:

1. Default GitOps pattern is not fully auditable

Auditability is frequently named as one of the key advantages of GitOps. However contrary to popular belief, default GitOps process is not fully auditable. Key reason for that is force push. Force push essentially allows to remove any unwanted blocks of commits from Git history on the central repository.

To solve this issue, many organizations disable force push (either for the whole repository or for specific branches). However, for some auditing requirements this still may be not good enough. Since force push is a Git feature and may be turned back on at any time.

This leads to 2 more options to ensure proper auditability:

a) Frequent signed and dated back-ups of Git central repository. Yet, for the purposes of GitOps specifically, when several deployments may happen within seconds, the interval of such back-ups would not be enough to cover pace of changes.

b) Using production logs of Git central repository server as audit trail and sign, date, encrypt and stream those logs continuously to another location. While this is a fine solution, parsing these logs would be the complicated part.

Bottom line – it is possible to build fully auditable logic around GitOps, but it is certainly not there from day one. Let me repeat here that auditability requirements may differ. Particularly, what basic GitOps implementation offers would be perfectly reasonable for many organizations. But there are other organizations who need all those additional workarounds.

2. Business Approvals

Typical approval model used in GitOps is via Pull Requests (PRs). Developers make changes, create PR, then approver may accept such PR, which would be recorded and change would be deployed. Such PR-based approval model is based on a premise that everybody already uses Git and knows what PR is.

All this is great to the point when we need business approvals in the mix. Business people may very well not know Git, not know what PR is and not understand what is happening.

So if we have business approvals we have to go back and have to build a process over GitOps process, which shoots our lead times through the roof.

3. Versioning and Configuration Management

I recently blogged about the underlying issue of versioning in microservices. This issue is very much in a full bloom when we talk about GitOps.

Key problem with GitOps is that we need to supply versions of components into our YAML files under Git version control – and those versions would then be picked up by our CD system of choice (such as Flux or Argo).

If an organization allows to always pick latest version of every component and then resolve differences on production – this is not a real problem for them. But for many organizations this would not be an option. They need to be strict about what changes went in and what production version they have at what point (this partly reflects the business approval problem above).

Feature flags may help only somewhat here, since there must be a high level of tolerance to the fact that each feature flag implementation may actually contain breaking changes too.

Now, if our organization is strict about its versioning policy and may not use latest and greatest all the time, it becomes very hard to keep track of all the possible permutations of components. If we are doing GitOps, the question now becomes “how we assign components to environments?” Or in other words, “how we know what versions of what components we should actually commit to Git?”

Speaking about configuration management, we are facing issue that is very similar to versioning. If we need to deal with configuration changes, we need to know which ones we may safely submit via our GitOps tool of choice. And keeping track of those changes is very much non-obvious.

Summary

GitOps is a great approach and earned a right to be seen as a big improvement over previous generation of CIOps. However, it does not solve all the questions in the world of Continuous Delivery and there are at least 3 significant issues still present with GitOps. Namely: Auditability, Business Approvals and Versioning and Configuration Management.