Hiring Lead Developers – Seeing The Project End State

I was recently reflecting on hiring and team management decisions. Particularly, how can you tell that a developer would make a good lead?

My current thinking on this is following. The most important differentiator of whether a developer is a good lead is the ability to see the project end state. While junior and intermediate developers are sort of coasting along, senior and lead developers are the ones who can formulate what exactly they are building.

Tricky part in this is understanding what bells and whistles will actually make it near future and which ones will not. In other words, key is to formulate what is the bare minimum for the project that more-or-less fits business requirements and is at the same time achievable by the team. Developers who get a good sense of this would likely make good leads.

It is interesting that lead developers may not be the best in terms of actual coding or technology expertise. So it is quite normal for teams to have management path for developers (to be a lead) and what I call architect path (to become technology expert). I know terminology may be confusing, as lead and architect terms are frequently mixed up in the industry.

Essentially, lead developers are the ones who envisions the end state and can schedule work from the prospective of end state. Therefore, lead developers are good at prioritizing work – simply because they know what they are building.

So how can we interview lead developer candidates? I would say, describe a hypothetical project (based on some specific project that you know really well) and ask what absolutely needs to be done to launch it. Ask about everything from recommended technology to desired team size and expected time frame. An answer you are looking for is a very concise description of specific steps needed. Certain questions posed by a candidate would also be a good indicator – i.e. trying to gauge performance characteristics or friendliness of early stage users.

Analyzing the candidate’s answer you need to try to match steps and technologies proposed to actual requirements as they were stated. Potential red flags for me would be something like proposing heavy stacks for early stage startup projects at prototyping phase (think Kafka for a blog platform). Similarly proposing light stacks for difficult problems (think WordPress for ML solution). Similarly mismatches in team sizes and time needed should also be considered as red flags. So what you are looking for is the answer that fits the problem and supporting questions that help candidate refine and better understand the problem. Good answer shows that candidate thinks about the end state of your project and thus would make a good lead.

Running k3s on Windows with WSL2

The original instructions were for microk8s but I had glitches with its operations on Windows.
So I replaced microk8s with k3s and it worked.
Algorithm of installation goes as following:
1. Install WSL2 – instructions here: https://docs.microsoft.com/en-us/windows/wsl/install-win10 (note that you need specific versions of Windows 10 referenced in the document). One of key bonuses right away with WSL2 – you now may have full docker running on Windows Home!

2. One thing I recommend right away after installing WSL2 – set some sane memory and CPU limits for it, otherwise it becomes resource hog quickly. How you can read here – https://medium.com/@lewwybogus/how-to-stop-wsl2-from-hogging-all-your-ram-with-docker-d7846b9c5b37

3. Set up your favorite Ubuntu 18.04 from Microsoft Store. Here comes surprise – it comes with systemd disabled and not working snaps.

4. To obtain systemd I tried bunch of articles, and finally this one worked – https://forum.snapcraft.io/t/running-snaps-on-wsl2-insiders-only-for-now/13033 (would appreciate feedback on this one, since I had few other things on my system by the time I tried this one – but hopefully it works right away).

5. Finally install k3s as usual as described at https://k3s.io/ .

6. Note, that to reach k3s cluster from your browser or powershell on host you would need to note ip of Ubuntu vm, which can be found for example by running ifconfig.

List All Docker Containers with IP Addresses

Solution for one container can be found in this stackoverflow: https://stackoverflow.com/questions/17157721/how-to-get-a-docker-containers-ip-address-from-the-host

Solution for all containers is below:

docker ps | awk 'NR>1{ print $1 }' | xargs docker inspect -f '{{range .NetworkSettings.Networks}}{{$.Name}}{{" "}}{{.IPAddress}}{{end}}'

Versioning Feature and Release Branches

For some time I was thinking how to solve a use case where we track both feature and release branches in Reliza Hub.

The problem encountered is related to versioning. Here is the problem in a nutshell: if all branches share same versioning schema, then releases overlap and by simply looking at versions we don’t know which is which. If we only deal with release branches, the solution would be what I presented earlier in this blog – namely, separating branch versions by Minor in SemVer case or by Month in the CalVer case.

However, feature branches complicate this – there are just too many of them to do Minor separation. These days a frequently used pattern is to create a feature branch per each task ticket or per each pull request, and then merge it into master (or in case of urgent fixes – also into release branch). Conventional naming for feature branches would be just the ticket name it references, i.e. TICKET-134 or PR-12. So, we end up having lots of such feature branches with unclear versioning.

The way we started to solve it with Reliza products is by adding specific versioning component Branch. So that our default versioning schema for feature branches becomes Branch.Micro. A sample release version for our feature branch could be TICKET-134.5 – meaning 6th release of our TICKET-134 branch.

There are of course variants, where you could do something like YYYY.0M.Branch.Micro – this way you can also track creation month of the branch. The main benefit of such approach is that feature branch releases do not mix with release branch releases, so we wouldn’t accidentally pull them into non-test environments.

Finally, we are going to introduce approvals for release branches. So that we explicitly indicate which feature branch should be used per environment.

The resulting release selection structure on Reliza Hub is going to look as following:

  1. Identify to what environment our instance belongs
  2. Identify project branch that is currently approved to this environment
  3. Identify most recent release from this branch approved to this environment -> this would be the release version that would be returned to UI and integrated CD system.

This internally creates 2-step selection process instead of 1-step process we used previously. That 1-step process required clients to explicitly specify desired branch, which was not obvious in many cases (as shown above). 2-step selection would add greater logic complexity onto Reliza Hub but remove complexity from the clients – who would just ask for latest approved release per environment and get it. Without worrying about branches.

We are currently actively working on implementing the above logic and it hopefully should be live by the end of June 2020. Meanwhile, I would appreciate any comments or feedback via DM to my LinkedIn.

7 Best Practices of Modern CI/CD

This is a summary of my research of modern CI/CD practices while working on Reliza Hub. This list is rather opinionated but I try to provide explanations why I hold specific opinions. Finally, I’m making this an ordered list, but it’s not actually sorted by importance.

So, let’s start:

1. Separate CI and CD – Use asynchronous pipelines

Too many times I see convoluted multistage CI+CD pipelines, all in one, with bunch of tests and approvals. Such super-pipelines are hitting multiple stages and culminating in production deployments.

While this is still much better than manual no-CI/CD approach, it is not great either. Key problem here is presented by pipelines stuck in the middle waiting for approvals or tests to complete. This quickly creates unmanageable Backlog of Work-in-Progress pipeline runs. Different versions are now tied up together and it is not clear which knot to untie first.

Separating CI and CD means that you have small pipelines that build artifacts. Then you have other small pipelines that perform tests, and yet others that do deployments.

Sometimes you can still mix things. For example, you may mix some quick unit tests into your build CI. But you definitely shouldn’t include your production-grade load testing in there.

As a result of separation, you get multiple small testing, approval and deployment components. They need to run independently and asynchronously – meaning that you also need some system of record to store results and progress (such as Reliza Hub).

Continue reading “7 Best Practices of Modern CI/CD”

Using Google App Engine and GitHub Actions for Static Websites

There are 2 most popular platforms today to host static websites: Netlify and Google App Engine (GAE).

Netlify gained a lot of popularity recently because it provides integrated experience with building and deployment static websites – all in the same platform.

GAE is more flexible in terms that it supports non-static web-sites as well but it certainly lacks the level of integration Netlify provides. So why would I still use GAE in many case? The answer is simple – frequently GAE provides better pricing options. At the same time it allows to have all the features of Netlify and more using few simple tools (GitHub Actions and Reliza Hub) which I’ll describe in this article.

Continue reading “Using Google App Engine and GitHub Actions for Static Websites”

On-demand builds on GitHub Actions with Reliza

10 days ago I published an article how we integrate Reliza Hub with GitHub Actions and ArgoCD for continuous delivery to Kubernetes. Today I am going to improve the approach described there with a new feature we just added to Reliza Hub. The feature is on-demand builds for GitHub Actions that would run on new releases and approvals.

As with the original article we will be discussing Continuous Integration and Continuous Deployment for the Mafia Game project. By the way, if you want to play Classic Mafia with me, I’m usually playing every Saturday with Vancouver Mafia Club.

So the point we want to improve from the original article is the GitHub Actions workflow to poll Reliza Hub for new production versions. This workflow was running on a cron every 5 minutes and if there was a new version, it would roll out an update. The improvement we want to make – instead of using cron there, we want to have on demand GitHub Actions run every time someone approves a Mafia Game product release. Below I will show step-by-step guide how to achieve that.

GitHub Actions Side

In the GitHub, proceed to Reliza Hub app at https://github.com/apps/reliza-hub.

Reliza Hub Application on GitHub

Click Configure. Then choose your user or organization where you want to install the application.

Install Reliza Hub Menu on GitHub
Install Reliza Hub Menu on GitHub

In my case, I will be selecting my user taleodor and only one repository – taleodor/mafia-deployment – where my GitHub Actions workflows are deployed.

Install Reliza Hub on GitHub - Select Repositories
Install Reliza Hub on GitHub – Select Repositories

Now I just click Install which brings me to the App Installation page shown below.

Reliza Hub App Installation Page on GitHub
Reliza Hub App Installation Page on GitHub

One thing we need to do here is note the installation id which is shown in the browser address bar. In my case that is 1234567 as shown on the image above. Kidding – I’ve hidden the real id, but you get the idea 🙂 .

We are now done with GitHub part and moving on to modify our workflow code to listen to trigger build events from Reliza.

GitHub Actions Workflow Code Side

Remember that we had cron doing updates which had this code at the top:

    - cron:  '*/5 * * * *'

Meaning that our action would be running every 5 minutes.

All we need to do here is change it to action listening code as following:

    types: [reliza-build-event]

This is all we needed to do in the workflow code side, and now we are moving to the Reliza Hub SaaS side.

Reliza Hub SaaS Side

First of all we need to ensure that we already have integrated repository registered on Reliza Hub. This is indeed the case for the Mafia Game project, but if not, we need to go to VCS menu and click on the plus-circle icon to add it there.

Mafia Game VCS Repositories in Reliza Hub

Now after we ensured we have our repository, remember that our Mafia Game project consists of 2 major parts: UI and back-end. Each of them should trigger deployment build on approval. So we need to add integration to each of those projects. I will demonstrate what needs to be done on the mafia-vue (UI) project. Doing so for the back-end would be essentially the same – so I will save some space by not mentioning it explicitly.

First we need to go to our project section and click on the tool icon to open Project Settings as shown below.

Mafia-vue Project - Project Settings via Tool icon
Mafia-vue Project – Project Settings via Tool icon

Then in the GitHub App Installation Id we enter our Installation Id from GitHub noted above. We also set checkbox called Trigger GitHub Action On Approval. Note that when this checkbox is checked, every new release for the project and every following approval will trigger build on GitHub Actions.

Finally we click edit icon near Not Set and select our VCS Repository. How it all looks is shown below.

Setting GitHub Actions Integration for the mafia-vue project in Reliza Hub
Setting GitHub Actions Integration for the mafia-vue project in Reliza Hub

I then click Save GitHub Integration button. And then click Trigger GitHub Build to see if our integration works as expected. If everything was configured correctly, you would see GitHub Actions build started by reliza-hub bot with the event called reliza-build-event in your GitHub Actions build page as shown below.

GitHub Actions workflow started by reliza-hub bot
GitHub Actions workflow started by reliza-hub bot

How it Works on Releases and Approvals End-to-End

Now, when everything is configured let us push a simple change to the Mafia Project. In this case, I want to add a short page about classic mafia clubs. I prepare my change and push it to the UI repository.

CI process using GitHub Actions in Mafia Vue repository picks up the change and starts the build.

When it is done, I am getting slack notification that the build is ready.

Slack notification that mafia-vue build is ready
Slack notification that mafia-vue build is ready

Following that our new GitHub Actions integration will trigger deployment build. This deployment build will pull up new project definition for testmafia.brolia.com server and commit it to the mafia-deployment project. This will in turn trigger new deployment for testmafia itself via ArgoCD. Finally, new Mafia product will be created. I will get notified of all that activity via slack:

Slack notifications for Mafia Deployments
Slack notifications for Mafia Game Deployments

Now, I would test the changes in testmafia.brolia.com and once happy set approval to my Product release.

Set PM Approval for Mafia Game Product Release in Reliza Hub
Set PM Approval for Mafia Game Product Release in Reliza Hub

Setting approvals triggers another build on GitHub Actions, which would this time promote our new release to the mafia.brolia.com production instance. Here is how it looks on Reliza Hub after the update:

Mafia Game Production instance in Reliza Hub
Mafia Game Production instance in Reliza Hub


We were able to replace deployment job working on cron schedule to on-demand version which is triggered by new releases and approvals. All that by extending GitHub Actions functionality with on-demand builds using new Reliza Hub integration capabilities.

Note that current integration is static in terms that it does not send any special parameters to GitHub Actions.

I am now thinking about creating a service with parametrized on-demand builds on GitHub Actions managed from Reliza Hub. You would be able to send custom parameters from there to GitHub Actions builds and trigger them. Would it be useful for you?

Storing Arbitrary Values in Java Keystore

Java Keystore is a nice tool, but it has a very limited number of inputs it supports natively. Mainly those are pem certificates and corresponding keys.

Fortunately, there is a way to store arbitrary data using keytool’s -importpass command and base64 encoding.

Here is how to achieve that. Let’s imagine we have some secret.bin file, we would like to store as a secret. Below is a shell scripting sequence using keytool that would create our keystore (remember to set desired alias and storepass):

base64 -w 0 secret.bin | keytool -importpass -keystore test.jks -alias key1 -storepass mypass

Now, we can read it in Java with code like the following (note, that I’m omitting proper error handling and stream closures – you should use try-with-resources pattern):

char[] pass = "mypass".toCharArray();
KeyStore ks = KeyStore.getInstance("JKS");
ks.load(new FileInputStream("/path/to/test.jks"), pass);
Key mykey = ks.getKey("key1", pass);
byte[] keybytes = mykey.getEncoded();
byte[] decodedKey = Base64.getDecoder().decode(keybytes);

This is it, now decodedKey variable contains your secret binary and you can use it further.