2 thoughts today while snowboarding

Tried snowboarding for the 2nd time in my life (was not too bad relative to the 1st time 😉 )
Had those 2 thoughts in the process:
1. Mountain skiing and snowboarding are really great sports to treat OCD: if you get too much control, you can’t get speed – you stop and you fall; if you get no control whatsover – you go to fast, and again – fall. So the idea is to find that optimal balance with some control but not too many (can’t control everything after all).
2. One thing that coronovirus story should re-enforce – is that remote workforce is the only way to go in the modern world. How many time did it occur that somebody would come to work sick, and then everybody in the office would go out sick, and then cycle repeats throughout the year. That is especially bad in crammed places, like call-centers. Has anyone tried to estimate the loss of productivity due to sickness (not even mentioning other things such as quality of life or life expectancy)? It is absolutely ridiculous to force everybody to work from the same space when there is no real need for that.

Kubernetes – list all deployed images with sha256 hash

While there is official documentation how to list all kubernetes images here, it’s missing imageID field that includes sha256 hash. Sha256 digest is crucial for our use-case at Reliza, so here are working commands to list all images and all image ids:

# get all imageIDs (with sha256 hash digest)
kubectl get pods --all-namespaces -o jsonpath="{.items[*].status.containerStatuses[0].imageID}"

Notes:

  • Tried on kubernetes 1.17
  • Images returned are whitespace separated
  • Images are duplicated (one image per pod) – but should be trivial to de-duplicate

How to make microk8s work with helm 3

This is a quick note for self. When running microk8s and trying to wire helm 3 I was getting “Error: Kubernetes cluster unreachable”. Workaround I found is the following:

mkdir /etc/microk8s
microk8s.config > /etc/microk8s/microk8s.conf
export KUBECONFIG=/etc/microk8s/microk8s.conf

This block above pretty much does the trick. Obviously, for production or near production use it’s worth adding cron and adding export command to something like .bash_profile.
P.s. What helped me a lot was this discussion of a similar issue for k3s: https://github.com/rancher/k3s/issues/1126

Ford v Ferrari – best business movie since Moneyball

Finally watched Ford v Ferrari yesterday – should have done it earlier but was busy and dealing with bunch of issues. It’s a terrific movie overall, very relevant to today. Even though we’d like to see some things changing since 1960s, unfortunately it’s often not the case.

After discussing with my wife here are few thoughts (no spoilers):

  • People are not rational agents and most decisions in life, including business, are based on emotions more than on anything else. (Known mantra but a nice reminder from the movie)
  • No matter where you are, there is always a “Leo Beebe” guy around (and sometimes more than one)
  • People live compartmentalized lives – meaning that there may be a lot of drama in one organization, but a completely different type of drama in the other – and those 2 organizations would not overlap at all. Therefore, the importance of each drama lies mostly within the group involved and nobody else cares. So being inside a drama, it is then important to zoom out your prospective and make a conscious decision whether this fight is any important to you. If you’re a professional racer and have a chance at winning Le Mans – then probably it is. On the other hand, if you’re struggling in a small poorly managed company pitted against a “Leo Beebe” – then probably it is not. Life is short and the only scenario when dealing with a “Leo Beebe” makes sense is when there is something really important to you around it.

P.S. Other great movies and series I recommend can be found here.

No good way to verify public image sha256 in docker hub – security concern

This is a little crazy but apparently we don’t have a good way to verify sha256 digests of public images in docker hub.

Related thread is here: https://github.com/docker/hub-feedback/issues/1925 and also this stackoverflow is useful: https://stackoverflow.com/questions/57316115/get-manifest-of-a-public-docker-image-hosted-on-docker-hub-using-the-docker-regi .

Problems in the nutshell:

  1. Publicly displayed digests on docker hub UI do not match those seen when pulling images locally
  2. Getting public image manifests is highly problematic (very hacky work-around involved)
  3. Public image may be re-pushed with same tag -> forcing new digest -> forcing details about last image erased. How we audit this to still be good?

Potentially, all those present serious level of security concern.

DevOps, DataOps in 2020 – Tectonic Shift

2020 is a remarkable year because how the things are going in DevOps and DataOps fields. Also let me mention DataOps challenges I listed a year ago here.Christmas Tree - DevOps and DataOps in 2020

To see where we are now I remind you of DORA’s State Of DevOps 2019 report (get your copy here if you haven’t done so yet) – and there were few other similar studies, that generally outline the trend that the amount of high performing software companies is in the range of ~5-15%. And low performing companies are at ~20%. What is stunning is there exists orders of magnitude difference between low and high performers.

As software becomes the core of every modern organization, this now translates to live or death situation for businesses. And medium performers are not spared as in many aspects they are also far behind high performers. Business can only sustain if it manages to embrace high performing software practices, starting with the mindset. This is what I’d like to call “Tectonic DevOps Shift” (many people refer to this process as the 4th Industrial Revolution).

It is important to note that high performers are also “not done yet”. Meaning that there is large room for improvement for most of them as it would take some time for everybody to settle on best practices. But looks like we’re getting close to at least good understanding what those best practices are.

So here are my thoughts about what’s most important to embrace at this time and what are the immediate trends in DevOps and DataOps that I see (will be a bit technical in parts):

1. Continuous Integration (CI) must be fully containerized

Essentially, whole build process must be described in a single Dockerfile and run with a single docker build command – thanks to multi-stage docker builds – for example, here is a sample how a build-stage maven file could look. This simplifies CI scripts to bare minimum and makes moving between CI platforms trivial.

2. Continuous Delivery (CD) must be fully containerized as well 

This is similar to previous point. However – I moved this to a separate point, since this could be done later (or not at all for older projects nearing retirement). Essentially, if your CI is containerized, you could still build your artifacts inside containers and then extract to legacy non-containerized environments.

3. Container orchestration as a key DevOps platform

Widespread adoption of Kubernetes, wide popularity of Docker Compose for POCs and demos and use of Docker Swarm for smaller projects allows for quick switching between various hosting options and cloud providers once containerization is fully adopted. Arguably, this field has still a lot of room to grow due to complexity or existing tools and frequently messy code and yaml mix – but the trend stays and this should be sorted out eventually.

4. Security is bigger than ever

Focus on cyber-security is very strong. DevSecOps field is a huge part of DevOps these days. To me particularly the main pain point is still how we move secrets to environments and manage those secrets. A lot of improvement has been made recently – with tools such as HashiCorp Vault and others, but still there is visible room to improvement.

On the other side, with Quantum Computing on the horizon there is an emerging demand to switch to Quantum-safe encryption. Most people understandably prefer to ignore this challenge for now, but it is a huge growing problem.

5. Templating 

Standard Dockerfiles, standard terraform scripts, standard cookbooks, playbooks – templates make it faster and easier to solve typical problems quickly. Several organizations are working to make this easier and organize already, but there is still room to grow.

6. DataOps – Data is like software but more fluid

Main requirement for DataOps projects is additional level of agility on top of traditional DevOps pipelining. Essentially, idea above about project build process living in a single Dockerfile servers that purpose of agility. As data is fluid there should be expectation of updating pipelines quickly and efficiently.

In this sense I hear many consultants saying that it takes large effort to build pipelines for the sake of ease of use and savings in the end. To me that doesn’t cut it any more – namely it’s missing a step of building pipelines themselves for agility. Meaning if you need to change pipeline later, that you should be able to do quickly. Achieving that level of excellence brings you closer to what successful DataOps practice needs.

7. Embracing adult organizations – moving to remote asynchronous workforce

 I have a strong belief that elementary school-like organizations, which believe that everybody must be in the same office to perform cannot really win the market if given strong adult competitors. In this sense adult organizations need to build trust among its people so they can perform asynchronously and remotely, which leads to huge advantages in a lot of sense. This process undergoes some degree of learning curve but in my experience remote workforce is already able to perform better in many instances.

8. Breaking silos by embracing transparency and clear communication lines 

This refers to the previous point of adult organizations – if we already have adult organization, this now allows for transparency and greater level of responsibility and commitment from every member of such organization. This finally leads for developers caring about end-user experience and not only about binary marking of ticket as “fixed”. I highly recommend “CEOs Should Tell It Like It Is” post by Ben Horowitz to further describe this idea.

9. Embracing DevOps and DataOps as core

Since the terms like Agile, DevOps, DataOps are so broad, some organizations trying to get better in releasing software and in business in general are lost when it comes what to do and where to start. There are two extremes that I see: 1st – everything we do must be Agile – which sometimes leads to comic situation where people lose common sense in the process. 2nd extreme – DevOps is something with infrastructure, CICD, security (essentially, technical stuff) – and it’s not our core, so we better hire consultants to do all that and focus on our core.

Both of those extremes usually don’t work well. My take – always start with common sense, DevOps starts with mindset – and mindset is the core of any business. Good consultants are good to quickly bring you up to speed and show you best practices and state of the art, but they can’t run the business for you.

At the same time, it is ok to outsource certain technical aspects (i.e., infrastructure monitoring and parts of incident management), as long as core thinking about how everything is wired together stays in-house. Also, my advice – is if you don’t know where to start, start with lead times measuring time between feature being dev complete and deployed to production. If you have instances where such time is measured in months, your organization is in big need of change.

Summary thoughts

In summary, I’d like to point that DevOps is half culture and half technical stuff. And that is roughly about how my points above got split. I’d also say that culture is arguably more important – since the right culture leads to the right technical decisions eventually, but the opposite is not always true. So always think about culture and embrace that DevOps is at the core of any software organization – which is essentially any organization out there these days 😉

Gene Kim’s “The Unicorn Project” – my view

“The Unicorn Project” by Gene Kim finally became generally available last week, and I took couple of days while stuck in Toronto to read it. The book describes same events as the DevOps classic – Gene Kim’s “The Phoenix Project”. At least because of that “The Unicorn Project” was a must read in the top of my reading list.

“The Unicorn Project” is largely orthogonal to “The Phoenix Project” as it shows different angle of the same events. However, almost 7 years have passed since “The Phoenix Project” was published – so what’s new in “The Unicorn Project”? Below are my most significant impressions after reading:

  • “The Unicorn Project” is much more female-centric, which is great and shows (hopefully) shifting trends in IT industry.
  • Psychological safety is now included as one of The Five Ideals. The Five Ideals is in turn one of the key new concepts introduced in the book. To me, psychological safety and more generally – humanity in teams – is the most important thing that changed in today’s perception of how organizations work relative to when I started my career as a management consultant in the early 2000s.
  • Sarah (the villain) did something good. She’s still the villain but the fact is – she is now more human also. Usually, the villains like Sarah appear in real life as the result of the Peter principle (where people promoted to the level of their incompetence) – which is way too common. So she could be a great employee at a lower level role, but instead was bringing a lot of harm in an executive position.
  • Business analytics and DataOps got a spot in the book.
  • More details were added about the actual mechanics of digital transformation (relative to “The Phoenix Project”). This is a great response to certain concerns that were raised past “The Phoenix Project” by some people – those people were mainly appealing that such quick organizational transformation is largely unrealistic, especially given the timeline. While the timeline is now included in the book, to me the most important is the sequence of events and their more detailed description – which was now added.
  • The now-popular concept of “Jobs to be done” and customer orientation in general was highlighted in the book. “Why” gets its prominent place – not only “what” and “how” – which is great. “The Unicorn Project” also encourages experimentation mentioning ideas for which I also highly recommend Kent Beck’s article.

There are also few things that I’d like to see more of (or maybe explored in the later books):

  • The concept of remote and asynchronous work was largely missing and mentioned only briefly. On the contrary, some focus was made on collocation of the teams (which I agree is a great first step – but only a first one). With the book focusing on team productivity (as “The Unicorn Project” is) – asynchronous remote work is a wildly important concept nowadays. As we’re moving to more mature organizations, we should embrace remote work. Maturity enables remoteness and naturally leads to it.
  • While DataOps got a spot, it was largely shown as a black box, with few exceptions. The internals of DataOps were largely skipped – that is data modelling, managing those models and unlocking usefulness in data.

Overall, “The Unicorn Project” does a great job of highlighting many of the latest developments in DevOps and IT industry as a whole. It’s an easy read and I truly enjoyed and loved it 🙂

No-frills secret sharing with openssl

Motivation

Sometimes we need to share a secret with a colleague, and frequently it’s a hassle to do so securely. Worst options include people simply sending plain-text secrets over email or slack. Better, if this is some sort of encrypted email service like ProtonMail, but still it’s a fairly brittle way if we’re dealing with something sensitive.
More secure (and more hassle) options include leveraging cloud vault solutions, GPG over email, using Shamir’s Secret Sharing, i.e. ssss linux tool. While those options are nice, sometimes the amount of hassle is just too much. Also speaking about vaults, frequently you don’t want to give any authorization on particular vault to the person with whom you’re sharing particular secret.

The solution I’m opting to use frequently and want describe here involves encrypting the secret via openssl with one-time password. The idea is that now we have our secret represented as 2 pieces (encrypted secret + one-time password), and then we send those pieces over 2 different channels (i.e., email and sms). The benefit of this solution is that a) the secret is not sent in plain text anywhere; b) compromising any one of those channels is not enough to retrieve the secret; c) the method is fairly simple and straightforward as shown below.

Openssl way walkthrough

  1. Create or generate a secure one-time password, i.e. using openssl -rand 30 | base64
  2. Encrypt your secret with openssl using:
    echo 'my secret' | openssl aes-256-cbc -a -pbkdf2 -salt
    then input your one-time password generated in p.1 when prompted (you need to do it twice – first time and during verification)
    You will obtain base64 output of your encrypted secret.
  3. Send output from p.2 and one-time password from p.1 to your recipient via different channels. I.e., send encrypted secret via email and password via slack.
  4. To decrypt your recipient would do:
    echo 'base64 encrypted secret' | openssl aes-256-cbc -a -d -pbkdf2 -salt
    and would need to supply one-time password when prompted.