Automated Tests CI/CD Workflow With Reliza Hub

We have recently extended Reliza Hub to support Automated QA functionality as a part of CI/CD pipeline. In this post I’m going to describe how we are dogfooding it.

Setup – we have a set of behave tests (behave – is a python implementation of Cucumber BDD).

Our stack is running on kubernetes, so the strategy we are going to use is the following. We will run our test container as a cron job, it would connect to Reliza Hub to check if the approval of specified type is needed for the deployed release – that said whether we need to run the actual test suite. Running the test suite may not be needed in case if the approval has already been given or if the release has previously been rejected for any reason.

We are also running Reliza Watcher to stream details of our deployed releases to production Reliza Hub. This is how Reliza Hub knows which releases are currently deployed on our test instance and manage their properties and match approval status.

To achieve the integration with CI/CD pipeline, we go through the following steps:

First, we dockerize the tests using technique from this project.

Once dockerized, we now need to ensure that the tests run on our test instance on new deployments. And that only the approved release may be promoted to our staging / production instances.

To achieve this, we add QA_AUTO approval requirement in our approval matrix on Reliza Hub. Note that Approval Matrix is located on the bottom of the Settings page in Reliza Hub.

Next, we create an approval key (using the plus button right above the approval matrix). Then record key ID and secrect, then edit the key and give it QA_AUTO approval permissions.

Next, we modify our Dockerfile to include reliza-cli component so we could interact with Reliza Hub using set of commands defined in Reliza CLI:

FROM relizaio/reliza-cli as clistage

FROM williamyeh/behave as artifact
WORKDIR /usr/src/app/reliza-qa/behave-tests
.... # some irrelevant build commands here
COPY --from=clistage /app/app /app/reliza-cli
COPY wrapper.sh /tmp/
RUN chmod 0700 /tmp/wrapper.sh

Finally, we modify wrapper.sh to interact with Reliza Hub as following:

# check if we need to handle approvals via Reliza Hub

if [ -z "$RELIZA_API_ID" ]; then
    # not reliza id provided, simply execute behave
    exec behave "$@"
else
    # check if approval is needed and we need to run the test
    approval_check=$(/app/reliza-cli isapprovalneeded -i $RELIZA_API_ID -k $RELIZA_API_KEY --approval $RELIZA_APPROVAL_TYPE --instance $RELIZA_INSTANCE --project $RELIZA_PROJECT)

    if [ $approval_check == '{"isNeeded":true}' ];
    then
        echo "Approval needed, performing behave tests"
        # execute behave
        behave "$@"
        
        # check exit status and either approve or disapprove release based on that
        test_status=$(echo $?)
        approve_status=""
        if [ $test_status -ne 0 ];
        then
            approve_status="--disapprove"
        fi
        /app/reliza-cli approverelease  -i $RELIZA_API_ID -k $RELIZA_API_KEY --approval $RELIZA_APPROVAL_TYPE --instance $RELIZA_INSTANCE --project $RELIZA_PROJECT $approve_status
    else
        echo $approval_check
        echo "Approval not needed, skipping behave tests"
    fi
fi

Last thing we are going to add this image to our test instance as a cron job. We would use the following definition:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: reliza-behave-tests
  namespace: tests
spec:
  schedule: "*/2 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: reliza-behave-tests
            image: reliza-behave-image
            env:
              - name: RELIZA_API_ID
                valueFrom:
                  secretKeyRef:
                    name: reliza-behave
                    key: reliza-api-id
              - name: RELIZA_API_KEY
                valueFrom:
                  secretKeyRef:
                    name: reliza-behave
                    key: reliza-api-key
              - name: RELIZA_APPROVAL_TYPE
                value: QA_AUTO
              - name: RELIZA_INSTANCE
                value: relizahub.com
              - name: RELIZA_PROJECT
                valueFrom:
                  secretKeyRef:
                    name: reliza-behave
                    key: reliza-project

# kubectl create secret generic reliza-behave -n tests --from-literal=reliza-api-id=APPROVAL_id --from-literal=reliza-api-key=key --from-literal=reliza-project=projectid

So every 2 minutes we would run a cron job on k8s, that would check first if specific approval type (given by our test suite) is needed for out test instance. If not, nothing else will happen. If yes, it will run the test suite and either approve or disapprove the release.

Only approved releases may move to the subsequent stages of our pipeline (that is uat, staging or production).

Future work: later on we are planning to support webhooks or triggers to run automated tests rather than relying on cron jobs.