Automated QA Tests CI/CD Workflow With Reliza Hub

UPDATE 2023-07-16: Added details on uploading test report to release on Reliza Hub.

We have recently extended Reliza Hub to support Automated QA functionality as a part of CI/CD pipeline. In this post I’m going to describe how we are dogfooding it.

Base Setup

We have a set of behave tests (behave – is a python implementation of Cucumber BDD).

Our stack is running on Kubernetes, so the strategy we are going to use is the following. We will run our test container as a cron job, it would connect to Reliza Hub to check if the approval of specified type is needed for the deployed release – that said whether we need to run the actual test suite. Running the test suite may not be needed in case if the approval has already been given or if the release has previously been rejected for any reason.

We are also running Reliza Watcher to stream details of our deployed releases to production Reliza Hub. This is how Reliza Hub knows which releases are currently deployed on our test instance and manage their properties and match approval status.

Integration with CI/CD

To achieve the integration with CI/CD pipeline, we go through the following steps:

First, we dockerize the tests using technique from this project.

Once dockerized, we now need to ensure that the tests run on our test instance on new deployments. And that only the approved release may be promoted to our staging / production instances.

To achieve this, we add QA_AUTO approval requirement in our approval matrix on Reliza Hub. Note that Approval Matrix is located on the bottom of the Settings page in Reliza Hub.

Next, we create an approval key (using the plus button right above the approval matrix). Then record key ID and secrect, then edit the key and give it QA_AUTO approval permissions.

Next, we modify our Dockerfile to include reliza-cli component so we could interact with Reliza Hub using set of commands defined in Reliza CLI:

FROM relizaio/reliza-cli as clistage

FROM python:3.9.5-buster as artifact
WORKDIR /usr/src/app/reliza-qa/behave-tests
RUN pip install uuid
RUN pip install requests
RUN pip install behave
RUN apt-get update && apt-get install -y libpq-dev gcc
RUN pip3 install psycopg2
RUN apt-get autoremove -y gcc
COPY . /usr/src/app/reliza-qa
COPY --from=clistage /app/app /app/reliza-cli
COPY wrapper.sh /tmp/
RUN chmod 0700 /tmp/wrapper.sh
ENTRYPOINT ["/tmp/wrapper.sh"]

Finally, we modify wrapper.sh to interact with Reliza Hub as following:

# check if we need to handle approvals via Reliza Hub

if [ -z "$RELIZA_API_ID" ]; then
    # no reliza id provided, simply execute behave
    exec behave "$@"
else
    # check if approval is needed and we need to run the test
    approval_check=$(/app/reliza-cli isapprovalneeded -i $RELIZA_API_ID -k $RELIZA_API_KEY --approval $RELIZA_APPROVAL_TYPE --instance $RELIZA_INSTANCE --project $RELIZA_PROJECT)

    if [ $approval_check == '{"isNeeded":true}' ];
    then
        echo "Approval needed, performing behave tests"
        # execute behave
        test_report_file_name=test-report-$(date -I).txt
        behave "$@" -o test-output/$test_report_file_name
        
        # check exit status and either approve or disapprove release based on that
        test_status=$(echo $?)
        approve_status=""
        if [ $test_status -ne 0 ];
        then
            approve_status="--disapprove"
        fi
        /app/reliza-cli approverelease  -i $RELIZA_API_ID -k $RELIZA_API_KEY --approval $RELIZA_APPROVAL_TYPE --instance $RELIZA_INSTANCE --project $RELIZA_PROJECT $approve_status
        /app/reliza-cli addDownloadableArtifact  -i "$RELIZA_API_ID" -k "$RELIZA_API_KEY" -u "$RH_URI_TO_STREAM_RESULTS" --instance "$RELIZA_INSTANCE" --project "$RELIZA_PROJECT" --artifactType "TEST_REPORT" -f "test-output/$test_report_file_name"
    else
        echo $approval_check
        echo "Approval not needed, skipping behave tests"
    fi
fi

Set up QA Test Cron Job on Test Instance

Last thing we are going to add this image to our test instance as a cron job. We would use the following definition:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: reliza-behave-tests
  namespace: tests
spec:
  schedule: "*/1 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: reliza-behave-tests
            image: reliza-behave-image
            env:
              - name: RELIZA_API_ID
                valueFrom:
                  secretKeyRef:
                    name: reliza-behave
                    key: reliza-api-id
              - name: RELIZA_API_KEY
                valueFrom:
                  secretKeyRef:
                    name: reliza-behave
                    key: reliza-api-key
              - name: RELIZA_APPROVAL_TYPE
                value: QA_AUTO
              - name: RELIZA_INSTANCE
                value: relizahub.com
              - name: RH_URI_TO_STREAM_RESULTS
                value: https://app.relizahub.com
              - name: RELIZA_PROJECT
                valueFrom:
                  secretKeyRef:
                    name: reliza-behave
                    key: reliza-project

# kubectl create secret generic reliza-behave -n tests --from-literal=reliza-api-id=APPROVAL_id --from-literal=reliza-api-key=key --from-literal=reliza-project=projectid

So every minute we would run a cron job on k8s, that would check first if a specific approval type (given by our test suite) is needed for out test instance. If not, nothing else will happen. If yes, it will run the test suite and either approve or disapprove the release.

Also, this will upload the test report to our release in Reliza Hub so it can be viewed from there. This is achieved by the following line in the wrapper above:

 /app/reliza-cli addDownloadableArtifact  -i "$RELIZA_API_ID" -k "$RELIZA_API_KEY" -u "$RH_URI_TO_STREAM_RESULTS" --instance "$RELIZA_INSTANCE" --project "$RELIZA_PROJECT" --artifactType "TEST_REPORT" -f "test-output/$test_report_file_name"

The documentation for adding downloadable artifacts can be found in Reliza CLI GitHub here.

Only approved releases may move to the subsequent stages of our pipeline (that is uat, staging or production).

Read more here for setting promotion between environments in Reliza Hub.