We have recently extended Reliza Hub to support Automated QA functionality as a part of CI/CD pipeline. In this post I’m going to describe how we are dogfooding it.
Setup – we have a set of behave tests (behave – is a python implementation of Cucumber BDD).
Our stack is running on kubernetes, so the strategy we are going to use is the following. We will run our test container as a cron job, it would connect to Reliza Hub to check if the approval of specified type is needed for the deployed release – that said whether we need to run the actual test suite. Running the test suite may not be needed in case if the approval has already been given or if the release has previously been rejected for any reason.
We are also running Reliza Watcher to stream details of our deployed releases to production Reliza Hub. This is how Reliza Hub knows which releases are currently deployed on our test instance and manage their properties and match approval status.
To achieve the integration with CI/CD pipeline, we go through the following steps:
First, we dockerize the tests using technique from this project.
Once dockerized, we now need to ensure that the tests run on our test instance on new deployments. And that only the approved release may be promoted to our staging / production instances.
To achieve this, we add QA_AUTO approval requirement in our approval matrix on Reliza Hub. Note that Approval Matrix is located on the bottom of the Settings page in Reliza Hub.

Next, we create an approval key (using the plus button right above the approval matrix). Then record key ID and secrect, then edit the key and give it QA_AUTO approval permissions.

Next, we modify our Dockerfile to include reliza-cli component so we could interact with Reliza Hub using set of commands defined in Reliza CLI:
FROM relizaio/reliza-cli as clistage
FROM williamyeh/behave as artifact
WORKDIR /usr/src/app/reliza-qa/behave-tests
.... # some irrelevant build commands here
COPY --from=clistage /app/app /app/reliza-cli
COPY wrapper.sh /tmp/
RUN chmod 0700 /tmp/wrapper.sh
Finally, we modify wrapper.sh to interact with Reliza Hub as following:
# check if we need to handle approvals via Reliza Hub
if [ -z "$RELIZA_API_ID" ]; then
# not reliza id provided, simply execute behave
exec behave "$@"
else
# check if approval is needed and we need to run the test
approval_check=$(/app/reliza-cli isapprovalneeded -i $RELIZA_API_ID -k $RELIZA_API_KEY --approval $RELIZA_APPROVAL_TYPE --instance $RELIZA_INSTANCE --project $RELIZA_PROJECT)
if [ $approval_check == '{"isNeeded":true}' ];
then
echo "Approval needed, performing behave tests"
# execute behave
behave "$@"
# check exit status and either approve or disapprove release based on that
test_status=$(echo $?)
approve_status=""
if [ $test_status -ne 0 ];
then
approve_status="--disapprove"
fi
/app/reliza-cli approverelease -i $RELIZA_API_ID -k $RELIZA_API_KEY --approval $RELIZA_APPROVAL_TYPE --instance $RELIZA_INSTANCE --project $RELIZA_PROJECT $approve_status
else
echo $approval_check
echo "Approval not needed, skipping behave tests"
fi
fi
Last thing we are going to add this image to our test instance as a cron job. We would use the following definition:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: reliza-behave-tests
namespace: tests
spec:
schedule: "*/2 * * * *"
concurrencyPolicy: Forbid
jobTemplate:
spec:
template:
spec:
containers:
- name: reliza-behave-tests
image: reliza-behave-image
env:
- name: RELIZA_API_ID
valueFrom:
secretKeyRef:
name: reliza-behave
key: reliza-api-id
- name: RELIZA_API_KEY
valueFrom:
secretKeyRef:
name: reliza-behave
key: reliza-api-key
- name: RELIZA_APPROVAL_TYPE
value: QA_AUTO
- name: RELIZA_INSTANCE
value: relizahub.com
- name: RELIZA_PROJECT
valueFrom:
secretKeyRef:
name: reliza-behave
key: reliza-project
# kubectl create secret generic reliza-behave -n tests --from-literal=reliza-api-id=APPROVAL_id --from-literal=reliza-api-key=key --from-literal=reliza-project=projectid
So every 2 minutes we would run a cron job on k8s, that would check first if specific approval type (given by our test suite) is needed for out test instance. If not, nothing else will happen. If yes, it will run the test suite and either approve or disapprove the release.
Only approved releases may move to the subsequent stages of our pipeline (that is uat, staging or production).
Future work: later on we are planning to support webhooks or triggers to run automated tests rather than relying on cron jobs.
1 comment