In this article, you will learn how to create preview environments for development purposes on Kubernetes with ArgoCD. Preview environments are quickly gaining popularity. This approach allows us to generate an on-demand namespace for testing a specific git branch before it’s merged. Sometimes we are also calling that approach “ephemeral environments” since they are provisioned only for a limited time. Several ways and tools may help in creating preview environments on Kubernetes. But if we use the GitOps approach in the CI/CD process it is worth considering ArgoCD for that. With ArgoCD and Helm charts, it is possible to organize that process in a fully automated and standardized way.

You can find several posts on my blog about ArgoCD and continuous delivery on Kubernetes. For a quick intro to CI/CD process with Tekton and ArgoCD, you can refer to the following article. For a more advanced approach dedicated to database management in the CD process see the following post.

Prerequisites

In order to do the exercise, you need to have a Kubernetes cluster. Then you need to install the tools we will use today – ArgoCD and Tekton. Here are the installation instructions for Tekton Pipelines and Tekton Triggers. Tekton is optional in our exercise. We will just use it to build the application image after pushing the commit to the repository.

ArgoCD is the key tool today. We can use the official Helm chart to install it on Kubernetes. Firstly. let’s add the following Helm repository:

$ helm repo add argo https://argoproj.github.io/argo-helm

After that, we can install ArgoCD in the current Kubernetes cluster in the argocd namespace using the following command:

$ helm install my-argo-cd argo/argo-cd -n argocd

Once we install ArgoCD, we can display a list of running pods. You should have a similar result to mine:

$ kubectl get pod
openshift-gitops-application-controller-0                     1/1     Running     0          1m
openshift-gitops-applicationset-controller-654f99c9b4-pwnc2   1/1     Running     0          1m
openshift-gitops-dex-server-5dc77fcb7d-6tkg5                  1/1     Running     0          1m
openshift-gitops-redis-87698688c-r59zf                        1/1     Running     0          1m
openshift-gitops-repo-server-5f6f7f4996-rfdg8                 1/1     Running     0          1m
openshift-gitops-server-dcf746865-tlmlp                       1/1     Running     0          1m

Finally, you also need to have an account on GitHub. In our scenario, ArgoCD will require access to the repository to obtain a list of opened pull requests. Therefore, we need to create a personal access token for authentication over GitHub. In your GitHub profile go to Settings > Developer Settings > Personal access tokens. Choose Tokens (classic) and then click the Generate new token button. Then you should enable the repo scope. Of course, you need to save the value of the generated token. We will create a secret on Kubernetes using that value.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. To do that you need to clone my two GitHub repositories. First of them, contains the source code of our sample app written in Kotlin. The second of them contains configuration managed by ArgoCD with YAML manifests for creating preview environments on Kubernetes. Finally, you should just follow my instructions.

How it works

Let’s describe our scenario. There are two repositories on GitHub. In the repository with app source code, we are creating branches for working on new features. Usually, when we are starting a new branch we are just at the beginning of our work. Therefore, we don’t want to deploy it anywhere. Once we make progress and we have a version for testing, we are creating a pull request. Pull request represents the relation between source and target branches. We may still push commits to the source branch. Once we merge a pull request all the commits from the source branch will also be merged.

After creating a new pull request we want ArgoCD to provision a new preview environment on Kubernetes. Once we merge a pull request we want ArgoCD to remove the preview environment automatically. Fortunately, ArgoCD can monitor pull requests with ApplicationSet generators. Our ApplicationSet will connect to the app source repository to detect new pull requests. However, it will use YAML manifests stored in a different, config repository. Those manifests contain a generic definition of our preview environments. They are written in Helm and may be shared across several different apps and scenarios. Here’s the diagram that illustrates our scenario. Let’s proceed to the technical details.

Using ArgoCD ApplicationSet and Helm Templates

ArgoCD requires access to the GitHub API to detect a current list of opened pull requests. Therefore we will create a Kubernetes Secret that contains our GitHub personal access token:

$ kubectl create secret generic github-token \
  --from-literal=token=<YOUR_GITHUB_PERSON_ACCESS_TOKEN>

In the config repository, we will define a template for our sample preview environment. It is available inside the preview directory. It contains the namespace declaration:

apiVersion: v1
kind: Namespace
metadata:
  name: {{ .Values.namespace }}

We are also defining Kubernetes Deployment for a sample app:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.name }}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {{ .Values.name }}
  template:
    metadata:
      labels:
        app: {{ .Values.name }}
    spec:
      containers:
      - name: {{ .Values.name }}
        image: quay.io/pminkows/{{ .Values.image }}:{{ .Values.version }}
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "1024Mi"
            cpu: "1000m"
        ports:
        - containerPort: 8080

Let’s also add the Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.name }}-service
spec:
  type: ClusterIP
  selector:
    app: {{ .Values.name }}
  ports:
  - port: 8080
    name: http-port

Finally, we can create the ArgoCD ApplicationSet with the Pull Request Generator. We are monitoring the app source code repository (1). In order to authenticate over GitHub, we are injecting the Secret containing access token (2). While the ApplicationSet targets the source code repository, the generated ArgoCD Application refers to the config repository (3). It also sets several Helm parameters. The name of the preview namespace is the same as the name of the branch with the preview prefix (4). The app image is tagged with the commit hash (5). We are also setting the name of the app image (6). All the configuration settings are applied automatically by ArgoCD (7).

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: sample-spring-preview
spec:
  generators:
    - pullRequest:
        github:
          owner: piomin
          repo: sample-spring-kotlin-microservice # (1)
          tokenRef:
            key: token
            secretName: github-token # (2)
        requeueAfterSeconds: 60
  template:
    metadata:
      name: 'sample-spring-{{branch}}-{{number}}'
    spec:
      destination:
        namespace: 'preview-{{branch}}'
        server: 'https://kubernetes.default.svc'
      project: default
      source:
        # (3)
        path: preview/
        repoURL: 'https://github.com/piomin/openshift-cluster-config.git'
        targetRevision: HEAD
        helm:
          parameters:
            # (4)
            - name: namespace
              value: 'preview-{{branch}}'
            # (5)
            - name: version
              value: '{{head_sha}}'
            # (6)
            - name: image
              value: sample-kotlin-spring
            - name: name
              value: sample-spring-kotlin
      # (7)
      syncPolicy:
        automated:
          selfHeal: true

Build Image with Tekton

ArgoCD is responsible for creating a preview environment on Kubernetes and applying the Deployment manifest there. However, we still need to build the image after a push to the source branch. In order to do that, we will create a Tekton pipeline. It’s a very simple pipeline. It just clones the repository and builds the image with the commit hash as a tag.

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: sample-kotlin-pipeline
spec:
  params:
    - description: branch
      name: git-revision
      type: string
  tasks:
    - name: git-clone
      params:
        - name: url
          value: 'https://github.com/piomin/sample-spring-kotlin-microservice.git'
        - name: revision
          value: $(params.git-revision)
        - name: sslVerify
          value: 'false'
      taskRef:
        kind: ClusterTask
        name: git-clone
      workspaces:
        - name: output
          workspace: source-dir
    - name: s2i-java-preview
      params:
        - name: PATH_CONTEXT
          value: .
        - name: TLSVERIFY
          value: 'false'
        - name: MAVEN_CLEAR_REPO
          value: 'false'
        - name: IMAGE
          value: >-
            quay.io/pminkows/sample-kotlin-spring:$(tasks.git-clone.results.commit)
      runAfter:
        - git-clone
      taskRef:
        kind: ClusterTask
        name: s2i-java
      workspaces:
        - name: source
          workspace: source-dir
  workspaces:
    - name: source-dir

This pipeline should be triggered by the push in the app repository. Therefore we have to create the TriggerTemplate and EventListener CRD objects.

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
  name: sample-kotlin-spring-trigger-template
  namespace: pminkows-cicd
spec:
  params:
    - default: master
      description: The git revision
      name: git-revision
    - description: The git repository url
      name: git-repo-url
  resourcetemplates:
    - apiVersion: tekton.dev/v1beta1
      kind: PipelineRun
      metadata:
        generateName: sample-kotlin-spring-pipeline-run-
      spec:
        params:
          - name: git-revision
            value: $(tt.params.git-revision)
        pipelineRef:
          name: sample-kotlin-pipeline
        serviceAccountName: pipeline
        workspaces:
          - name: source-dir
            persistentVolumeClaim:
              claimName: kotlin-pipeline-pvc
---
apiVersion: triggers.tekton.dev/v1alpha1
kind: EventListener
metadata:
  name: sample-kotlin-spring
spec:
  serviceAccountName: pipeline
  triggers:
    - bindings:
        - kind: ClusterTriggerBinding
          ref: github-push
      name: trigger-1
      template:
        ref: sample-kotlin-spring-trigger-template

After that Tekton automatically creates Kubernetes Service with the webhook for triggering the pipeline.

Finally, we need to set the webhook URL in our GitHub app repository. That’s all that we need to do. Let’s see how it works.

Kubernetes Preview Environments in Action

Creating environment

In the first step, we will create two branches in our sample GitHub repository: branch-a and branch-c. If we push some changes into each of those branches, our pipeline should be triggered by the webhook. It will build the image from the source branch and push it to the remote registry.

As you see, our pipeline was running two times.

Here’s the Quay registry with our sample app images. They are tagged using the commit hash.

Now, we can create pull requests for our branches. As you see I have two pull requests in the sample repository.

Let’s take a look at one of our pull requests. Firstly, pay attention to the pull request id (55) and a list of commits assigned to the pull request.

ArgoCD monitors a list of opened PRs via ApplicationSet. Each time it detects a new PR it creates a dedicated ArgoCD Application for synchronizing YAML manifests stored in the Git config repository with the target Kubernetes cluster. We have two opened PRs, so there are two applications in ArgoCD.

We can take a look at the ArgoCD Application details. As you see it creates the namespace containing Kubernetes Deployment and Service for our app.

Let’s display a list of running in one of our preview namespaces:

$ kubectl get po -n preview-branch-a
NAME                                    READY   STATUS    RESTARTS   AGE
sample-spring-kotlin-5c7cc45bc7-wck78   1/1     Running   0          22m

Let’s verify the tag of the image used in the pod:

Adding commits to the existing PR

What about making some changes in one of our preview branches? Our latest commit with the “Make some changes” title will be automatically included in the PR.

ArgoCD ApplicationSet will detect a commit in the pull request. Then it will update the ArgoCD Application with the latest commit hash (71d05d8). As a result, it will try to run a new pod containing the latest version of the app. In the meantime, our pipeline is building a new image staged by the commit hash. As you see, the image is not available yet.

Let’s display a list of running pods in the preview-branch-c namespace:

$ kubectl get po -n preview-branch-c
NAME                                    READY   STATUS             RESTARTS   AGE
sample-spring-kotlin-67f6947c89-xn2r8   1/1     Running            0          29m
sample-spring-kotlin-6d844c8c94-qjrr4   0/1     ImagePullBackOff   0          24s

Once the pipeline will finish the build, it pushes the image to the Quay registry:

And the latest version of the app from the branch-c is available on Kubernetes:

Now, you can close or merge the pull request. As a result, ArgoCD will automatically remove our preview namespace with the app.

Final Thoughts

In this article, I showed you how to create and manage preview environments on Kubernetes in the GitOps way with ArgoCD. In this concept, a preview environment exists on Kubernetes as long as the particular pull request lives in the GitHub repository. ArgoCD uses a global, generic template for creating such an environment. Thanks to that, we can have a single, shared process across the whole organization.