This article will teach you how to continuously promote application releases between environments on Kubernetes using the GitOps approach. Promotion between environments is one of the most challenging aspects in the continuous delivery process realized according to the GitOps principles. That’s because typically we manage that process independently per each environment by providing changes in the Git configuration repository. If we use Argo CD, it comes down to creating three Application CRDs that refer to different places or files inside a repository. Each application is responsible for synchronizing changes to e.g. a specified namespace in the Kubernetes cluster. In that case, a promotion is driven by the commit in the part of the repository responsible for managing a given environment. Here’s the diagram that illustrates the described scenario.

The solution to these challenges is Kargo, an open-source tool implementing continuous promotion within CI/CD pipelines. It provides a structured mechanism for promoting changes in complex environments involving Kubernetes and GitOps. I’ve been following Kargo for several months. It’s an interesting tool that provides stage-to-stage promotion using GitOps principles with Argo CD. It reached the version in October 2024. Let’s take a closer look at it.

If you are interested in promotion between environments on Kubernetes using the GitOps approach, you can read about another tool that tackles that challenge – Devtron. Here’s the link to my article that explains its concept.

Understand the Concept

Before we start with Kargo, we need to understand the concept around that tool. Let’s analyze several basic terms defined and implemented by Kargo.

A project is a bunch of related Kargo resources that describe one or more delivery pipelines. It’s the basic unit of organization and multi-tenancy in Kargo. Every Kargo project has its own cluster-scoped Kubernetes resource of type Project. We should put all the resources related to a certain project into the same Kubernetes namespace.

A stage represents environments in Kargo. Stages are the most important concept in Kargo. We can link them together in a directed acyclic graph to describe a delivery pipeline. Typically, a delivery pipeline starts with a test or dev stage and ends with one or more prod stages.

A Freight object represents resources that Kargo promotes from one stage to another. It can reference one or more versioned artifacts, such as container images, Kubernetes manifests loaded from Git repositories, or Helm charts from chart repositories. 

A warehouse is a source of freight. It can refer to container image repositories, Git, or Helm chart repositories.

In that context, we should treat a promotion as a request to move a piece of freight into a specified stage.

Source Code

If you want to try out this exercise, go ahead and take a look at my source code. To do that, just clone my GitHub repository. It contains the sample Spring Boot application in the basic directory. The application exposes a single REST endpoint that returns the app Maven version number. Go to that directory. Then, you can follow my further instructions.

Kargo Installation

A few things must be ready before installing Kargo. We must have a Kubernetes cluster and the helm CLI installed on our laptop. I use Minikube. Kargo integrates with Cert-Manager, Argo CD, and Argo Rollouts. We can install all those tools using official Helm charts. Let’s begin with Cert-Manager. First, we must add the jetstack Helm repository:

helm repo add jetstack https://charts.jetstack.io

Here’s the helm command that installs it in the cert-manager namespace.

helm install cert-manager --namespace cert-manager jetstack/cert-manager \
  --set crds.enabled=true \
  --set crds.keep=true

To install Argo CD, we must first add the following Helm repository:

helm repo add argo https://argoproj.github.io/argo-helm

Then, we can install Argo CD in the argocd namespace.

helm install argo-cd argo/argo-cd

The Argo Rollouts chart is located in the same Helm repository as the Argo CD chart. We will also install it in the argocd namespace:

helm install my-argo-rollouts argo/argo-rollouts

Finally, we can proceed to the Kargo installation. We will install it in the kargo namespace. The installation command sets two Helm parameters. We should set the Bcrypt password hash and a key used to sign JWT tokens for the admin account:

helm install kargo \
  oci://ghcr.io/akuity/kargo-charts/kargo \
  --namespace kargo \
  --create-namespace \
  --set api.adminAccount.passwordHash='$2y$10$xu2U.Ux5nV5wKmerGcrDlO261YeiTlRrcp2ngDGPxqXzDyiPQvDXC' \
  --set api.adminAccount.tokenSigningKey=piomin \
  --wait

We can generate and print the password hash using e.g. htpasswd. Here’s the sample command:

htpasswd -nbB admin 123456

After installation is finished, we can verify it by displaying a list of pods running in the kargo namespace.

$ kubectl get po -n kargo
NAME                                          READY   STATUS    RESTARTS   AGE
kargo-api-dbb4d5cb7-zvnc6                     1/1     Running   0          44s
kargo-controller-c4964bbb7-4ngnv              1/1     Running   0          44s
kargo-management-controller-dc5569759-596ch   1/1     Running   0          44s
kargo-webhooks-server-6df6dd58c-g5jlp         1/1     Running   0          44s

Kargo provides a UI dashboard that allows us to display and manage continuous promotion configuration. Let’s expose it locally on the 8443 port using a port-forward feature:

kubectl port-forward svc/kargo-api -n kargo 8443:443

Once we sign in to the dashboard using the admin password set during an installation we can create a new kargo-demo project:

Sample Application

Our sample application is simple. It exposes a single GET /basic/ping endpoint that returns the version number read from Maven pom.xml.

@RestController
@RequestMapping("/basic")
public class BasicController {

    @Autowired
    Optional<BuildProperties> buildProperties;

    @GetMapping("/ping")
    public String ping() {
        return "I'm basic:" + buildProperties.orElseThrow().getVersion();
    }
}

We will build an application image using the Jib Maven Plugin. It is already configured in Maven pom.xml. I set my Docker Hub as the target registry, but you can relate it to your account.

<plugin>
  <groupId>com.google.cloud.tools</groupId>
  <artifactId>jib-maven-plugin</artifactId>
  <version>3.4.4</version>
  <configuration>
    <container>
      <user>1001</user>
    </container>
    <to>
      <image>piomin/basic:${project.version}</image>
    </to>
  </configuration>
</plugin>

The following Maven command builds the application and its image from a source code. Before the build, we should increase the version number in the project.version field in pom.xml. We begin with the 1.0.0 version, which should be pushed to your registry before proceeding.

mvn clean package -DskipTests jib:build

Here’s the result of my initial build.

Let’s switch to the Docker Registry dashboard after pushing the 1.0.0 version.

Configure Kargo for Promotion on Kubernetes

First, we will create the Kargo Warehouse object. It refers to the piomin/basic repository containing the image with our sample app. The Warehouse object is responsible for discovering new image tags pushed into the registry. We also use my Helm chart to deploy the image to Kubernetes. However, we will only use the latest version of that chart. Otherwise, we should also place that chart inside the basic Warehouse object to enable new chart version discovery.

apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
 name: basic
 namespace: demo
spec:
 subscriptions:
 - image:
     discoveryLimit: 5
     repoURL: piomin/basic

Then, we will create the Argo CD ApplicationSet to generate an application per environment. There are three environments: test, uat, prod. Each Argo CD application must be annotated by kargo.akuity.io/authorized-stage containing the project and stage name. Argo uses multiple sources. The argocd-showcase repository contains Helm values files with parameters per each stage. The piomin.github.io/helm-charts repository provides the spring-boot-api-app Helm chart that refers to those values.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
 name: demo
 namespace: argocd
spec:
 generators:
 - list:
     elements:
     - stage: test
     - stage: uat
     - stage: prod
 template:
   metadata:
     name: demo-{{stage}}
     annotations:
       kargo.akuity.io/authorized-stage: demo:{{stage}}
   spec:
     project: default
     sources:
       - chart: spring-boot-api-app
         repoURL: 'https://piomin.github.io/helm-charts/'
         targetRevision: 0.3.8
         helm:
           valueFiles:
             - $values/values/values-{{stage}}.yaml
       - repoURL: 'https://github.com/piomin/argocd-showcase.git'
         targetRevision: HEAD
         ref: values
     destination:
       server: https://kubernetes.default.svc
       namespace: demo-{{stage}}
     syncPolicy:
       syncOptions:
       - CreateNamespace=true

Now, we can proceed to the most complex element of our exercise – stage creation. The Stage object refers to the previously created Warehouse object to request freight to promote. Then, it defines the steps to perform during the promotion process. Kargo’s promotion steps define the workflow of a promotion process. They do the things needed to promote a piece of freight into the next stage. We can use several built-in steps that cover the most common operations like cloning Git repo, updating Helm values, or pushing changes to the remote repository. Our stage definition contains five steps. After cloning the repository with Helm values, we must update the image.tag parameter in the values-test.yaml file with the tag value read from the basic Warehouse. Then, Kargo commits and pushes changes to the configuration repository and triggers Argo CD application synchronization.

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
 name: test
 namespace: demo
spec:
 requestedFreight:
 - origin:
     kind: Warehouse
     name: basic
   sources:
     direct: true
 promotionTemplate:
   spec:
     vars:
     - name: gitRepo
       value: https://github.com/piomin/argocd-showcase.git
     - name: imageRepo
       value: piomin/basic
     steps:
       - uses: git-clone
         config:
           repoURL: ${{ vars.gitRepo }}
           checkout:
           - branch: master
             path: ./out
       - uses: helm-update-image
         as: update-image
         config:
           path: ./out/values/values-${{ ctx.stage }}.yaml
           images:
           - image: ${{ vars.imageRepo }}
             key: image.tag
             value: Tag
       - uses: git-commit
         as: commit
         config:
           path: ./out
           messageFromSteps:
           - update-image
       - uses: git-push
         config:
           path: ./out
       - uses: argocd-update
         config:
           apps:
           - name: demo-${{ ctx.stage }}
             sources:
             - repoURL: ${{ vars.gitRepo }}
               desiredRevision: ${{ outputs.commit.commit }}

Here’s the values-test.yaml file in the Argo CD configuration repository.

image:
  repository: piomin/basic
  tag: 1.0.0
app:
  name: basic
  environment: test

Here’s the Stage definition of the uat environment. It is pretty similar to the definition of the test environment. It just defines the previous source stage to test.

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: uat
  namespace: demo
spec:
 requestedFreight:
 - origin:
     kind: Warehouse
     name: basic
   sources:
     stages:
       - test
 promotionTemplate:
   spec:
     vars:
     - name: gitRepo
       value: https://github.com/piomin/argocd-showcase.git
     - name: imageRepo
       value: piomin/basic
     steps:
       - uses: git-clone
         config:
           repoURL: ${{ vars.gitRepo }}
           checkout:
           - branch: master
             path: ./out
       - uses: helm-update-image
         as: update-image
         config:
           path: ./out/values/values-${{ ctx.stage }}.yaml
           images:
           - image: ${{ vars.imageRepo }}
             key: image.tag
             value: Tag
       - uses: git-commit
         as: commit
         config:
           path: ./out
           messageFromSteps:
           - update-image
       - uses: git-push
         config:
           path: ./out
       - uses: argocd-update
         config:
           apps:
           - name: demo-${{ ctx.stage }}
             sources:
             - repoURL: ${{ vars.gitRepo }}
               desiredRevision: ${{ outputs.commit.commit }}

Perform Promotion Process

Once a new image tag is published to the registry, it becomes visible in the Kargo Dashboard. We must click the “Promote into Stage” button to promote a selected version to the target stage.

Then we should specify the source image tag. A choice is obvious since we only have the image tagged with 1.0.0. After approving the selection by clicking the “Yes” button Kargo starts a promotion process on Kubernetes.

After a while, we should have a new image promoted to the test stage.

Let’s repeat the promotion process of the 1.0.0 version for the other two stages. Each stage should be at a healthy status. That status is read directly from the corresponding Argo CD Application.

Let’s switch to the Argo CD dashboard. There are three applications.

We can make a test call of the sample application HTTP endpoint. Currently, all the environments run the same 1.0.0 version of the app. Let’s enable port forwarding for the basic service in the demo-prod namespace.

kubectl port-forward svc/basic -n demo-prod 8080:8080

The endpoint returns the application name and version as a response.

$ curl http://localhost:8080/basic/ping
I'm basic:1.0.0

Then, we will build another three versions of the basic application beginning from 1.0.1 to 1.0.5.

Once we push each version to the registry, we can refresh the list of images. With the default configuration, Kargo should add the latest version to the list. After pushing the 1.0.3 version I promoted it to the test stage. Then I refreshed a list after pushing the 1.0.4 tag. Now, the 1.0.3 tag can be promoted to a higher environment. In the illustration below, I’m promoting it to the uat stage.

After that, we can promote the 1.0.4 version to the test stage, and refresh a list of images once again to see the currently pushed 1.0.5 tag.

Here’s another promotion. This time, I moved the 1.0.3 version to the prod stage.

After clicking on the image tag tile, we will see its details. For example, the 1.0.3 tag has been verified on the test and uat stages. There is also an approval section. However, we still didn’t approve freight. To do that, we need to switch to the kargo CLI.

The kargo CLI binary for a particular OS on the project GitHub releases page. We must download and copy it to the directory under the PATH. Then, we can sign in to the Kargo server running on Kubernetes using admin credentials.

kargo login https://localhost:8443 --admin \
  --password 123456 \
  --insecure-skip-tls-verify

We can approve a specific freight. Let’s display a list of Freight objects.

$ kubectl get freight -n demo
NAME                                       ALIAS            ORIGIN (KIND)   ORIGIN (NAME)   AGE
0501f8d8018a953821ea437078c1ec34e6db5a6b   ideal-horse      Warehouse       basic           8m7s
683be41b1cef57ed755fc7a0f8e8d7776f90c63a   wiggly-tuatara   Warehouse       basic           11m
6a1219425ddeceabfe94f0605d7a5f6d9d20043e   ulterior-zebra   Warehouse       basic           13m
f737178af6492ea648f85fc7d082e34b7a085927   eager-snail      Warehouse       basic           7h49m

The kargo approve command takes Freight ID as the input parameter.

kargo approve --freight 6a1219425ddeceabfe94f0605d7a5f6d9d20043e \
  --stage uat \
  --project demo

Now, the image tag details window should display the approved stage name.

Let’s enable port forwarding for the basic service in the demo-uat namespace.

kubectl port-forward svc/basic -n demo-uat 8080:8080

Then we call the /basic/ping endpoint to check out the current version.

$ curl http://localhost:8080/basic/ping
I'm basic:1.0.3

Final Thoughts

This article explains the idea behind continuous app promotion between environments on Kubernetes with Kargo and Argo CD. Kargo is a relatively new project in the Kubernetes ecosystem, that smoothly addresses the challenges related to GitOps promotion. It seems promising. I will closely monitor the further development of this project.