Continuous Delivery on Kubernetes with Argo CD and Datree
In this article, you will learn how to use Datree with Argo CD to validate Kubernetes manifests in your continuous delivery process. I have already introduced the Datree tool in one of my previous articles about CI/CD with Tekton here. So, if you need to learn about basics please refer to that article or to the Datree quickstart.
With Datree you can automatically detect misconfigurations in Kubernetes manifests. Argo CD allows you to manage the continuous delivery process on Kubernetes declaratively using the GitOps approach. It can apply the manifests stored in a Git repository to a Kubernetes cluster. But, what about validation? Argo CD doesn’t have any built-in mechanisms for validating configuration taken from Git. This is where Datree comes in. Let’s see how to integrate both of these tools into your CD process.
Source Code
If you would like to try it by yourself, you can always take a look at my source code. In order to do that you need to clone my GitHub repository. This repository contains sample Helm charts, which Argo CD will use as Deployment templates. After that, you should just follow my instructions.
Introduction
We will deploy our sample image using two different Helm charts. Therefore, we will have two applications defined in ArgoCD per each Deployment. Our goal is to configure Argo CD to automatically use Datree as a Kubernetes manifest validation tool for all applications. To achieve that, we need to create the ArgoCD plugin for Datree. It is quite a similar approach to the case described in this article. However, there we used a ready plugin for integration between Argo CD and HashiCorp’s Vault.
There are two different ways to run a custom plugin in Argo CD. We can add the plugin config to the Argo CD main ConfigMap or run it as a sidecar to the argocd-repo-server Pod. In this article, we will use the first approach. This approach requires us to ensure that relevant binaries are available inside the argocd-repo-server pod. They can be added via volume mounts or using a custom image. We are going to create a custom image of the Argo CD that contains the Datree binaries.
Build a custom Argo CD that contains Datree
We will use the official Argo CD Helm chart for installation on Kubernetes. This chart is based on the following Argo CD image: quay.io/argoproj/argocd. In fact, the image is used by several components including argocd-server and argocd-repo-server. We are going to replace the default image for the argocd-repo-server:
In the first step, we will create a Dockerfile that extends a base image. We will install the Datree CLI there. Since Datree requires curl and unzip we also need to add them to the final image. Here’s our Dockerfile based on the latest version of the Argo CD image:
FROM quay.io/argoproj/argocd:v2.4.11
USER root
RUN apt-get update
RUN apt-get install -y curl
RUN apt-get install -y unzip
RUN apt-get clean
RUN curl https://get.datree.io | /bin/bash
LABEL release=1-with-curl
USER 999I have already built and published the image on my Docker Hub account. You can pull it from the repository piomin/argocd under the v2.4.11-datree tag. If you would like to build it by yourself just run the following command:
$ docker build -t piomin/argocd:v2.4.11-datree .Here’s the output:
Install Argo CD with the Datree Plugin on Kubernetes
We will use Helm to install Argo CD on Kubernetes. Instead of the default image, we need to use our custom Argo CD image built in the previous step. We also need to enable and configure our plugin in the Argo CD ConfigMap. You would also have to set the DATREE_TOKEN environment variable containing your Datree token. Fortunately, we can set all those things in a single place. Here’s our Helm values.yaml that overrides default configuration settings for Argo CD.
server:
config:
configManagementPlugins: |
- name: datree
generate:
command: ["bash", "-c"]
args: ['if [[ $(helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE -f <(echo "$ARGOCD_ENV_HELM_VALUES") . > result.yaml | datree test -o json result.yaml) && $? -eq 0 ]]; then cat result.yaml; else exit 1; fi']
repoServer:
image:
repository: piomin/argocd
tag: v2.4.11-datree
env:
- name: DATREE_TOKEN
value: <YOUR_DATREE_TOKEN>You can obtain the token from the Datree dashboard after login. Just go to your account settings:
Then, you need to add the Helm repository with Argo CD charts:
$ helm repo add argo https://argoproj.github.io/argo-helmAfter that, just install it with the custom settings provided in values.yaml file:
$ helm install argocd argo/argo-cd \
--version 5.1.0 \
--values values.yaml \
-n argocdOnce you installed Argo CD in the argocd namespace, you can display the argocd-cm ConfigMap.
$ kubectl get cm argocd-cm -n argocd -o yamlThe argocd-cm ConfigMap contains the definition of our plugin. Inside the args parameter, we need to pass the command for creating Deployment manifests. Each time we call the helm template command we pass the generated YAML file to the Datree CLI, which will run a policy check against it. If all of the rules pass the datree test command will return exit code 0. Otherwise, it will return a value other than 0. If the Datree policy check finishes successfully we need to send the YAML manifest to the output. Thanks to that, Argo CD will apply it to the Kubernetes cluster.
Create Helm charts
The instance of ArgoCD with a plugin for Datree is ready and we can now proceed to the app deployment phase. Firstly, let’s create Helm templates. Here’s a very basic Helm template for our sample app. It just exposes ports outside the container and sets environment variables. It is available under the simple-with-envs directory.
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.app.name }}
spec:
replicas: {{ .Values.app.replicas }}
selector:
matchLabels:
app: {{ .Values.app.name }}
template:
metadata:
labels:
app: {{ .Values.app.name }}
spec:
containers:
- name: {{ .Values.app.name }}
image: {{ .Values.image.registry }}/{{ .Values.image.repository }}:{{ .Values.image.tag }}
ports:
{{- range .Values.app.ports }}
- containerPort: {{ .value }}
name: {{ .name }}
{{- end }}
{{- if .Values.app.envs }}
env:
{{- range .Values.app.envs }}
- name: {{ .name }}
value: {{ .value }}
{{- end }}
{{- end }}We can test it locally with Helm and Datree CLI. Here’s a set of test values:
image:
registry: quay.io
repository: pminkows/sample-kotlin-spring
tag: "1.0"
app:
name: sample-spring-boot-kotlin
replicas: 1
ports:
- name: http
value: 8080
envs:
- name: PASS
value: exampleThe following command generates the YAML manifest using test-values.yaml and performs a Datree policy check.
$ helm template --values test-values.yaml . > result.yaml | \
datree test result.yamlHere’s the result of our test analysis. As you can see, there are a lot of violations reported by Datree:
For example, we should add liveness and readiness probes, and disable root access to the container. Here is another Helm template that fixes all the problems reported by Datree:
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.app.name }}
labels:
app: {{ .Values.app.name }}
env: {{ .Values.app.environment }}
owner: {{ .Values.app.owner }}
spec:
replicas: {{ .Values.app.replicas }}
selector:
matchLabels:
app: {{ .Values.app.name }}
template:
metadata:
labels:
app: {{ .Values.app.name }}
env: {{ .Values.app.environment }}
spec:
containers:
- name: {{ .Values.app.name }}
image: {{ .Values.image.registry }}/{{ .Values.image.repository }}:{{ .Values.image.tag }}
resources:
requests:
memory: {{ .Values.app.resources.memoryRequest }}
cpu: {{ .Values.app.resources.cpuRequest }}
limits:
memory: {{ .Values.app.resources.memoryLimit }}
cpu: {{ .Values.app.resources.cpuLimit }}
livenessProbe:
initialDelaySeconds: {{ .Values.app.liveness.initialDelaySeconds }}
httpGet:
port: {{ .Values.app.liveness.port }}
path: {{ .Values.app.liveness.path }}
failureThreshold: {{ .Values.app.liveness.failureThreshold }}
successThreshold: {{ .Values.app.liveness.successThreshold }}
timeoutSeconds: {{ .Values.app.liveness.timeoutSeconds }}
periodSeconds: {{ .Values.app.liveness.periodSeconds }}
readinessProbe:
initialDelaySeconds: {{ .Values.app.readiness.initialDelaySeconds }}
httpGet:
port: {{ .Values.app.readiness.port }}
path: {{ .Values.app.readiness.path }}
failureThreshold: {{ .Values.app.readiness.failureThreshold }}
successThreshold: {{ .Values.app.readiness.successThreshold }}
timeoutSeconds: {{ .Values.app.readiness.timeoutSeconds }}
periodSeconds: {{ .Values.app.readiness.periodSeconds }}
ports:
{{- range .Values.app.ports }}
- containerPort: {{ .value }}
name: {{ .name }}
{{- end }}
{{- if .Values.app.envs }}
env:
{{- range .Values.app.envs }}
- name: {{ .name }}
value: {{ .value }}
{{- end }}
{{- end }}
securityContext:
runAsNonRoot: trueYou can perform the same test as before for the new Helm chart. Just go to the full-compliant directory.
Create Argo CD Applications with Helm charts
Finally, we can create ArgoCD applications that use our Helm charts. Let’s create an app for the simple-with-envs chart (1). Instead of a typical helm type, we should set the plugin type (2). The name of our plugin is datree (3). With this type of ArgoCD app we have to set Helm parameters inside the HELM_VALUES environment variable (4).
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: simple-failed
namespace: argocd
spec:
destination:
name: ''
namespace: default
server: 'https://kubernetes.default.svc'
source:
path: simple-with-envs # (1)
repoURL: 'https://github.com/piomin/sample-generic-helm-charts.git'
targetRevision: HEAD
plugin: # (2)
name: datree # (3)
env:
- name: HELM_VALUES # (4)
value: |
image:
registry: quay.io
repository: pminkows/sample-kotlin-spring
tag: "1.4.30"
app:
name: sample-spring-boot-kotlin
replicas: 1
ports:
- name: http
value: 8080
envs:
- name: PASS
value: example
project: defaultHere’s the UI of our ArgoCD Application:
Let’s verify what happened:
ArgoCD performs a policy check using Datree. As you probably remember this Helm chart does not meet the rules defined in Datree. Therefore the process exited with error code 1.
Now, let’s create an ArgoCD Application for the second Helm chart:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: full-compliant-ok
namespace: argocd
spec:
destination:
name: ''
namespace: default
server: 'https://kubernetes.default.svc'
source:
path: full-compliant
repoURL: 'https://github.com/piomin/sample-generic-helm-charts.git'
targetRevision: HEAD
plugin:
name: datree
env:
- name: HELM_VALUES
value: |
image:
registry: quay.io
repository: pminkows/sample-kotlin-spring
tag: "1.4.30"
app:
name: sample-spring-boot-kotlin
replicas: 2
environment: test
owner: piomin
resources:
memoryRequest: 128Mi
memoryLimit: 512Mi
cpuRequest: 500m
cpuLimit: 1
liveness:
initialDelaySeconds: 10
port: 8080
path: /actuator/health/liveness
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 3
periodSeconds: 5
readiness:
initialDelaySeconds: 10
port: 8080
path: /actuator/health/readiness
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 3
periodSeconds: 5
ports:
- name: http
value: 8080
envs:
- name: PASS
value: example
project: defaultThis time the ArgoCD Application is created successfully:
That’s because Datree analysis finished successfully:
Finally, we can just synchronize the app to deploy our image on Kubernetes.
Final Thoughts
ArgoCD allows us to install custom plugins. Thanks to that we can integrate the ArgoCD deployment process with the Datree policy check, which is performed each time a new version of Kubernetes manifest is processed by ArgoCD. It applies to all the apps that refer to the plugin.
Leave a Reply