Continuous Delivery on Kubernetes with Database using ArgoCD and Liquibase
In this article, you will learn how to design a continuous delivery process on Kubernetes with ArgoCD and Liquibase. We will consider the application that connects to a database and update the schema on a new release. How to do it properly for a cloud-native application? Moreover, how to do it properly on Kubernetes?
Fortunately, there are two types of tools that perfectly fit the described process. Firstly, we need a tool that allows us to easily deploy applications in multiple environments. That’s what we may achieve on Kubernetes with ArgoCD. In the next step, we need a tool that automatically updates a database schema on demand. There are many tools for that. Since we need something lightweight and easy to containerize my choice fell on Liquibase. It is not my first article about Liquibase on Kubernetes. You can also read more about the blue-green deployment approach with Liquibase here.
However, in this article, we will focus on a little bit different problem. First, let’s describe it.
Introduction
I think that one of the biggest challenges around continuous delivery is integration to the databases. Consequently, we should treat this integration the same as a standard configuration. It’s time to treat database code like application code. Otherwise, our CI/CD process fails at the database.
Usually, when we consider the CI/CD process for the application we have multiple target environments. According to the cloud-native patterns, each application has its own separated database. Moreover, it is unique per each environment. So each time, we run our delivery pipeline, we have to update the database instance on the particular environment. We should do it just before running a new version application (e.g. with the blue-green approach).
Here’s the visualization of our scenario. We are releasing the Spring Boot application in three different environments. Those environments are just different namespaces on Kubernetes: dev, test and prod. The whole process is managed by ArgoCD and Liquibase. In the dev environment, we don’t use any migration tool. Let’s say we leave it to developers. Our mechanism is active for the test and prod namespaces.
Modern Java frameworks like Spring Boot offer built-in integration with Liquibase. In that concept, we just need to create a Liquibase changelog and set its location in Spring configuration. Our framework is running such a script on application startup. Since it is a very useful approach in development, I would not recommend it for production deployment. Especially if you deploy your application on Kubernetes. Why? You will find a detailed explanation in the article I have already mentioned in the first paragraph.
Source Code
If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should just follow my instructions 🙂
Also, one thing before we start. Here I described the whole process of building and deploying applications on Kubernetes with ArgoCD and Tekton. Typically there is a continuous integration phase, which is realized with Tekton. In this phase, we are just building and pushing the image. We are not releasing changes to the database. Especially, that we have multiple environments.
The picture visible below illustrates our approach in that context. ArgoCD synchronizes configuration and applies the changes into the Kubernetes cluster and a database using Liquibase. It doesn’t matter if a database is running on Kubernetes or not. However, in our case, we assume it is deployed on the namespace as the application.
Docker Image with Liquibase
There is an official Liquibase image on Docker Hub. We need to execute the update command using this image. To do that I prepared a custom image based on the official Liquibase image. You can see the Dockerfile below. But you can as well pull the image I published in my Docker registry docker.io/piomin/liquibase:latest.
FROM liquibase/liquibase
ENV URL=jdbc:postgresql://postgresql:5432/test
ENV USERNAME=postgres
ENV PASSWORD=postgres
ENV CHANGELOGFILE=changelog.xml
CMD ["sh", "-c", "docker-entrypoint.sh --url=${URL} --username=${USERNAME} --password=${PASSWORD} --classpath=/liquibase/changelog --changeLogFile=${CHANGELOGFILE} update"]We will run that image as the init container inside the pod with our application. Thanks to that approach we can be sure that it updates database schema just before starting the container with the application.
Sample Spring Boot application
We will use one of my sample Spring Boot applications in this exercise. You may find it in my GitHub repository here. It connects to the PostgreSQL database:
spring:
application:
name: person-service
datasource:
url: jdbc:postgresql://person-db:5432/${DATABASE_NAME}
username: ${DATABASE_USER}
password: ${DATABASE_PASSWORD}You can clone the application source code by yourself. But you can as well pull the ready image located here: quay.io/pminkows/person-app. The application is compiled with Java 17 and uses Spring Data JPA as an ORM layer to integrate with the database.
<properties>
<java.version>17</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>
</dependencies>
Use Liquibase with ArgoCD and Kustomize
In our scenario, the Liquibase init container should be included in the Deployment only for test and prod namespaces. On the dev environment, the pod should just contain a single container with the Spring Boot application. In order to implement this behavior with ArgoCD, we may use Kustomize. Kustomize has the concepts of bases and overlays. A base is a directory with a kustomization.yaml, which contains a set of resources and associated customization. Thanks to overlays, we may include additional elements into the base manifests. So, here’s the structure of our configuration repository in GitHub:
Here’s our base Deployment file. It’s pretty simple. The only thing we need to do is to inject database credentials from secrets:
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-spring-deployment
spec:
replicas: 1
selector:
matchLabels:
app: sample-spring
template:
metadata:
labels:
app: sample-spring
spec:
containers:
- name: sample-spring
image: quay.io/pminkows/person-app:1.0
ports:
- containerPort: 8080
env:
- name: DATABASE_USER
valueFrom:
secretKeyRef:
name: postgres
key: database-user
- name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: postgres
key: database-password
- name: DATABASE_NAME
valueFrom:
secretKeyRef:
name: postgres
key: database-nameWe also have the Liquibase changeLog.sql file in the base directory. We should place it inside Kubernetes ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: changelog-cm
data:
changeLog.sql: |
--liquibase formatted sql
--changeset piomin:1
create table person (
id serial primary key,
name varchar(255),
gender varchar(255),
age int,
externalId int
);
insert into person(name, age, gender) values('John Smith', 25, 'MALE');
insert into person(name, age, gender) values('Paul Walker', 65, 'MALE');
insert into person(name, age, gender) values('Lewis Hamilton', 35, 'MALE');
insert into person(name, age, gender) values('Veronica Jones', 20, 'FEMALE');
insert into person(name, age, gender) values('Anne Brown', 60, 'FEMALE');
insert into person(name, age, gender) values('Felicia Scott', 45, 'FEMALE');Also, let’s take look at the base/kustomization.yaml file:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- changelog.yamlIn the overlays/liquibase/liquibase-container.yaml file we are defining the init container that should be included in our base Deployment. There are four parameters available to override. Therefore, we will set the address of a target database, username, password, and location of the Liquibase changelog file. The changeLog.sql file is available to the container as a mounted volume under location /liquibase/changelog. Of course, I’m using the image described in the Docker Image with Liquibase section.
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-spring-deployment
spec:
template:
spec:
initContainers:
- name: liquibase
image: docker.io/piomin/liquibase:latest
env:
- name: URL
value: jdbc:postgresql://person-db:5432/sampledb
- name: USERNAME
valueFrom:
secretKeyRef:
name: postgres
key: database-user
- name: PASSWORD
valueFrom:
secretKeyRef:
name: postgres
key: database-password
- name: CHANGELOGFILE
value: changeLog.sql
volumeMounts:
- mountPath: /liquibase/changelog
name: changelog
volumes:
- name: changelog
configMap:
name: changelog-cmAnd last manifest in the repository. Here’s the overlay kustomization.yaml file. It uses the whole structure from the base catalog and includes the init container to the application Deployment:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
patchesStrategicMerge:
- liquibase-container.yamlCreate ArgoCD applications
Since our configuration is ready, we may proceed to the last step. Let’s create ArgoCD applications responsible for synchronization between Git repository and both Kubernetes cluster and a target database. In order to create the ArgoCD application, we need to apply the following manifest. It refers to the Kustomize overlay defined in the /overlays/liquibase directory. The declaration for test or prod environment looks as shown below:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: liquibase-prod
spec:
destination:
namespace: prod
server: 'https://kubernetes.default.svc'
project: default
source:
path: overlays/liquibase
repoURL: 'https://github.com/piomin/sample-argocd-liquibase-kustomize.git'
targetRevision: HEADOn the other hand, the dev environment doesn’t require an init container with Liquibase. Therefore, it uses the base directory:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: liquibase-dev
spec:
destination:
namespace: dev
server: 'https://kubernetes.default.svc'
project: default
source:
path: base
repoURL: 'https://github.com/piomin/sample-argocd-liquibase-kustomize.git'
targetRevision: HEADSince ArgoCD supports Kustomize, we just need to create applications. Alternatively, we could have created them using ArgoCD UI. Finally, here’s a list of ArgoCD applications responsible for applying changes to the Postgres database. Here’s the UI view for all the environments after synchronization (Sync button).
We can also verify the logs printed by the Liquibase init container after ArgoCD synchronization:
We may also verify a list of running pods in one of the target namespace, e.g. prod.
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
person-db-1-vn8kw 1/1 Running 0 82m
sample-spring-deployment-7695d64c54-9hf75 1/1 Running 0 7m16sAnd also print out Liquibase logs using kubectl:
$ kubectl logs sample-spring-deployment-7695d64c54-9hf75 -c liquibase
4 COMMENTS