Continuous Integration with Jenkins on Kubernetes
Although Jenkins is a mature solution, it still can be the first choice for building CI on Kubernetes. In this article, I’ll show how to install Jenkins on Kubernetes, and use it for building a Java application with Maven. You will learn how to use and customize the Helm for this installation. We will implement typical steps for building and deploying Java applications on Kubernetes.
Here’s the architecture of our solution.
Clone the source code
If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my repository sample-spring-boot-on-kubernetes. Then you should just follow my instructions 🙂
Deploy Jenkins on Kubernetes with Helm
We may install Jenkins server on Kubernetes using the Helm package manager. The official Helm chart spawns agents on Kubernetes and utilizes Jenkins Kubernetes Plugin. It comes with a default configuration. However, we may override all the properties using the JCaSC Plugin. It is worth reading more about this plugin before continuing. You can find out more about it here.
Firstly, we will add a new Helm repository and update it.
$ helm repo add jenkins https://charts.jenkins.io
$ helm repo updateThen, we may install Jenkins by executing the following command. The Jenkins instance is running in the dedicated namespace. So, before installing it we need to create a new namespace with the kubectl create ns jenkins command. In order to include an additional configuration, we need to create a YAML file and pass it with -f option.
$ helm install -f k8s/jenkins-helm-config.yaml jenkins jenkins/jenkins -n jenkinsCustomize Jenkins
We need to configure several things before creating a build pipeline. Let’s consider the following list of tasks:
- Customize the
defaultagent – we need to mount a volume into thedefaultagent pod to store workspace files and share them with the other agents. - Create the
mavenagent – we need to define a new agent able to perform a Maven build. It should use the same persistent volume as thedefaultagent. Also, it should use the JDK 11, because our application is compiled with that version of Java. Finally, we will increase the default CPU and memory limits for the agent pods. - Create GitHub credentials – the Jenkins pipeline needs to be able to clone the source code from GitHub
- Install the Kubernetes Continuous Deploy plugin – we will use this plugin to deploy resource configurations to a Kubernetes cluster.
- Create
kubeconfigcredentials – we have to provide a configuration of our Kubernetes context.
Let’s take a look at the whole Jenkins configuration file. Consequently, it contains agent, additionalAgents sections, and defines a JCasC script with the credentials definition.
agent:
podName: default
customJenkinsLabels: default
volumes:
- type: PVC
claimName: jenkins-agent
mountPath: /home/jenkins
readOnly: false
additionalAgents:
maven:
podName: maven
customJenkinsLabels: maven
image: jenkins/jnlp-agent-maven
tag: jdk11
volumes:
- type: PVC
claimName: jenkins-agent
mountPath: /home/jenkins
readOnly: false
resources:
limits:
cpu: "1"
memory: "2048Mi"
master:
JCasC:
configScripts:
creds: |
credentials:
system:
domainCredentials:
- domain:
name: "github.com"
description: "GitHub domain"
specifications:
- hostnameSpecification:
includes: "github.com"
credentials:
- usernamePassword:
scope: GLOBAL
id: github_credentials
username: piomin
password: ${GITHUB_PASSWORD}
- credentials:
- kubeconfig:
id: "docker-desktop"
kubeconfigSource:
directEntry:
content: |-
apiVersion: v1
kind: Config
preferences: {}
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01URXdPVEE1TkRjd04xb1hEVE13TVRFd056QTVORGN3TjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTStrCnhKd3l6UVEydXNvRHh5RmwxREdwZWZQTVA0RGFVaVJsK01SQ1p1S0NFWUFkL0ZQOWtFS0RlVXMydmVURi9jMXYKUjZpTDlsMVMvdmN6REoyRXRuZUd0TXVPdWFXNnFCWkN5OFJ2NmFReHd0UEpnWVZGTHBNM2dXYmhqTHp3RXplOApEQlhxekZDZkNobXl3SkdORVdWV0s4VnBuSlpMbjRVbUZKcE5RQndsSXZwRC90UDJVUVRiRGNHYURqUE5vY2c0Cms1SmNOc3h3SDV0NkhIN0JZMW9jTEFLUUhsZ2V4V2ZObWdRRkM2UUcrRVNsWkpDVEtNdVVuM2JsRWVlYytmUWYKaVk3YmdOb3BjRThrS0lUY2pzMG95dGVyNmxuY2ZLTVBqSnc2RTNXMmpXRThKU2Z2WDE2dGVhZUZISDEyTmRqWgpWTER2ZWc3eVBsTlRmRVJld25FQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZMWjUzVEhBSXp0bHljV0NrS1hhY2l4K0Y5a1FNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFBZWllMTRoSlZkTHF3bUY0SGVPS0ZhNXVUYUt6aXRUSElJNHJhU3cxTUppZElQTmZERwprRk5KeXM1M2I4MHMveWFXQ3BPbXdCK1dEak9hWmREQjFxcXVxU1FxeGhkNWMxU2FBZ1VXTGp5OXdLa1dPMzBTCjB3RTRlVkY3Q1c5VGpSMEoyVXV6UEVXdFBKRWF4U2xKMGhuZlQyeFYvb0N5OE9kMm9EZjZkSFRvbE5UTUEyalcKZjRZdXl3U1Z5a2RNaXZYMU5xZzdmK3RrcEVwb25PdkQ4ZmFEL2dXZmpaWHNFdHo4NXRNcTVLd2NQNUh2ZDJ0ZgoyKzBSbEtFT0pyY1dyL1lEc2w3dWdDdkFJTVk4WGdJL1E5dTZZTjAzTngzWXdSS2UrMElpSzcyOHVuNVJaVEVXCmNZNHc0YkpudlN6WWpKeUJIaHNiQVNTNzN6NndXVEo4REhKSwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
server: https://kubernetes.default
name: docker-desktop
contexts:
- context:
cluster: docker-desktop
user: docker-desktop
name: docker-desktop
current-context: docker-desktop
users:
- name: docker-desktop
user:
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURGVENDQWYyZ0F3SUJBZ0lJRnh2QzMyK2tPMEl3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TURFeE1Ea3dPVFEzTURkYUZ3MHlNVEV4TURrd09UUTNNekZhTURZeApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sc3dHUVlEVlFRREV4SmtiMk5yWlhJdFptOXlMV1JsCmMydDBiM0F3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQ3M0TXdUU3ByMkRoOTMKTlpERldsNWQyaWgwbllBdTJmTk1RYjZ2ZHR5RUVpTUVpNk5BM05qRGM4OWl5WUhOU2J4YmVNNlNUMzRlTFIwaQpXbHJJSlhhVjNBSXhnbFo4SkdqczVUSHRlM1FjNXZVSkJJWXhndFJFTFlJMGlJekpZdEhoU1NwMFU0eWNjdzl5CnVGSm1YTHVBRVdXR0tTcitVd2Y3RWtuWmJoaFRNQWI0RUF1NlR6dkpyRHhpTDAzU0UrSWhJMTJDV1Y3cVRqZ1gKdGI1OXdKcWkwK3ZJSDBSc3dxOUpnemtQTUhLNkFBZkgxbmFmZ3VCQjM2VEppcUR6YWFxV2VZTmludlIrSkVHMAptakV3NWlFN3JHdUgrZVBxSklvdTJlc1YvN1hyYWx2UEl2Zng2ajFvRWI4NWtna2RuV0JiQlNmTmJCdnhHQU1uCmdnLzdzNHdoQWdNQkFBR2pTREJHTUE0R0ExVWREd0VCL3dRRUF3SUZvREFUQmdOVkhTVUVEREFLQmdnckJnRUYKQlFjREFqQWZCZ05WSFNNRUdEQVdnQlMyZWQweHdDTTdaY25GZ3BDbDJuSXNmaGZaRURBTkJna3Foa2lHOXcwQgpBUXNGQUFPQ0FRRUFpbUg1c1JqcjB6WjlDRkU2SVVwNVRwV2pBUXhma29oQkpPOUZmRGE3N2kvR1NsYm1jcXFrCldiWUVYRkl3MU9EbFVjUy9QMXZ5d3ZwV0Y0VXNXTGhtYkF5ZkZGbXZRWFNEZHhYbTlkamI2OEVtRWFPSlI1VTYKOHJOTkR0TUVEY25sbFd2Qk1CRXBNbkNtcm9KcXo3ZzVzeDFQSmhwcDBUdUZDQTIwT2FXb3drTUNNUXRIZlhLQgpVUDA2eGxRU2o1SGNOS1BSQWFyQzBtSzZPVUhybExBcUIvOCtDQlowVUY2MXhTTGN1WFJvYU52S1ZDWHZnQy9kCkQ4ckxuWXFmbWl6WHMvcHJ3dEhsaVFBR2lmemU1MmttbTkyR2RrS2V1SmFRbmM5RWwrd2RZaUVBSHVKU1YvK04Kc2VRelpTa0ZmT2ozbHUxdWtoSDg4dGcxUUp2TkpuM1FhQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBck9ETUUwcWE5ZzRmZHpXUXhWcGVYZG9vZEoyQUx0bnpURUcrcjNiY2hCSWpCSXVqClFOell3M1BQWXNtQnpVbThXM2pPa2s5K0hpMGRJbHBheUNWMmxkd0NNWUpXZkNSbzdPVXg3WHQwSE9iMUNRU0cKTVlMVVJDMkNOSWlNeVdMUjRVa3FkRk9NbkhNUGNyaFNabHk3Z0JGbGhpa3EvbE1IK3hKSjJXNFlVekFHK0JBTAp1azg3eWF3OFlpOU4waFBpSVNOZGdsbGU2azQ0RjdXK2ZjQ2FvdFByeUI5RWJNS3ZTWU01RHpCeXVnQUh4OVoyCm40TGdRZCtreVlxZzgybXFsbm1EWXA3MGZpUkJ0Sm94TU9ZaE82eHJoL25qNmlTS0x0bnJGZisxNjJwYnp5TDMKOGVvOWFCRy9PWklKSFoxZ1d3VW56V3diOFJnREo0SVArN09NSVFJREFRQUJBb0lCQVFDWEZZTGtYVEFlVit0aAo2RnRVVG96b0lxOTJjdXRDaHRHZFZGdk14dWtqTnlLSloydk9WUFBQcE5lYXN4YVFqWjlpcGFxS3JaUS8xUmVBCkhVejNXOTVPUzg5UzYyQ2Y3OFlQT3FLdXRGU2VxYTErS3drSUhobGFXQmRSeUFDYVE1VysrSTEweWt1NXNzak8KYm8zOHpaQkQ5WEF2bHF6dlJTdFZYZjlTV1doQzBlWnRKTm84QU4yZnpkdkRjUUgwOVRsejh1S05EaUNra2RYQQpHTTdZTUdoQktYWGd6YlcxSUVMejRlRUpDZDh0dklReitwcWtxRktIcHRjNnVJY1hLQjFxUGVGRDRSMm9iNUlNCnl5MUpBWlZyR0JHaUk5d1p5OFU1a253UW93emwwUTEwZXlRdUkwTG42SWthZG5SQktMRHcrczRGaE1UQVViOWYKT1NBR3JaVnRBb0dCQU9RTDJzSEN3T25KOW0xYmRiSlVLeTFsRHRsTmV4aDRKOGNERVZKR3NyRVNndlgrSi9ZZQpXb0NHZXc3cGNXakFWaWJhOUMzSFBSbEtOV2hXOExOVlpvUy9jQUxCa1lpSUZNdDlnT1NpZmtCOFhmeVJQT3hJCmNIc2ZjOXZ2OEFJcmxZVVpCNjI1ak8rUFZTOXZLOXZXUGtka3d0MlFSUHlqYlEwVS9mZmdvUWVIQW9HQkFNSVIKd0lqL3RVbmJTeTZzL1JXSlFiVmxPb1VPVjFpb0d4WmNOQjBkaktBV3YwUksvVHdldFBRSXh2dmd1d2RaVFdiTApSTGk5M3RPY3U0UXhNOFpkdGZhTnh5dWQvM2xXSHhPYzRZa0EwUHdSTzY4MjNMcElWNGxrc0tuNUN0aC80QTFJCmw3czV0bHVEbkI3dFdYTFM4MHcyYkE4YWtVZXlBbkZmWXpTTUR1a1hBb0dBSkRFaGNialg1d0t2Z21HT2gxUEcKV25qOFowNWRwOStCNkpxN0NBVENYVW5qME9pYUxQeGFQcVdaS0IreWFQNkZiYnM0SDMvTVdaUW1iNzNFaTZHVgpHS0pOUTVLMjV5VTVyNlhtYStMQ0NMZjBMcDVhUGVHdFFFMFlsU0k2UkEzb3Qrdm1CUk02bzlacW5aR1dNMWlJCkg4cUZCcWJiM0FDUDBSQ3cwY01ycTBjQ2dZRUFvMWM5cmhGTERMYStPTEx3OE1kdHZyZE00ZUNJTTk2SnJmQTkKREtScVQvUFZXQzJscG94UjBYUHh4dDRIak0vbERiZllSNFhIbm1RMGo3YTUxU1BhbTRJSk9QVHFxYjJLdW44NApkSTl6VmpWSy90WTJRYlBSdVpvOTkxSGRod3RhRU5RZ29UeVo5N3gyRXJIQ3I1cE5uTC9SZzRUZzhtOHBEek14CjFIQnR2RkVDZ1lFQTF5aHNPUDBRb3F2SHRNOUFuWVlua2JzQU12L2dqT3FrWUk5cjQ2c1V3Mnc3WHRJb1NTYlAKU0hmbGRxN0IxVXJoTmRJMFBXWXRWZ3kyV1NrN0FaeG8vUWtLZGtPbTAxS0pCY2xteW9JZDE0a0xCVkZxbUV6Rgp1c2l4MmpwdTVOTWhjUWo4aFY2Sk42aXdraHJkYjByZVpuMGo4MG1ZRE96d3hjMmpvTmxSWjN3PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
scope: GLOBALBefore installing Jenkins we need to create the PersistentVolumeClaim object. This volume is used by the agents to store workspace files.
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: jenkins-agent
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Gi
storageClassName: hostpathFinally, let’s create it inside the jenkins namespace.
$ kubectl create -f k8s\jenkins-agent-pvc.yamlExplore Jenkins on Kubernetes
After starting the Jenkins instance, we may log in to its web management console. The default root username is admin. In order to obtain the password, we should execute the following command.
$ kubectl get secret --namespace jenkins jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode
YdeQuJZHa1The Jenkins instance is available on port 8080 inside the Kubernetes cluster. Let’s expose it on the local port with the kubectl port-forward command. Now, we can access it on the address http://localhost:8080.
$ kubectl port-forward service/jenkins 8080:8080 -n jenkinsLet’s log in to the Jenkins console. After that, we can verify the correctness of our installation. Firstly, we need to navigate to the “Manage Jenkins”, and then to the “Manage Credentials” section. As you can see below, there are two credentials there: github_credentials and docker-desktop.
Then, let’s move back to the “Manage Jenkins”, and go to the “Manage Nodes and Clouds” section. In the “Configure Clouds” tab, there is the Kubernetes configuration as shown below. It contains two pod templates: default and maven.
Explore a sample application
The sample application is built on top of Spring Boot. We use Maven for building it. On the other hand, we use the Jib plugin for creating a Docker image. Thanks to that, we won’t have to install any other tools to build Docker images with Jenkins.
<properties>
<java.version>11</java.version>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>build-info</goal>
</goals>
</execution>
</executions>
<configuration>
<excludeDevtools>false</excludeDevtools>
</configuration>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>jib</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<build>
<plugins>
<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>jib-maven-plugin</artifactId>
<version>2.4.0</version>
<configuration>
<to>piomin/sample-spring-boot-on-kubernetes</to>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>The Deployment manifest contains the PIPELINE_NAMESPACE parameter. Our pipeline replaces it with the target namespace name. Therefore we may deploy our application in the multiple Kubernetes namespaces.
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-spring-boot-on-kubernetes-deployment
namespace: ${PIPELINE_NAMESPACE}
spec:
selector:
matchLabels:
app: sample-spring-boot-on-kubernetes
template:
metadata:
labels:
app: sample-spring-boot-on-kubernetes
spec:
containers:
- name: sample-spring-boot-on-kubernetes
image: piomin/sample-spring-boot-on-kubernetes
ports:
- containerPort: 8080
env:
- name: MONGO_DATABASE
valueFrom:
configMapKeyRef:
name: mongodb
key: database-name
- name: MONGO_USERNAME
valueFrom:
secretKeyRef:
name: mongodb
key: database-user
- name: MONGO_PASSWORD
valueFrom:
secretKeyRef:
name: mongodb
key: database-passwordCreate Jenkins pipeline on Kubernetes
Finally, we may create the Jenkins pipeline for our application. It consists of six steps. Firstly, we clone the source code repository from GitHub. We use the default Jenkins agent for it. In the next three stages we use the maven agent. In the second stage, we build the application. Then, we run JUnit tests. After that, we build a Docker image in the “dockerless” mode using the Maven Jib plugin. In the last two stages, we can take an advantage of the Kubernetes Continuous Deploy plugin. We use an already created kubeconfig credentials, and deployment-template.yaml file from the source code. We just need to set PIPELINE_NAMESPACE environment variable.
pipeline {
agent {
label "default"
}
stages {
stage('Checkout') {
steps {
script {
git url: 'https://github.com/piomin/sample-spring-boot-on-kubernetes.git', credentialsId: 'github_credentials'
}
}
}
stage('Build') {
agent {
label "maven"
}
steps {
sh 'mvn clean compile'
}
}
stage('Test') {
agent {
label "maven"
}
steps {
sh 'mvn test'
}
}
stage('Image') {
agent {
label "maven"
}
steps {
sh 'mvn -P jib -Djib.to.auth.username=${DOCKER_LOGIN} -Djib.to.auth.password=${DOCKER_PASSWORD} compile jib:build'
}
}
stage('Deploy on test') {
steps {
script {
env.PIPELINE_NAMESPACE = "test"
kubernetesDeploy kubeconfigId: 'docker-desktop', configs: 'k8s/deployment-template.yaml'
}
}
}
stage('Deploy on prod') {
steps {
script {
env.PIPELINE_NAMESPACE = "prod"
kubernetesDeploy kubeconfigId: 'docker-desktop', configs: 'k8s/deployment-template.yaml'
}
}
}
}
}Run the pipeline
Our sample Spring Boot application connects to MongoDB. Therefore, we need to deploy the Mongo instance to the test and prod namespaces before running the pipeline. We can use manifest mongodb-deployment.yaml in k8s directory.
$ kubectl create ns test
$ kubectl apply -f k8s/mongodb-deployment.yaml -n test
$ kubectl create ns prod
$ kubectl apply -f k8s/mongodb-deployment.yaml -n prodFinally, let’s run our test pipeline. It finishes successfully as shown below.
Now, we may check out a list of running pods in the prod namespace.
Conclusion
Helm and JCasC plugin simplify the installation of Jenkins on Kubernetes. Also, Maven comes with a huge set of plugins, that may be used with Docker and Kubernetes. In this article, I showed how to use the Jib Maven Plugin to build an image of your application and Jenkins plugins to run pipelines and deploys on Kubernetes. You can compare it with the concept over GitLab CI presented in the article GitLab CI/CD on Kubernetes. Enjoy 🙂
8 COMMENTS