Renew Certificates on Kubernetes with Cert Manager and Reloader
In this article, you will learn how to renew certificates in your Spring Boot apps on Kubernetes with cert-manager and Stakater Reloader. We are going to run two simple Spring Boot apps that communicate with each other over SSL. The TLS cert used in that communication will be automatically generated by Cert Manager. With Cert Manager we can easily rotate certs after a certain time. In order to automatically use the latest TLS certs we need to restart our apps. We can achieve it with Stakater Reloader.
Before we start, it is worth reading the following article. It shows how to use cert-manager together with Istio to create secure gateways on Kubernetes.
Source Code
If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the ssl directory. You will find two Spring Boot apps: secure-callme and secure-caller. After that, you should just follow my instructions. Let’s begin.
How it works
Before we go into the technical details, let me write a little bit more about the architecture of our solution. Our challenge is pretty common. We need secure SSL/TLS communication between the services running on Kubernetes. Instead of manually generating and replacing certs inside the apps, we need an automatic approach.
Here come the cert-manager and the Stakater Reloader. Cert Manager is able to generate certificates automatically, based on the provided CRD object. It also ensures the certificates are valid and up-to-date and will attempt to renew certificates before expiration. It puts all the required inside Kubernetes Secret. On the other hand, Stakater Reloader is able to watch if any changes happen in ConfigMap or Secret. Then it performs a rolling upgrade on pods, which use the particular ConfigMap or Secret. Here is the visualization of the described architecture.
Prerequisites
Of course, you need to have a Kubernetes cluster. In this exercise, I’m using Kubernetes on Docker Desktop. But you can as well use any other local distribution like minikube, kind, or a cloud-hosted instance. No matter which distribution you choose you also need to have:
- Skaffold (optionally) – a CLI tool to simplify deploying the Spring Boot app on Kubernetes and applying all the manifests in that exercise using a single command. You can find installation instructions here
- Helm – used to install additional tools on Kubernetes like Stakater Reloader or cert-manager
Install Cert Manager and Stakater Reloader
In order to install both cert-manager and Reloader on Kubernetes we will use Helm charts. We don’t need any specific settings just defaults. Let’s begin with the cert-manager. Before installing the chart we have to add CRD resources for the latest version 1.10.1:
$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.crds.yamlThen, we need to add the jetstack chart repository:
$ helm repo add jetstack https://charts.jetstack.ioAfter that, we can install the chart using the following command:
$ helm install my-release cert-manager jetstack/cert-managerThe same with the Stakater Reloader – first we need to add the stakater charts repository:
$ helm repo add stakater https://stakater.github.io/stakater-chartsThen, we can install the latest version of the chart:
$ helm install my-reloader stakater/reloaderIn order to verify that the installation finished successfully we can display a list of running pods:
$ kubectl get po
NAME READY STATUS RESTARTS AGE
my-cert-manager-578884c6cf-f9ppt 1/1 Running 0 1m
my-cert-manager-cainjector-55d4cd4bb6-6mgjd 1/1 Running 0 1m
my-cert-manager-webhook-5c68bf9c8d-nz7sd 1/1 Running 0 1m
my-reloader-reloader-7566fdc68c-qj9l4 1/1 Running 0 1mThat’s all. Now we can proceed to the implementation.
HTTPS with Spring Boot
Our first app secure-callme exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller app. Here’s the @RestController implementation:
@RestController
public class SecureCallmeController {
@GetMapping("/callme")
public String call() {
return "I'm `secure-callme`!";
}
}Now our goal is to enable HTTPS for that app, and of course, make it work properly on Kubernetes. First, we should change the default server port for the Spring Boot app to 8443. Then we have to enable SSL and provide locations of key stores. Additionally, we will force verification of the client’s certificate with the server.ssl.client-auth property. Here’s the configuration for our Spring Boot inside the application.yml file.
server.port: 8443
server.ssl:
enabled: true
key-store: ${CERT_PATH}/keystore.jks
key-store-password: ${PASSWORD}
trust-store: ${CERT_PATH}/truststore.jks
trust-store-password: ${PASSWORD}
client-auth: NEEDWe will set the values of CERT_PATH and PASSWORD at the level of Kubernetes Deployment. Now, let’s switch to the secure-caller implementation. We have to configure SSL on the REST client side. Since we use Spring RestTemplate for calling services, we need to add customize its default behavior. Firstly, let’s include the Apache HttpClient dependency.
<dependency>
<groupId>org.apache.httpcomponents.client5</groupId>
<artifactId>httpclient5</artifactId>
</dependency>Now, we will use Apache HttpClient as a low-level client for the Spring RestTemplate. We need to define a key store and trust store for the client since a server-side requires and verifies client cert. In order to create RestTempate @Bean we use RestTemplateBuilder.
@SpringBootApplication
public class SecureCaller {
public static void main(String[] args) {
SpringApplication.run(SecureCaller.class, args);
}
@Autowired
ClientSSLProperties clientSSLProperties;
@Bean
RestTemplate builder(RestTemplateBuilder builder) throws
GeneralSecurityException, IOException {
final SSLContext sslContext = new SSLContextBuilder()
.loadTrustMaterial(
new File(clientSSLProperties.getTrustStore()),
clientSSLProperties.getTrustStorePassword().toCharArray())
.loadKeyMaterial(
new File(clientSSLProperties.getKeyStore()),
clientSSLProperties.getKeyStorePassword().toCharArray(),
clientSSLProperties.getKeyStorePassword().toCharArray()
)
.build();
final SSLConnectionSocketFactory sslSocketFactory =
SSLConnectionSocketFactoryBuilder.create()
.setSslContext(sslContext)
.build();
final HttpClientConnectionManager cm =
PoolingHttpClientConnectionManagerBuilder.create()
.setSSLSocketFactory(sslSocketFactory)
.build();
final HttpClient httpClient = HttpClients.custom()
.setConnectionManager(cm)
.evictExpiredConnections()
.build();
return builder
.requestFactory(() ->
new HttpComponentsClientHttpRequestFactory(httpClient))
.build();
}
}The client credentials are taken from configuration settings under the client.ssl key. Here is the @ConfigurationProperties class used the RestTemplateBuilder in the previous step.
@Configuration
@ConfigurationProperties("client.ssl")
public class ClientSSLProperties {
private String keyStore;
private String keyStorePassword;
private String trustStore;
private String trustStorePassword;
// GETTERS AND SETTERS ...
}Here’s the configuration for the secure-caller inside application.yml file. The same as for the secure-callme we expose the REST endpoint over HTTPS.
server.port: 8443
server.ssl:
enabled: true
key-store: ${CERT_PATH}/keystore.jks
key-store-password: ${PASSWORD}
trust-store: ${CERT_PATH}/truststore.jks
trust-store-password: ${PASSWORD}
client-auth: NEED
client.url: https://${HOST}:8443/callme
client.ssl:
key-store: ${CLIENT_CERT_PATH}/keystore.jks
key-store-password: ${PASSWORD}
trust-store: ${CLIENT_CERT_PATH}/truststore.jks
trust-store-password: ${PASSWORD}The secure-caller app calls GET /callme exposed by the secure-callme app using customized RestTemplate.
@RestController
public class SecureCallerController {
RestTemplate restTemplate;
@Value("${client.url}")
String clientUrl;
public SecureCallerController(RestTemplate restTemplate) {
this.restTemplate = restTemplate;
}
@GetMapping("/caller")
public String call() {
return "I'm `secure-caller`! calling... " +
restTemplate.getForObject(clientUrl, String.class);
}
}Generate and Renew Certificates on Kubernetes with Cert Manager
With cert-manager, we can automatically generate and renew certificates on Kubernetes. Of course, we could generate TLS/SSL certs using e.g. openssl as well and then apply them on Kubernetes. However, Cert Manager simplifies that process. It allows us to declaratively define the rules for the certs generation process. Let’s see how it works. Firstly, we need the issuer object. We can create a global issuer for the whole as shown. It uses the simplest option – self-signed.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: ss-clusterissuer
spec:
selfSigned: {}After that, we can generate certificates. Here’s the cert-manager Certificate object for the secure-callme app. There are some important things here. First of all, we can generate key stores together with a certificate and private key (1). The object refers to the ClusterIssuer created in the previous step (2). The name of Kubernetes Service used during communication is secure-callme, so the cert needs to have that name as CN. In order to enable certificate rotation we need to set validity time. The lowest possible value is 1 hour (4). So each time 5 minutes before expiration cert-manager will automatically renew a certificate (5). It won’t rotate the private key. In order to enable it we should set the parameter rotationPolicy to Always.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: secure-callme-cert
spec:
keystores: # (1)
jks:
passwordSecretRef:
name: jks-password-secret
key: password
create: true
issuerRef: # (2)
name: ss-clusterissuer
group: cert-manager.io
kind: ClusterIssuer
privateKey:
algorithm: ECDSA
size: 256
dnsNames:
- secure-callme
secretName: secure-callme-cert
commonName: secure-callme # (3)
duration: 1h # (4)
renewBefore: 5m # (5)The Certificate object for secure-caller is very similar. The only difference is in the CN field. We will use the port-forward option during the test, so I’ll set the domain name to localhost (1).
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: secure-caller-cert
spec:
keystores:
jks:
passwordSecretRef:
name: jks-password-secret
key: password
create: true
issuerRef:
name: ss-clusterissuer
group: cert-manager.io
kind: ClusterIssuer
privateKey:
algorithm: ECDSA
size: 256
dnsNames:
- localhost
- secure-caller
secretName: secure-caller-cert
commonName: localhost # (1)
duration: 1h
renewBefore: 5mAfter applying both manifests we can display a list of Certificates. Each of them is related to the Secret with the same name:
$ kubectl get certificate
NAME READY SECRET AGE
secure-caller-cert True secure-caller-cert 1m
secure-callme-cert True secure-callme-cert 1mHere are the details of the secure-callme-cert Secret. It contains the key store and trust store in the JKS format. We will use both of them in the Spring Boot SSL configuration (server.ssl.trust-store and server.ssl.key-store properties). There is also a certificate (tls.crt), a private key (tls.key), and CA (ca.crt).
$ kubectl describe secret secure-callme-cert
Name: secure-callme-cert
Namespace: default
Labels: <none>
Annotations: cert-manager.io/alt-names: secure-callme
cert-manager.io/certificate-name: secure-callme-cert
cert-manager.io/common-name: secure-callme
cert-manager.io/ip-sans:
cert-manager.io/issuer-group: cert-manager.io
cert-manager.io/issuer-kind: ClusterIssuer
cert-manager.io/issuer-name: ss-clusterissuer
cert-manager.io/uri-sans:
Type: kubernetes.io/tls
Data
====
ca.crt: 550 bytes
keystore.jks: 1029 bytes
tls.crt: 550 bytes
tls.key: 227 bytes
truststore.jks: 422 bytesDeploy and Reload Apps on Kubernetes
Since we have already prepared all the required components and objects, we may proceed with the deployment of our apps. As I mentioned in the “Prerequisites” section we will use Skaffold for building and deploying apps on the local cluster. Let’s begin with the secure-callme app.
First of all, we need to reload the app each time the secure-callme-cert changes. It occurs once per hour when the cert-manager renews the TLS certificate. In order to enable the automatic restart of the pod with Stakater Reloader we need to annotate the Deployment with secret.reloader.stakater.com/reload (1). The annotation should contain the name of Secret, which triggers the app reload. Of course, we also need to mount key store and trust store files (3) and set the mount path for the Spring Boot app available under the CERT_PATH env variable (2). We are mounting the whole secure-callme-cert Secret.
apiVersion: apps/v1
kind: Deployment
metadata:
name: secure-callme
annotations:
# (1)
secret.reloader.stakater.com/reload: "secure-callme-cert"
spec:
selector:
matchLabels:
app.kubernetes.io/name: secure-callme
template:
metadata:
labels:
app.kubernetes.io/name: secure-callme
spec:
containers:
- image: piomin/secure-callme
name: secure-callme
ports:
- containerPort: 8443
name: https
env:
- name: PASSWORD
valueFrom:
secretKeyRef:
key: password
name: jks-password-secret
- name: CERT_PATH # (2)
value: /opt/secret
volumeMounts:
- mountPath: /opt/secret # (3)
name: cert
volumes:
- name: cert
secret:
secretName: secure-callme-cert # (4)The password to the key store files is available inside the jks-password-secret:
kind: Secret
apiVersion: v1
metadata:
name: jks-password-secret
data:
password: MTIzNDU2
type: OpaqueThere is also the Kubernetes Service related to the app:
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: secure-callme
name: secure-callme
spec:
ports:
- name: https
port: 8443
targetPort: 8443
selector:
app.kubernetes.io/name: secure-callme
type: ClusterIPNow, go to the secure-callme directory and just run the following command:
$ skaffold runThe Deployment manifest of the secure-caller app is a little bit more complicated. The same as before we need to reload the app on Secret change (1). However, this app uses two secrets. The first of them contains server certs (secure-caller-cert), while the second contains certs for communication with secure-callme. Consequently, we are mounting two secrets (5) and we are setting the path with server key stores (2) and client key stores (3).
apiVersion: apps/v1
kind: Deployment
metadata:
name: secure-caller
# (1)
annotations:
secret.reloader.stakater.com/reload: "secure-caller-cert,secure-callme-cert"
spec:
selector:
matchLabels:
app.kubernetes.io/name: secure-caller
template:
metadata:
labels:
app.kubernetes.io/name: secure-caller
spec:
containers:
- image: piomin/secure-caller
name: secure-caller
ports:
- containerPort: 8443
name: https
env:
- name: PASSWORD
valueFrom:
secretKeyRef:
key: password
name: jks-password-secret
# (2)
- name: CERT_PATH
value: /opt/secret
# (3)
- name: CLIENT_CERT_PATH
value: /opt/client-secret
- name: HOST
value: secure-callme
volumeMounts:
- mountPath: /opt/secret
name: cert
- mountPath: /opt/client-secret
name: client-cert
volumes:
# (5)
- name: cert
secret:
secretName: secure-caller-cert
- name: client-cert
secret:
secretName: secure-callme-certThen, go to the secure-caller directory and deploy the app. This time we enable port-forward to easily test the app locally.
$ skaffold run --port-forwardLet’s display a final list of all running apps. We have cert-manager components, the Stakater reloader, and our two sample Spring Boot apps.
$ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
my-cert-manager 1/1 1 1 1h
my-cert-manager-cainjector 1/1 1 1 1h
my-cert-manager-webhook 1/1 1 1 1h
my-reloader-reloader 1/1 1 1 1h
secure-caller 1/1 1 1 1h
secure-callme 1/1 1 1 1h
Testing Renew of Certificates on Kubernetes
The secure-caller app is available on the 8443 port locally, while the secure-callme app is available inside the cluster under the service name secure-callme. Let’s make a test call. Firstly, we need to download certificates and private keys stored on Kubernetes:
$ kubectl get secret secure-caller-cert \
-o jsonpath \
--template '{.data.tls\.key}' | base64 --decode > tls.key
$ kubectl get secret secure-caller-cert \
-o jsonpath \
--template '{.data.tls\.crt}' | base64 --decode > tls.crt
$ kubectl get secret secure-caller-cert \
-o jsonpath \
--template '{.data.ca\.crt}' | base64 --decode > ca.crtNow, we can call the GET /caller endpoint using the following curl command. Under the hood, the secure-caller calls the endpoint GET /callme exposed by the secure-callme also over HTTPS. If you did everything according to the instruction you should have the same result as below.
$ curl https://localhost:8443/caller \
--key tls.key \
--cert tls.crt \
--cacert ca.crt
I'm `secure-caller`! calling... I'm `secure-callme`!Our certificate is valid for one hour.
Let’s see what happens after one hour. A new certificate has already been generated and both our apps have been reloaded. Now, if try to call the same endpoint as before using old certificates you should see the following error.
Now, if you repeat the first step in that section it should work properly. We just need to download the certs to make a test. The internal communication over SSL works automatically after a reload of apps.
Final Thoughts
Of course, there are some other ways for achieving the same result as in our exercise. For example, you can a service mesh tool like Istio and enable mutual TLS for internal communication. You will still need to handle the automatic renewal of certificates somehow in that scenario. Cert Manager may be replaced with some other tools like HashiCorp Vault, which provides features for generating SSL/TLS certificates. You can as well use Spring Cloud Kubernetes with Spring Boot for watching for changes in secrets and reloading them without restarting the app. However, the solution used to renew certificates on Kubernetes presented in that article is simple and will work for any type of app.
Leave a Reply