This article will teach you how to configure a hot reload of SSL certificates for the Spring Boot app running on Kubernetes. We will use the two features introduced in the 3.1 and 3.2 versions of the Spring Boot framework. The first of them allows us to leverage SSL bundles for configuring and consuming a custom SSL trust material on both the server and client sides. The second one makes it easy to hot reload SSL certificates and keys for embedded web servers in the Spring Boot app. Let’s see how it works in practice!

In order to generate SSL certificates on Kubernetes we will use cert-manager. “Cert-manager” can rotate certificates after a specified period and save them as Kubernetes Secrets. I have already described how to implement a similar scenario with an automatic restart of a pod on a secret update in the article here. We were using the Stakater Reloader tool to restart the pod automatically on a new version of Secret. However, this time we use Spring Boot features to avoid having to restart an app (pod).

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the ssl directory. You will find two Spring Boot apps: secure-callme-bundle and secure-caller-bundle. After that, you should just follow my instructions. Let’s begin.

How It Works

Before we go into the technical details, let me write a little bit more about the architecture of our solution. Our challenge is pretty common. We need to design a solution for enabling SSL/TLS communication between the services running on Kubernetes. This solution must take into account a scenario of certificates reloading. Moreover, it must happen at the same time for the both server and client sides to avoid errors in the communication. On the server side, we use an embedded Tomcat server. In the client-side app, we use the Spring RestTemplate object.

“Cert-manager” can generate certificates automatically, based on the provided CRD object. It ensures the certificates are valid and up-to-date and will attempt to renew certificates before expiration. It serves all the required staff as the Kubernetes Secret. Such a secret is then mounted as a volume into the app pod. Thanks to that we don’t need to restart a pod, to see the latest certificates or “keystores” inside the pod. Here is the visualization of the described architecture.

Install cert-manager on Kubernetes

In order to install both “cert-manager” on Kubernetes we will use its Helm chart. We don’t need any specific settings. Before installing the chart we have to add CRD resources for the latest version 1.14.2:

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.2/cert-manager.crds.yaml

Then, we need to add the jetstack chart repository:

$ helm repo add jetstack https://charts.jetstack.io

After that, we can install the chart in the cert-manager namespace using the following command:

$ helm install my-release cert-manager jetstack/cert-manager \
    -n cert-manager

In order to verify that the installation finished successfully we can display a list of running pods:

$ kubectl get po
NAME                                          READY   STATUS    RESTARTS   AGE
my-cert-manager-578884c6cf-f9ppt              1/1     Running   0          1m
my-cert-manager-cainjector-55d4cd4bb6-6mgjd   1/1     Running   0          1m
my-cert-manager-webhook-5c68bf9c8d-nz7sd      1/1     Running   0 

That’s all. Now we can proceed to the implementation.

Spring Boot SSL Hot Reload on the Embedded Server

Sample App Implementation

Our first app secure-callme-bundle exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller-bundle app. Here’s the @RestController implementation:

@RestController
public class SecureCallmeController {

    @GetMapping("/callme")
    public String call() {
        return "I'm `secure-callme`!";
    }

}

Now our main goal is to enable HTTPS for that app and make it work properly on Kubernetes. First, we should change the default server port for the Spring Boot app to 8443 (1). Starting from Spring Boot 3.1 we can use the spring.ssl.bundle.* properties instead of the server.ssl.* properties to configure SSL trust material for the web server (3). There are two types of trusted material it can support. In order to configure bundles using Java keystore files, we have to use the spring.ssl.bundle.jks group. On the other hand, it is possible to configure bundles using PEM-encoded text files with the spring.ssl.bundle.pem properties group.

In the exercise, we will use the Java keystore files (JKS). We define a single SSL bundle under the server name. It contains both keystore and truststore locations. With the reload-on-update property, we can instruct Spring Boot to watch the files in the background and trigger a web server reload if they change. Additionally, we will force verification of the client’s certificate with the server.ssl.client-auth property (2). Finally, the name of the bundle needs to be set for the web server with the server.ssl.bundle property. Here’s the full configuration of our Spring Boot app inside the application.yml file.

# (1)
server.port: 8443

# (2)
server.ssl:
  client-auth: NEED
  bundle: server

# (3)
---
spring.config.activate.on-profile: prod
spring.ssl.bundle.jks:
  server:
    reload-on-update: true
    keystore:
      location: ${CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
    truststore:
      location: ${CERT_PATH}/truststore.jks
      password: ${PASSWORD}
      type: JKS

Generate Certificates with Cert-manager

Before we deploy the callme-secure-bundle app on Kubernetes, we need to configure “cert-manager” and generate the required certificates. Firstly, we need to define the CRD object responsible for issuing certificates. Here’s the ClusterIssuer object that generates self-signed certificates.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ss-cluster-issuer
spec:
  selfSigned: {}

Here’s the Kubernetes Secret with the password used for securing generated keystores:

kind: Secret
apiVersion: v1
metadata:
  name: jks-password-secret
data:
  password: MTIzNDU2
type: Opaque

After that, we can generate certificates. Here’s the Certificate object for the app. There are some important things here. First of all, we can generate key stores together with a certificate and private key (1). The object refers to the ClusterIssuer, which has been created in the previous step (2). The name of Kubernetes Service used during communication is secure-callme-bundle, so the cert needs to have that name as CN. In order to enable certificate rotation, we need to set validity time. The lowest possible value is 1 hour (4). So each time 5 minutes before expiration “cert-manager” will automatically renew a certificate (5). However, it won’t rotate the private key.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: secure-callme-cert
spec:
  keystores:
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef:
    name: ss-cluster-issuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - secure-callme-bundle
    - localhost
  secretName: secure-callme-cert
  commonName: secure-callme-bundle
  duration: 1h
  renewBefore: 5m

Deploy on Kubernetes

After creating a certificate we can proceed to the secure-callme-bundle app deployment. It mounts the Secret containing certificates and keystores as a volume. The name of the output Secret is determined by the value of the spec.secretName defined in the Certificate object. We need to inject some environment variables into the Spring Boot app. It requires the password to the keystores (PASSWORD), the location of the mounted trusted material inside the pod (CERT_PATH), and activate the prod profile (SPRING_PROFILES_ACTIVE).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-callme-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-callme-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-callme-bundle
    spec:
      containers:
        - image: piomin/secure-callme-bundle
          name: secure-callme-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            - name: CERT_PATH
              value: /opt/secret
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
      volumes:
        - name: cert
          secret:
            secretName: secure-callme-cert

Here’s the Kubernetes Service related to the app:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: secure-callme-bundle
  name: secure-callme-bundle
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: secure-callme-bundle
  type: ClusterIP

Firstly, make sure you are inside the secure-callme-bundle directory. Let’s build and run the app on Kubernetes with Skaffold and enable “port-forwarding” under 8443 port:

$ skaffold dev --port-forward

Skaffold will not only run the app but also apply all required Kubernetes objects defined in the app k8s directory. It applies also to the “cert-manager” Certificate object. Once the skaffold dev command finishes successfully, we access our HTTP endpoint under the http://127.0.0.1:8443 address.

Let’s call the GET /callme endpoint. Although, we enabled the --insecure option the request failed since the web server requires client authentication. To avoid it, we should include both key and certificate files in the curl command. However,

$ curl https://localhost:8443/callme --insecure -v
*   Trying [::1]:8443...
* Connected to localhost (::1) port 8443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Request CERT (13):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Certificate (11):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=secure-callme-bundle
*  start date: Feb 18 20:13:00 2024 GMT
*  expire date: Feb 18 21:13:00 2024 GMT
*  issuer: CN=secure-callme-bundle
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> GET /callme HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/8.4.0
> Accept: */*
>
* LibreSSL SSL_read: LibreSSL/3.3.6: error:1404C412:SSL routines:ST_OK:sslv3 alert bad certificate, errno 0
* Closing connection
curl: (56) LibreSSL SSL_read: LibreSSL/3.3.6: error:1404C412:SSL routines:ST_OK:sslv3 alert bad certificate, errno 0

Spring Boot SSL Hot Reload with RestTemplate

Sample App Implementation

Let’s switch to the secure-caller-bundle app. This app also exposes a single HTTP endpoint. Inside this endpoint implementation method, we call the GET /callme endpoint exposed by the secure-callme-bundle app. We use the RestTemplate bean for that.

@RestController
public class SecureCallerBundleController {

    RestTemplate restTemplate;

    @Value("${client.url}")
    String clientUrl;

    public SecureCallerBundleController(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @GetMapping("/caller")
    public String call() {
        return "I'm `secure-caller`! calling... " +
                restTemplate.getForObject(clientUrl, String.class);
    }
}

This time we need to define two SSL bundles in the application settings. The server bundle is for the web server, which is pretty similar to the bundle defined in the previous app sample. The client bundle is dedicated to the RestTemplate bean. It uses the keystore and truststore taken from Secret generated for the server-side app. With those files, the RestTemplate bean can authenticate against the secure-callme-bundle app. Of course, we also need to automatically reload the SslBundle bean after a certificate rotation.

server.port: 8443
server.ssl.bundle: server

---
spring.config.activate.on-profile: prod
client.url: https://${HOST}:8443/callme
spring.ssl.bundle.jks:
  server:
    reload-on-update: true
    keystore:
      location: ${CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
  client:
    reload-on-update: true
    keystore:
      location: ${CLIENT_CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
    truststore:
      location: ${CLIENT_CERT_PATH}/truststore.jks
      password: ${PASSWORD}
      type: JKS

Spring Boot 3.1 with the bundles’ concept extremely simplifies SSL context configuration for Spring REST clients like RestTemplate or WebClient. However, currently (Spring Boot 3.2.2) there is no built-in implementation for reloading e.g. Spring RestTemplate on the SslBundle update. Therefore we need to add a portion of code to achieve that. Fortunately, SslBundles allows us to define a custom handler that fires on the bundle update event. We need to define the handler for the client bundle. Once it receives a rotated version of SslBundle, it replaces the existing RestTemplate bean in the context with a new one using RestTemplateBuilder.

@SpringBootApplication
public class SecureCallerBundle {

   private static final Logger LOG = LoggerFactory
      .getLogger(SecureCallerBundle.class);

   public static void main(String[] args) {
      SpringApplication.run(SecureCallerBundle.class, args);
   }

   @Autowired
   ApplicationContext context;

   @Bean("restTemplate")
   RestTemplate builder(RestTemplateBuilder builder, SslBundles sslBundles) {
      sslBundles.addBundleUpdateHandler("client", sslBundle -> {
         try {
            LOG.info("Bundle updated: " + sslBundle.getStores().getKeyStore().getCertificate("certificate"));
         } catch (KeyStoreException e) {
            LOG.error("Error on getting certificate", e);
         }
         DefaultSingletonBeanRegistry registry = (DefaultSingletonBeanRegistry) context
            .getAutowireCapableBeanFactory();
         registry.destroySingleton("restTemplate");
         registry.registerSingleton("restTemplate", 
            builder.setSslBundle(sslBundle).build());
      });
      return builder.setSslBundle(sslBundles.getBundle("client")).build();
   }
}

Deploy on Kubernetes

Let’s take a look at the Kubernetes Deployment manifest for the current app. This time, we are mounting two secrets as volumes. The first one is generated for the current app web server, while the second one is generated for the secure-callme-bundle app and is used by the RestTemplate in establishing secure communication. We also set the address of the target service to inject it into the app (HOST) and activate the prod profile (SPRING_PROFILES_ACTIVE).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-caller-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-caller-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-caller-bundle
    spec:
      containers:
        - image: piomin/secure-caller-bundle
          name: secure-caller-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            - name: CERT_PATH
              value: /opt/secret
            - name: CLIENT_CERT_PATH
              value: /opt/client-secret
            - name: HOST
              value: secure-callme-bundle
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
            - mountPath: /opt/client-secret
              name: client-cert
      volumes:
        - name: cert
          secret:
            secretName: secure-caller-cert
        - name: client-cert
          secret:
            secretName: secure-callme-cert

Let’s deploy the app with the skaffold dev --port-forward command. Once again, it will deploy all the required staff on Kubernetes. Since we already exposed the secure-callme-bundle app with the “port-forward” option, the current app is exposed under the 8444 port.

Let’s try to call the GET /caller endpoint. Under the hood, it calls the endpoint exposed by the secure-callme-bundle app with RestTemplate. As you see, the secure communication is successfully established.

curl https://localhost:8444/caller --insecure -v
*   Trying [::1]:8444...
* Connected to localhost (::1) port 8444
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=secure-caller-bundle
*  start date: Feb 18 20:40:11 2024 GMT
*  expire date: Feb 18 21:40:11 2024 GMT
*  issuer: CN=secure-caller-bundle
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> GET /caller HTTP/1.1
> Host: localhost:8444
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 200
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 57
< Date: Sun, 18 Feb 2024 21:26:42 GMT
<
* Connection #0 to host localhost left intact
I'm `secure-caller`! calling... I'm secure-callme-bundle!

Now, we can wait one hour until the “cert-manager” rotates the certificate inside the secure-callme-cert Secret. However, we can also remove the secret, since “cert-manager” will regenerate it based on the Certificate object. Here’s the secret with certificates and keystores used to establish secure communication between both our sample Spring Boot apps.

No matter if you wait until the 1h rotation occurs or do it manually by removing the secret, you should see the following log inside the secure-callme-bundle app pod. It means that Spring Boot has received the SslBundle update event and then reloaded a Tomcat server.

The SslBundle event is also handled on the secure-caller-bundle app side. It refreshes the RestTemplate bean and prints information with the latest certificate in the logs.

Final Thoughts

The latest releases of Spring Boot simplify the management of SSL certificates on the both server and client sides a lot. Thanks to SslBundles we can easily handle the certificate rotation process without restarting the pod on Kubernetes. There are still some other things to consider don’t covered by this article. It includes the mechanism of distributing the trust bundles across the apps. However, for example, to manage trust bundles in the Kubernetes environment we can use the “cert-manager” trust-manager feature.