This article will teach you how to dynamically create and rotate SSL certificates used in service-to-service communication with OpenShift and Spring Boot. We will achieve it with a single annotation on a Kubernetes service and the “SSL Bundles” mechanism introduced in Spring Boot 3.1. For generating the SSL on OpenShift, we will use the mechanism called “Service Serving Certificates”. It generates a certificate and key in PEM format. With Spring Boot 3.1 SslBundles, it won’t be a problem for our Java app, since it supports PEM-encoded certificates. The “Service Serving Certificates” mechanism automatically rotates CA certificates every 13 months.

There is a similar article on my blog about “cert-manager” and Spring Boot SSL hot reload more focused on Kubernetes. You can also read the post on how to integrate Spring Boot with Vault PKI to generate certs automatically.

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Let’s check out the openshift branch. Then, we need to go to the ssl directory. You will find two Spring Boot apps: secure-callme-bundle and secure-caller-bundle. After that, you should just follow my instructions. Let’s begin.

$ git clone https://github.com/piomin/sample-spring-security-microservices.git
$ git checkout openshift
$ cd ssl

How It Works

OpenShift brings a lot of interesting features to the Kubernetes platform. The “Service Serving Certificates” is one of the examples. Thanks to that mechanism, we don’t need to install any external tool for generating and rotating certificates dynamically like “cert-manager”. The OpenShift CA service automatically rotates certificates 13 months before expiration and issues new certificates for 26 months. After a rotation, the previous CA is trusted until it expires. To achieve it, OpenShift implements the root CA key rollover process described in  https://tools.ietf.org/html/rfc4210#section-4.4. This allows a grace period for all affected services to refresh their key material before the expiration. Sounds interesting? Let’s proceed to the implementation.

In order to test a mechanism, we will run two simple Spring Boot apps on OpenShift that communicate using SSL certificates. OpenShift takes care of issuing/rotating certificates and exposing the CA bundle as a ConfigMap. Access to this CA certificate allows TLS clients to verify connections to services using Service Serving Certificates”. Here’s the illustration of our apps’ architecture on OpenShift.

Create a Server Side App

Our first app secure-callme-bundle exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller-bundle app. Here’s the @RestController implementation:

@RestController
public class SecureCallmeController {

    @GetMapping("/callme")
    public String call() {
        return "I'm `secure-callme`!";
    }

}

Now, our main goal is to enable HTTPS for that app and make it work properly with the OpenShift “Service Serving Certificates”. Firstly, we should change the default server port for the Spring Boot app to 8443 (1). Starting from Spring Boot 3.1 we can use the spring.ssl.bundle.* properties instead of the server.ssl.* properties to configure keystores for the web server (3). We can configure SSL bundles using PEM-encoded text files directly in the app with the spring.ssl.bundle.pem properties group. The certificate (tls.crt) and private key (tls.key) will be mounted as a volume to the pod. Finally, the name of the bundle needs to be set for the web server with the server.ssl.bundle property (2). Here’s the full configuration of our Spring Boot app inside the application.yml file.

# (1)
server.port: 8443

# (2)
server.ssl.bundle: server

# (3)
---
spring.config.activate.on-profile: prod
spring.ssl.bundle.pem:
  server:
    keystore:
      certificate: ${CERT_PATH}/tls.crt
      private-key: ${CERT_PATH}/tls.key

The only thing we need to do to generate SSL certificates for our app is to annotate the Kubernetes Service with service.beta.openshift.io/serving-cert-secret-name. It should contain the name of the target Kubernetes Secret as a value.

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: secure-callme-bundle
  annotations:
    service.beta.openshift.io/serving-cert-secret-name: secure-callme-bundle
  name: secure-callme-bundle
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: secure-callme-bundle
  type: ClusterIP

Once we apply such a Service to the cluster, OpenShift generates the secure-callme-bundle Secret that contains the certificate and private key.

After that, we just need to deploy the app and mount the generated Secret as a volume. The volume path is passed to the app as the CERT_PATH environment variable.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-callme-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-callme-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-callme-bundle
    spec:
      containers:
        - image: piomin/secure-callme-bundle
          name: secure-callme-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: CERT_PATH
              value: /opt/secret
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
      volumes:
        - name: cert
          secret:
            secretName: secure-callme-bundle

Finally, we need to expose the HTTP endpoint outside of the cluster. We can easily do it with the OpenShift Route. Don’t forget the set the TLS termination as passthrough. Thanks to that OpenShift router sends traffic directly to the destination without TLS termination.

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: secure-callme-bundle
  labels:
    app: secure-callme-bundle
    app.kubernetes.io/component: secure-callme-bundle
    app.kubernetes.io/instance: secure-callme-bundle
  annotations:
    openshift.io/host.generated: 'true'
spec:
  tls:
    termination: passthrough
  to:
    kind: Service
    name: secure-callme-bundle
    weight: 100
  port:
    targetPort: https
  wildcardPolicy: None

Create a Client Side App

Let’s switch to the secure-caller-bundle app. This app also exposes a single HTTP endpoint. Inside this endpoint implementation method, we call the GET /callme endpoint exposed by the secure-callme-bundle app. We use the RestTemplate bean for that.

@RestController
public class SecureCallerBundleController {

    RestTemplate restTemplate;

    @Value("${client.url}")
    String clientUrl;

    public SecureCallerBundleController(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @GetMapping("/caller")
    public String call() {
        return "I'm `secure-caller`! calling... " +
                restTemplate.getForObject(clientUrl, String.class);
    }
}

This time we need to define two SSL bundles in the application settings. The server bundle is for the web server, which is pretty similar to the bundle defined in the previous app sample. The client bundle is dedicated to the RestTemplate bean. It uses the truststore taken from ConfigMap with CA certificates. With the certs inside the service-ca.crt file, the RestTemplate bean can authenticate against the secure-callme-bundle app.

server.port: 8443
server.ssl.bundle: server

---
spring.config.activate.on-profile: prod
client.url: https://${HOST}:8443/callme
spring.ssl.bundle.pem:
  server:
    keystore:
      certificate: ${CERT_PATH}/tls.crt
      private-key: ${CERT_PATH}/tls.key
  client:
    truststore:
      certificate: ${CLIENT_CERT_PATH}/service-ca.crt

Don’t forget about creating the RestTemplate bean. It needs to the SslBundles bean and set it on the RestTemplateBuilder.

@Bean
RestTemplate builder(RestTemplateBuilder builder, SslBundles sslBundles) {
   return builder
      .setSslBundle(sslBundles.getBundle("client")).build();
}

We need to create the ConfigMap annotated with service.beta.openshift.io/inject-cabundle=true. After that, the OpenShift cluster automatically injects the service CA certificate into the service-ca.crt key on the config map.

apiVersion: v1
kind: ConfigMap
metadata:
  name: secure-caller-bundle
  namespace: demo-secure
  annotations:
    service.beta.openshift.io/inject-cabundle: "true"
data: {}

Let’s display the details of the secure-caller-bundle ConfigMap. As you see, OpenShift injects the PEM-encoded CA certificates under the service-ca.crt key to that config map.

Finally, we can deploy our client-side app. Here’s the YAML manifest with the Deployment object. Since the secure-caller-bundle app acts as a web server and also uses a REST client to invoke the secure-callme-bundle we need to mount two volumes. The first one (/opt/secret) contains a certificate and private key for the web server (1). Consequently, the second one (/opt/client-cert) holds the CA bundle used by the RestTemplate client (2). This client communicates with secure-callme-bundle internally through the Kubernetes Service (3).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-caller-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-caller-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-caller-bundle
    spec:
      containers:
        - image: piomin/secure-caller-bundle
          name: secure-caller-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: CERT_PATH
              value: /opt/secret
            - name: CLIENT_CERT_PATH
              value: /opt/client-cert
            # (3)
            - name: HOST
              value: secure-callme-bundle.demo-secure.svc
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            # (1)
            - mountPath: /opt/secret
              name: cert
            # (2)
            - mountPath: /opt/client-cert
              name: client-cert
      volumes:
        - name: cert
          secret:
            secretName: secure-caller-bundle
        - name: client-cert
          configMap:
            name: secure-caller-bundle

Of course, we should also create a passthrough Route to expose our app the outside of the cluster:

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: secure-caller-bundle
  labels:
    app: secure-caller-bundle
    app.kubernetes.io/component: secure-caller-bundle
    app.kubernetes.io/instance: secure-caller-bundle
  annotations:
    openshift.io/host.generated: 'true'
spec:
  tls:
    termination: passthrough
  to:
    kind: Service
    name: secure-caller-bundle
    weight: 100
  port:
    targetPort: https
  wildcardPolicy: None

Testing OpenShift Certs SSL Rotation with Spring Boot

The main problem with testing the certs rotation scenario is in the 13-month interval. We can’t change that period in the configuration. However, there is a way how we can force OpenShift to rotate the CA certs on demand. In order to do that, we need to set the spec.unsupportedConfigOverrides.forceRotation.reason property on the cluster-wide ServiceCA object with any value. Of course, you shouldn’t do that at production, since it is not supported by OpenShift. It is just designed for testing or development purposes.

apiVersion: operator.openshift.io/v1
kind: ServiceCA
metadata:
spec:
  logLevel: Normal
  managementState: Managed
  operatorLogLevel: Normal
  unsupportedConfigOverrides:
    forceRotation:
      reason: testing

We will call our apps through the OpenShift Route. So, let’s display a current list of routes related to our sample apps:

$ oc get route
NAME                   HOST/PORT                                                     PATH   SERVICES               PORT    TERMINATION   WILDCARD
secure-caller-bundle   secure-caller-bundle-demo-secure.apps.ocp1.eastus.aroapp.io          secure-caller-bundle   https   passthrough   None
secure-callme-bundle   secure-callme-bundle-demo-secure.apps.ocp1.eastus.aroapp.io          secure-callme-bundle   https   passthrough   None

Once we set that property, OpenShift immediately rotates that CA certificate. It will also take care of synchronizing the latest certificates with all the secrets and config maps containing annotations required by the “Service Serving Certificates” mechanism. As you see, now, our secure-caller-bundle ConfigMap contains two CAs: an old one and a new one.

$ oc describe cm secure-caller-bundle
Name:         secure-caller-bundle
Namespace:    demo-secure
Labels:       <none>
Annotations:  service.beta.openshift.io/inject-cabundle: true

Data
====
service-ca.crt:
----
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIITnr+EqYsYtEwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwODY0MDc4OTAe
Fw0yNDAzMDExMDM5MTVaFw0yNjA0MzAxMDM5MTZaMDYxNDAyBgNVBAMMK29wZW5z
aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDg2NDA3ODkwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDN0No0DIZ+BCdPvQ9TIvEjwGUSoUXM+DYI
Dxlk2Ef6goFv3YsAgNJTxznntRc69dJTrLtRrmFN0d31JPM/gJOMkKMvTWnz8Qzx
athuvvwbivJNKh/qn+Dhewbjrx5LGpq3v9VQ7t/5Rf5F9VMpyp738EwfAy2cfxTA
sxgYpvU/foivM9U0uMaZyXA5vLsGN6cKpEREGMqKbIKSXNYXD9/lbHv2eyr+5+s9
keYEDkGTWMceISihm5mGNwOZpLZhNwFgSZR9O63TIgGc/TEZ3EYwcIjmlVOSwQ/u
sM43HL1jtYKw3AovQOc5qc2kT5eSPV7jVCCfNYnj3BLvt0AYUXE/AgMBAAGjYzBh
MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBT3iI1f
71ZVt6OVLC/+0oxNL7BqhzAfBgNVHSMEGDAWgBT3iI1f71ZVt6OVLC/+0oxNL7Bq
hzANBgkqhkiG9w0BAQsFAAOCAQEATcT/LK5pAaOGz0j3MR4NhsfmAjo4K8SoGmFE
F4lDzDWZTaJuJHCHqpblgGKLfokU9BQC3L4NZLIisOj3CWF3zIpC6MOsU7x9qDm0
aZpI1gqhRXQZOqVpE9XJ0XBEOLP9hlzxa66SQlALlU5CWbDNEMtXsMMQf1lSgIRz
arWK8QwmPsmg6duoVtFcCMY2klkVacZctiMgM4wvf1CCJt3TkiggwO3t4IdiV8Xa
Z9d/LlV/bFzboxGxOb2IMHj5VXKPM6HmC4slBSqKeyr7PYyM7rjhorIOOner9JrX
RhbC8/uARciLbcFKyOOCSLoe5m1i8QPrRmCm/GhVS9M4x86TXw==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIIFlMUWK4HjW4wDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwODY0MDc4OTAe
Fw0yNDAyMjgxMTUyMjNaFw0yNjA0MjgxMTUyMjRaMDYxNDAyBgNVBAMMK29wZW5z
aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDg2NDA3ODkwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDY6AZbLN0kT2BPXbDn7iZmbyYZgyI8Y3jh
hRq9JrX0ZIIs5BvFcRRkm2IcJ8gsMF4VztGCkNoDQ07ojAIHI/FxssAU8wwoz2QP
eH2qzBpLA1lBAYvtjki//55STJvJF+7Z6qbcDwUVX6/r+hrpy5MIjaQmRV4DRLpp
ZD3HrqYozJEvCKaA2pinC2VyW8IBDB0FDvPdBxvzjrCDjssr/v0jqFgUGFJJmKEj
EaFQACeMlOS4Q5avEglfwoLS+RGgPjE1s1gpxir2dZCdnM8b9B3CFy2sS+SSHN4y
+FJRJBv4CbMAMP+DLIsHY872c2XmOlleJNFpiFWi9r33LCc7MnbjAgMBAAGjYzBh
MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRey/oP
FWW5kG8ifSXqIXBh2L+FdzAfBgNVHSMEGDAWgBT3iI1f71ZVt6OVLC/+0oxNL7Bq
hzANBgkqhkiG9w0BAQsFAAOCAQEAhbimn2Tv5w9Cy7i6dNdYYdVhA52cCnWWDDhf
s/dZ+Zl1aP3f179XO0xz3ovLuE227Z9AYPGtPW6c0FQgxgDgl2pYUX4J88caTUG/
eR9EG9LEfrL6OlKMjMvAFWrL64bcAAiuoz/gWB0a29xMnUsA52+7OpnzmHMduaaz
5JfRaTwkjGGe8ay2XsTUNrNGqZJGKHhE+hrUVlFsS7GpaSEe9GllkS9lGUZp2Hei
lTxwnBr9j+aP/QWXwuAVxT1DUAByJYoTPrpMhk+hKQUj/8TvxmMmsksZ57steqRI
/gWYd4j2SohN3jd1mv6rQ+jsGGAfCNmYVZL1oevNaqfnENt0IA==
-----END CERTIFICATE-----

However, if we don’t restart the pods, our apps still use the old keystore and truststore (unless they have the reload-on-update option enabled).

Here’s a list of running pods.

oc get po
NAME                                    READY   STATUS    RESTARTS   AGE
secure-caller-bundle-57c7658d85-bj4q5   1/1     Running   0          2d23h
secure-callme-bundle-cbf59569f-n6gg5    1/1     Running   0          2d23h

Let’s just restart the secure-callme-bundle app by deleting its pod. Do not restart the secure-callme-bundle pod, because we want to test the scenario where it uses the old truststore.

$ oc delete po secure-callme-bundle-cbf59569f-n6gg5
pod "secure-callme-bundle-cbf59569f-n6gg5" deleted

Then, we can call the GET /callme endpoint exposed by the secure-callme-bundle app through the Route once again. As you see, it servers the latest, reloaded certificate.

Let’s perform a final test. Now, we call the GET /caller endpoint exposed by the secure-caller-bundle app. Under the hood, it uses RestTemplate to call the GET /callme endpoint exposed by the secure-callme-bundle app via secure-callme-bundle.demo-secure.svc address. As you see the communication still works fine although the secure-caller-bundle app still uses the keystore and truststore.

Final Thoughts

The OpenShift “Service Serving Certificates” mechanism is simple to configure and doesn’t require that you handle the process of reloading apps with the rotated trusted material. It also has some drawbacks. You cannot change the rotation period from 13 months into a different value, and there is a global CA bundle shared across all the services running on the cluster. If those things are problematic for you I advise you to use “cert-manager” and a tool that allows you to restart pods automatically on the Secret or ConfigMap change. It is also possible to leverage the Spring SslBundles to reload certs without restarting the pod. With Spring Boot apps you can also use OpenShift Service Mesh, which can handle SSL communication at the platform level.