In this article, we will make a performance comparison between several most popular JDK implementations for the app running on Kubernetes. This post also answers some questions and concerns about my Twitter publication you see below. I compared Oracle JDK with Eclipse Temurin. The result was quite surprising for me, so I decided to tweet to get some opinions and feedback.

Unfortunately, those results were wrong. Or maybe I should say, were not averaged well enough. After this publication, I also received interesting materials presented on London Java Community. It compares the performance of the Payara application server running on various JDKs. Here’s the link to that presentation (~1h). The results showed there seem to confirm my results. Or at least they confirm the general rule – there are some performance differences between Open JDK implementations. Let’s check it out.

This time I’ll do a very accurate comparison with several repeats to get reproducible results. I’ll test the following JVM implementations:

For all the tests I’ll use Paketo Java buildpack. We can easily switch between several JVM implementations with Paketo. I’ll test a simple Spring Boot 3 app that uses Spring Data to interact with the Mongo database. Let’s proceed to the details!

If you have already built images with Dockerfile it is possible that you were using the official OpenJDK base image from the Docker Hub. However, currently, the announcement on the image site says that it is officially deprecated and all users should find suitable replacements. In this article, we will compare all the most popular replacements, so I hope it may help you to make a good choice 🙂

Testing Environment

Before we run tests it is important to have a provisioned environment. I’ll run all the tests locally. In order to build images, I’m going to use Paketo Buildpacks. Here are some details of my environment:

1. Machine: MacBook Pro 32G RAM Intel 
2. OS: macOS Ventura 13.1
3. Kubernetes (v1.25.2) on Docker Desktop: 14G RAM + 4vCPU

We will use Java 17 for app compilation. In order to run load tests, I’m going to leverage the k6 tool. Our app is written in Spring Boot. It connects to the Mongo database running on the same instance of Kubernetes. Each time I’m testing a new JVM provider I’m removing the previous version of the app and database. Then I’m deploying the new, full configuration once again. We will measure the following parameters:

1. App startup time (the best  
2. result and average) – we will read it directly from the Spring Boot logs 
3. Throughput – with k6 we will simulate 5 and 10 virtual users. It will measure the number of processing requests 
4. The size of the image
5. The RAM memory consumed by the pod during the load tests. Basically, we will execute the kubectl top pod command

We will also set the memory limit for the container to 1G. In our load tests, the app will insert data into the Mongo database. It is exposing the REST endpoint invoked during the tests. To measure startup time as accurately as possible I’ll restart the app several times.

Let’s take a look at the Deployment YAML manifest. It injects credentials to the Mongo database and set the memory limit to 1G (as I already mentioned):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-spring-boot-on-kubernetes-deployment
spec:
  selector:
    matchLabels:
      app: sample-spring-boot-on-kubernetes
  template:
    metadata:
      labels:
        app: sample-spring-boot-on-kubernetes
    spec:
      containers:
      - name: sample-spring-boot-on-kubernetes
        image: piomin/sample-spring-boot-on-kubernetes
        ports:
        - containerPort: 8080
        env:
          - name: MONGO_DATABASE
            valueFrom:
              configMapKeyRef:
                name: mongodb
                key: database-name
          - name: MONGO_USERNAME
            valueFrom:
              secretKeyRef:
                name: mongodb
                key: database-user
          - name: MONGO_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mongodb
                key: database-password
          - name: MONGO_URL
            value: mongodb
        readinessProbe:
          httpGet:
            port: 8080
            path: /readiness
            scheme: HTTP
          timeoutSeconds: 1
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        resources:
          limits:
            memory: 1024Mi

Source Code and Images

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. You will also find all the images in my Docker Hub repository piomin/sample-spring-boot-on-kubernetes. Every single image is tagged with the vendor’s name.

Our Spring Boot app exposes several endpoints, but I’ll test the POST /persons endpoint for inserting data into Mongo. In the integration with Mongo, I’m using the Spring Data MongoDB project and its CRUD repository pattern.

// controller

@RestController
@RequestMapping("/persons")
public class PersonController {

   private PersonRepository repository;

   PersonController(PersonRepository repository) {
      this.repository = repository;
   }

   @PostMapping
   public Person add(@RequestBody Person person) {
      return repository.save(person);
   }

   // other endpoints implementation
}


// repository

public interface PersonRepository extends CrudRepository<Person, String> {

   Set<Person> findByFirstNameAndLastName(String firstName, 
                                          String lastName);
   Set<Person> findByAge(int age);
   Set<Person> findByAgeGreaterThan(int age);

}

The Size of the Image

The size of the image is the simplest option to measure. If you would like to check what is exactly inside the image you can use the dive tool. The difference in the size between vendors results from the number of java tools and binaries included inside. From my perspective, the smaller the size the better. I’d rather not use anything that is inside the image. Of course, except all the staff required to run my app successfully. But you may have a different case. Anyway, here’s the content of the app for the Oracle JDK after executing the dive piomin/sample-spring-boot-on-kubernetes:oracle command. As you see, JDK takes up most of the space.

On the other hand, we can analyze the smallest image. I think it explains the differences in image size since Zulu contains JRE, not the whole JDK.

Here are the result ordered from the smallest image to the biggest.

Let’s visualize our first results. I think it excellent shows which image contains JDK and which JRE.

Startup Time

Honestly, it is not very easy to measure a startup time, since the difference between the vendors is not large. Also, the subsequent results for the same provider may differ a lot. For example, on the first try the app starts in 5.8s and after the pod restart 8.4s. My methodology was pretty simple. I restarted the app several times for each JDK provider to measure the average startup time and the fastest startup in the series. Then I repeated the same exercise again to verify if the results are repeatable. The proportions between the first and second series of startup time between corresponding vendors were similar. In fact, the difference between the fastest and the slowest average startup time is not large. I get the best result for Eclipse Temurin (7.2s) and the worst for IBM Semeru OpenJ9 (9.05s).

Let’s see the full list of results. It shows the average startup time of the application from the fastest one.

Once again, here’s the graphical representation of our results. The differences between vendors are sometimes rather cosmetic. Maybe, if the same exercise once again from the beginning the results would be quite different.

As I mentioned before, I also measured the fastest attempt. This time the best top 3 are Eclipse Temurin, Amazon Corretto, and BellSoft Liberica.

Memory

I’m measuring the memory usage of the app under the heavy load with a test simulating 10 users continuously sending requests. It gives me a really large throughput at the level of the app – around 500 requests per second. The results are in line with the expectations. Almost all the vendors have very similar memory usage except IBM Semeru, which uses OpenJ9 JVM. In theory, OpenJ9 should also give us a better startup time. However, in my case, the significant difference is just in the memory footprint. For IBM Semeru the memory usage is around 135MB, while for other vendors it varies in the range of 210-230MB.

Here’s the graphical visualization of our results:

Throughput

In order to generate high incoming traffic to the app I used the k6 tool. It allows us to create tests in JavaScript. Here’s the implementation of our test. It is calling the HTTP POST /persons endpoint with input data in JSON. Then it verifies if the request has been successfully processed on the server side.

import http from 'k6/http';
import { check } from 'k6';

export default function () {

  const payload = JSON.stringify({
      firstName: 'aaa',
      lastName: 'bbb',
      age: 50,
      gender: 'MALE'
  });

  const params = {
    headers: {
      'Content-Type': 'application/json',
    },
  };

  const res = http.post(`http://localhost:8080/persons`, payload, params);

  check(res, {
    'is status 200': (res) => res.status === 200,
    'body size is > 0': (r) => r.body.length > 0,
  });
}

Here’s the k6 command for running our test. It is possible to define the duration and number of simultaneous virtual users. In the first step, I’m simulating 5 virtual users:

$ k6 run -d 90s -u 5 load-tests.js

Then, I’m running the tests for 10 virtual users twice per vendor.

$ k6 run -d 90s -u 10 load-tests.js

Here are the sample results printed after executing the k6 test:

I repeated the exercise per the JDK vendor. Here are the throughput results for 5 virtual users:

Here are the throughput results for 10 virtual users:

Final Thoughts

After repeating the load tests several times I need to admit that there are no significant differences in performance between all JDK vendors. We were using the same JVM settings for testing (set by the Paketo Buildpack). Probably, the more tests I will run, the results between different vendors would be even more similar. So, in summary, the results from my tweet have not been confirmed. Ok, so let’s back to the question – which JDK to choose on Kubernetes?

Probably it somehow depends on where you are running your cluster. If for example, it’s Kubernetes EKS on AWS it’s worth using Amazon Corretto. However, if you are looking for the smallest image size you should choose between Azul Zulu, IBM Semeru, BellSoft Liberica, and Adoptium Eclipse Temurin. Additionally, IBM Semeru will consume significantly less memory than other distributions, since it is built on top of OpenJ9.

Don’t forget about best practices when deploying Java apps on Kubernetes. Here’s my article about it.