Spring Cloud Kubernetes with Spring Boot 3
In this article, you will learn how to create, test, and run apps with Spring Cloud Kubernetes, and Spring Boot 3. You will see how to use tools like Skaffold, Testcontainers, Spring Boot Admin, and the Fabric8 client in the Kubernetes environment. The main goal of this article is to update you with the latest version of the Spring Cloud Kubernetes project. There are several other posts on my blog with similar content. You can refer to the following article describing the best practices for running Java apps on Kubernetes. You can also read about microservices with Spring Cloud Kubernetes in the post published some years ago. It is quite outdated. I’ll show some changes since then. Let’s begin!
Source Code
If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should just follow my instructions.
Firstly, let’s discuss our repository. It contains five apps. There are three microservices (employee-service, department-service, organization-service) communicating with each other through the REST client and connecting to the Mongo database. There is also the API gateway (gateway-service) created with the Spring Cloud Gateway project. Finally, the admin-service directory contains the Spring Boot Admin app used for monitoring all other apps. You can easily deploy all the apps from the source code using a single Skaffold command. If you run the following command from the repository root directory it will build the images with Jib Maven Plugin and deploy all apps on your Kubernetes cluster:
$ skaffold runOn the other hand, you can go to the particular app directory and deploy only it using exactly the same command. All the required Kubernetes YAML manifests for each app are placed inside the k8s directories. There is also a global configuration with e.g. Mongo deployment in the project root k8s directory. Here’s the structure of our sample repo:
How It Works
In our sample architecture, we will use Spring Cloud Kubernetes Config for injecting configuration via ConfigMap and Secret and Spring Cloud Kubernetes Discovery for inter-service communication with the OpenFeign client. All our apps are running within the same namespace, but we could as well deploy them across several different namespaces and handle communication between them with OpenFeign. The only thing we should do in that case is to set the property spring.cloud.kubernetes.discovery.all-namespaces to true. For more details, you can refer to the following article.
In front of our services, there is an API gateway. It is a separate app, but we could as well install it on Kubernetes using the native CRD integration. For more details, you can refer to the following post on the Spring blog. In our case, this is a standard Spring Boot 3 app that just includes and uses the Spring Cloud Gateway module. It also uses Spring Cloud Kubernetes Discovery together with Spring Cloud OpenFeign to locate and call the downstream services. Here’s the diagram that illustrates our architecture.
Using Spring Cloud Kubernetes Config
I’ll describe implementation details by the example of department-service. It exposes some REST endpoints but also calls the endpoints exposed by the employee-service. Besides the standard modules, we need to include Spring Cloud Kubernetes in the Maven dependencies. Here, we have to decide if we use the Fabric8 client or the Kubernetes Java Client. Personally, I have an experience with Fabric8, so I’ll use the spring-cloud-starter-kubernetes-fabric8-all starter to include both config and discovery modules.
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-kubernetes-fabric8-all</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>As you see our app is connecting to the Mongo database. Let’s provide connection details and credentials required by the app. In the k8s directory, you will find the configmap.yaml file. It contains the address of Mongo and the database name. Those properties are injected into the pod as the application.properties file. And now the most important thing. The name of the ConfigMap has to be the same as the name of our app. The name of the Spring Boot is indicated by the spring.application.name property.
kind: ConfigMap
apiVersion: v1
metadata:
name: department
data:
application.properties: |-
spring.data.mongodb.host: mongodb
spring.data.mongodb.database: admin
spring.data.mongodb.authentication-database: adminIn the current case, the name of the app is department. Here’s the application.yml file inside the app:
spring:
application:
name: departmentThe same naming rule applies to Secret. We are keeping sensitive data like the username and password to the Mongo database inside the following Secret. You can also find that content inside the secret.yaml file in the k8s directory.
kind: Secret
apiVersion: v1
metadata:
name: department
data:
spring.data.mongodb.password: UGlvdF8xMjM=
spring.data.mongodb.username: cGlvdHI=
type: OpaqueNow, let’s proceed to the Deployment manifest. We will clarify the two first points here later. Spring Cloud Kubernetes requires special privileges on Kubernetes to interact with the master API (1). We don’t have to provide a tag for the image – Skaffold will handle it (2). In order to enable loading properties from ConfigMap we need to set the spring.config.import=kubernetes: property (a new way) or set the property spring.cloud.bootstrap.enabled to true (the old way). Instead of using properties directly, we will set the corresponding environment variables on the Deployment (3). By default, consuming secrets through the API is not enabled for security reasons. In order to enable it, we will set the SPRING_CLOUD_KUBERNETES_SECRETS_ENABLEAPI environment variable to true (4).
apiVersion: apps/v1
kind: Deployment
metadata:
name: department
labels:
app: department
spec:
replicas: 1
selector:
matchLabels:
app: department
template:
metadata:
labels:
app: department
spec:
serviceAccountName: spring-cloud-kubernetes # (1)
containers:
- name: department
image: piomin/department # (2)
ports:
- containerPort: 8080
env:
- name: SPRING_CLOUD_BOOTSTRAP_ENABLED # (3)
value: "true"
- name: SPRING_CLOUD_KUBERNETES_SECRETS_ENABLEAPI # (4)
value: "true"Using Spring Cloud Kubernetes Discovery
We have already included the Spring Cloud Kubernetes Discovery module in the previous section using the spring-cloud-starter-kubernetes-fabric8-all starter. In order to provide a declarative REST client we will also include the Spring Cloud OpenFeign module:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>Now, we can declare the @FeignClient interface. The important thing here is the name of a discovered service. It should be the same as the name of the Kubernetes Service defined for the employee-service app.
@FeignClient(name = "employee")
public interface EmployeeClient {
@GetMapping("/department/{departmentId}")
List<Employee> findByDepartment(@PathVariable("departmentId") String departmentId);
@GetMapping("/department-with-delay/{departmentId}")
List<Employee> findByDepartmentWithDelay(@PathVariable("departmentId") String departmentId);
}Here’s the Kubernetes Service manifest for the employee-service app. The name of the service is employee (1). The label spring-boot is set for Spring Boot Admin discovery purposes (2). You can find the following YAML in the employee-service/k8s directory.
apiVersion: v1
kind: Service
metadata:
name: employee # (1)
labels:
app: employee
spring-boot: "true" # (2)
spec:
ports:
- port: 8080
protocol: TCP
selector:
app: employee
type: ClusterIPJust to clarify – here’s the implementation of the employee-service API methods called by the OpenFeign client in the department-service.
@RestController
public class EmployeeController {
private static final Logger LOGGER = LoggerFactory
.getLogger(EmployeeController.class);
@Autowired
EmployeeRepository repository;
// ... other endpoints implementation
@GetMapping("/department/{departmentId}")
public List<Employee> findByDepartment(@PathVariable("departmentId") String departmentId) {
LOGGER.info("Employee find: departmentId={}", departmentId);
return repository.findByDepartmentId(departmentId);
}
@GetMapping("/department-with-delay/{departmentId}")
public List<Employee> findByDepartmentWithDelay(@PathVariable("departmentId") String departmentId) throws InterruptedException {
LOGGER.info("Employee find: departmentId={}", departmentId);
Thread.sleep(2000);
return repository.findByDepartmentId(departmentId);
}
}That’s all that we have to do. Now, we can just call the endpoint using the OpenFeign client from department-service. For example on the “delayed” endpoint, we can use Spring Cloud Circuit Breaker with Resilience4J.
@RestController
public class DepartmentController {
private static final Logger LOGGER = LoggerFactory
.getLogger(DepartmentController.class);
DepartmentRepository repository;
EmployeeClient employeeClient;
Resilience4JCircuitBreakerFactory circuitBreakerFactory;
public DepartmentController(
DepartmentRepository repository,
EmployeeClient employeeClient,
Resilience4JCircuitBreakerFactory circuitBreakerFactory) {
this.repository = repository;
this.employeeClient = employeeClient;
this.circuitBreakerFactory = circuitBreakerFactory;
}
@GetMapping("/{id}/with-employees-and-delay")
public Department findByIdWithEmployeesAndDelay(@PathVariable("id") String id) {
LOGGER.info("Department findByIdWithEmployees: id={}", id);
Department department = repository.findById(id).orElseThrow();
CircuitBreaker circuitBreaker = circuitBreakerFactory.create("delayed-circuit");
List<Employee> employees = circuitBreaker.run(() ->
employeeClient.findByDepartmentWithDelay(department.getId()));
department.setEmployees(employees);
return department;
}
@GetMapping("/organization/{organizationId}/with-employees")
public List<Department> findByOrganizationWithEmployees(@PathVariable("organizationId") String organizationId) {
LOGGER.info("Department find: organizationId={}", organizationId);
List<Department> departments = repository.findByOrganizationId(organizationId);
departments.forEach(d -> d.setEmployees(employeeClient.findByDepartment(d.getId())));
return departments;
}
}Testing with Fabric8 Kubernetes
We have already finished the implementation of our service. All the Kubernetes YAML manifests are prepared and ready to deploy. Now, the question is – can we easily test that everything works fine before we proceed to the deployment on the real cluster? The answer is – yes. Moreover, we can choose between several tools. Let’s begin with the simplest option – Kubernetes mock server. In order to use it, we to include an additional Maven dependency:
<dependency>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-server-mock</artifactId>
<version>6.7.1</version>
<scope>test</scope>
</dependency>Then, we can proceed to the test. In the first step, we need to provide several test annotations. Inside @SpringBootTest we should simulate the Kubernetes platform with spring.main.cloud-platform property set to KUBERNETES (1). Normally Spring Boot is able to autodetect if it is running on Kubernetes. In that case, we need “trick him”, because we are just simulating the API, not running the test on Kubernetes. We also need to enable the old way of ConfigMap injection with the spring.cloud.bootstrap.enabled=true property.
Once we annotate the test method with @EnableKubernetesMockClient (2) we can use an auto-configured static instance of the Fabric8 KubernetesClient (3). During the test Fabric8 library runs a web server that mocks all the API requests sent by the client. By the way, we are using Testcontainers for running Mongo (4). In the next step, we are creating the ConfigMap that injects Mongo connection settings into the Spring Boot app (5). Thanks to the Spring Cloud Kubernetes Config it is automatically loaded by the app and the app is able to connect the Mongo database on the dynamically generated port.
Spring Cloud Kubernetes comes with auto-configured Fabric8 KubernetesClient. We need to force it to connect to the mock API server. Therefore we should override kubernetes.master property used by the Fabric8 KubernetesClient into the master URL taken from the test “mocked” instance (6). Finally, we can just implement test methods in the standard way.
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
properties = {
"spring.main.cloud-platform=KUBERNETES",
"spring.cloud.bootstrap.enabled=true"}) // (1)
@EnableKubernetesMockClient(crud = true) // (2)
@Testcontainers
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class EmployeeKubernetesMockTest {
private static final Logger LOG = LoggerFactory
.getLogger(EmployeeKubernetesMockTest.class);
static KubernetesClient client; // (3)
@Container // (4)
static MongoDBContainer mongodb = new MongoDBContainer("mongo:5.0");
@BeforeAll
static void setup() {
ConfigMap cm = client.configMaps()
.create(buildConfigMap(mongodb.getMappedPort(27017)));
LOG.info("!!! {}", cm); // (5)
// (6)
System.setProperty(Config.KUBERNETES_MASTER_SYSTEM_PROPERTY,
client.getConfiguration().getMasterUrl());
System.setProperty(Config.KUBERNETES_TRUST_CERT_SYSTEM_PROPERTY, "true");
System.setProperty(Config.KUBERNETES_NAMESPACE_SYSTEM_PROPERTY, "default");
}
private static ConfigMap buildConfigMap(int port) {
return new ConfigMapBuilder().withNewMetadata()
.withName("employee").withNamespace("default")
.endMetadata()
.addToData("application.properties",
"""
spring.data.mongodb.host=localhost
spring.data.mongodb.port=%d
spring.data.mongodb.database=test
spring.data.mongodb.authentication-database=test
""".formatted(port))
.build();
}
@Autowired
TestRestTemplate restTemplate;
@Test
@Order(1)
void addEmployeeTest() {
Employee employee = new Employee("1", "1", "Test", 30, "test");
employee = restTemplate.postForObject("/", employee, Employee.class);
assertNotNull(employee);
assertNotNull(employee.getId());
}
@Test
@Order(2)
void addAndThenFindEmployeeByIdTest() {
Employee employee = new Employee("1", "2", "Test2", 20, "test2");
employee = restTemplate.postForObject("/", employee, Employee.class);
assertNotNull(employee);
assertNotNull(employee.getId());
employee = restTemplate
.getForObject("/{id}", Employee.class, employee.getId());
assertNotNull(employee);
assertNotNull(employee.getId());
}
@Test
@Order(3)
void findAllEmployeesTest() {
Employee[] employees =
restTemplate.getForObject("/", Employee[].class);
assertEquals(2, employees.length);
}
@Test
@Order(3)
void findEmployeesByDepartmentTest() {
Employee[] employees =
restTemplate.getForObject("/department/1", Employee[].class);
assertEquals(1, employees.length);
}
@Test
@Order(3)
void findEmployeesByOrganizationTest() {
Employee[] employees =
restTemplate.getForObject("/organization/1", Employee[].class);
assertEquals(2, employees.length);
}
}Now, after running the tests we can take a look at the logs. As you see, our test is loading properties from the employee ConfigMap.
Finally, it is able to successfully connect Mongo on the dynamic port and run all the tests against that instance.
Testing with Testcontainers on k3s
As I mentioned before, there are several tools we can use for testing with Kubernetes. This time we will see how to do it with Testcomntainers. We have already used it in the previous section for running the Mongo database. But there is also the Testcontainers module for Rancher’s k3s Kubernetes distribution. Currently, it is in the incubating state, but it doesn’t bother us to try it. In order to use it in the project we need to include the following Maven dependency:
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>k3s</artifactId>
<scope>test</scope>
</dependency>Here’s the implementation of the same tests as in the previous section, but this time with the k3s container. We don’t have to create any mocks. Instead, we will create the K3sContainer object (1). Before running the tests we need to create and initialize KubernetesClient. Testcontainers K3sContainer provides the getKubeConfigYaml() method for getting kubeconfig data. With the Fabric8 Config object we can initialize the client from that kubeconfig (2) (3). After that, we will create the ConfigMap with Mongo connection details (4). Finally, we have to override the master URL for Spring Cloud Kubernetes auto-configured Fabric8 client. In comparison to the previous section, we also need to set Kubernetes client certificates and keys (5).
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
properties = {
"spring.main.cloud-platform=KUBERNETES",
"spring.cloud.bootstrap.enabled=true"})
@Testcontainers
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class EmployeeKubernetesTest {
private static final Logger LOG = LoggerFactory
.getLogger(EmployeeKubernetesTest.class);
@Container
static MongoDBContainer mongodb = new MongoDBContainer("mongo:5.0");
@Container
static K3sContainer k3s = new K3sContainer(DockerImageName
.parse("rancher/k3s:v1.21.3-k3s1")); // (1)
@BeforeAll
static void setup() {
Config config = Config
.fromKubeconfig(k3s.getKubeConfigYaml()); // (2)
DefaultKubernetesClient client = new
DefaultKubernetesClient(config); // (3)
ConfigMap cm = client.configMaps().inNamespace("default")
.create(buildConfigMap(mongodb.getMappedPort(27017)));
LOG.info("!!! {}", cm); // (4)
System.setProperty(Config.KUBERNETES_MASTER_SYSTEM_PROPERTY,
client.getConfiguration().getMasterUrl());
// (5)
System.setProperty(Config.KUBERNETES_CLIENT_CERTIFICATE_DATA_SYSTEM_PROPERTY,
client.getConfiguration().getClientCertData());
System.setProperty(Config.KUBERNETES_CA_CERTIFICATE_DATA_SYSTEM_PROPERTY,
client.getConfiguration().getCaCertData());
System.setProperty(Config.KUBERNETES_CLIENT_KEY_DATA_SYSTEM_PROPERTY,
client.getConfiguration().getClientKeyData());
System.setProperty(Config.KUBERNETES_TRUST_CERT_SYSTEM_PROPERTY,
"true");
System.setProperty(Config.KUBERNETES_NAMESPACE_SYSTEM_PROPERTY,
"default");
}
private static ConfigMap buildConfigMap(int port) {
return new ConfigMapBuilder().withNewMetadata()
.withName("employee").withNamespace("default")
.endMetadata()
.addToData("application.properties",
"""
spring.data.mongodb.host=localhost
spring.data.mongodb.port=%d
spring.data.mongodb.database=test
spring.data.mongodb.authentication-database=test
""".formatted(port))
.build();
}
@Autowired
TestRestTemplate restTemplate;
@Test
@Order(1)
void addEmployeeTest() {
Employee employee = new Employee("1", "1", "Test", 30, "test");
employee = restTemplate.postForObject("/", employee, Employee.class);
assertNotNull(employee);
assertNotNull(employee.getId());
}
@Test
@Order(2)
void addAndThenFindEmployeeByIdTest() {
Employee employee = new Employee("1", "2", "Test2", 20, "test2");
employee = restTemplate
.postForObject("/", employee, Employee.class);
assertNotNull(employee);
assertNotNull(employee.getId());
employee = restTemplate
.getForObject("/{id}", Employee.class, employee.getId());
assertNotNull(employee);
assertNotNull(employee.getId());
}
@Test
@Order(3)
void findAllEmployeesTest() {
Employee[] employees =
restTemplate.getForObject("/", Employee[].class);
assertEquals(2, employees.length);
}
@Test
@Order(3)
void findEmployeesByDepartmentTest() {
Employee[] employees =
restTemplate.getForObject("/department/1", Employee[].class);
assertEquals(1, employees.length);
}
@Test
@Order(3)
void findEmployeesByOrganizationTest() {
Employee[] employees =
restTemplate.getForObject("/organization/1", Employee[].class);
assertEquals(2, employees.length);
}
}Run Spring Kubernetes Apps on Minikube
In this exercise, I’m using Minikube, but you can as well use any other distribution like Kind or k3s. Spring Cloud Kubernetes requires additional privileges on Kubernetes to be able to interact with the master API. So, before running the apps we will create the spring-cloud-kubernetes ServiceAccount with the required privileges. Our role needs to have access to the configmaps, pods, services, endpoints and secrets. If we do not enable discovery across all namespaces (the spring.cloud.kubernetes.discovery.all-namespaces property), it can be Role within the namespace. Otherwise, we should create a ClusterRole.
apiVersion: v1
kind: ServiceAccount
metadata:
name: spring-cloud-kubernetes
namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: spring-cloud-kubernetes
namespace: default
rules:
- apiGroups: [""]
resources: ["configmaps", "pods", "services", "endpoints", "secrets"]
verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: spring-cloud-kubernetes
namespace: default
subjects:
- kind: ServiceAccount
name: spring-cloud-kubernetes
namespace: default
roleRef:
kind: ClusterRole
name: spring-cloud-kubernetesOf course, you don’t have to apply the manifests visible above by yourself. As I mentioned at the beginning of the article, there is a skaffold.yaml file in the repository root directory file that contains the whole configuration. It runs manifests with Mongo Deployment (1) and with privileges (2) together with all the services.
apiVersion: skaffold/v4beta5
kind: Config
metadata:
name: sample-spring-microservices-kubernetes
build:
artifacts:
- image: piomin/admin
jib:
project: admin-service
- image: piomin/department
jib:
project: department-service
args:
- -DskipTests
- image: piomin/employee
jib:
project: employee-service
args:
- -DskipTests
- image: piomin/gateway
jib:
project: gateway-service
- image: piomin/organization
jib:
project: organization-service
args:
- -DskipTests
tagPolicy:
gitCommit: {}
manifests:
rawYaml:
- k8s/mongodb-*.yaml # (1)
- k8s/privileges.yaml # (2)
- admin-service/k8s/*.yaml
- department-service/k8s/*.yaml
- employee-service/k8s/*.yaml
- gateway-service/k8s/*.yaml
- organization-service/k8s/*.yamlAll we need to do it to deploy all the apps by executing the following skaffold command:
$ skaffold devOnce we will do it we can display a list of running s pods:
kubectl get pod
NAME READY STATUS RESTARTS AGE
admin-5f8c8498f-vtstx 1/1 Running 0 2m38s
department-746774879b-llrdn 1/1 Running 0 2m38s
employee-5bbf6b765f-7hsv7 1/1 Running 0 2m37s
gateway-578cb64558-m9n7f 1/1 Running 0 2m37s
mongodb-7f68b8b674-dbfnb 1/1 Running 0 2m38s
organization-5688c58656-bv8n6 1/1 Running 0 2m37sWe can also display a list of services. Some of them, like admin or gateway, are exposed as NodePort. Thanks to that we can easily access them outside of our Kubernetes cluster.
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
admin NodePort 10.101.220.141 <none> 8080:31368/TCP 3m53s
department ClusterIP 10.108.144.90 <none> 8080/TCP 3m52s
employee ClusterIP 10.99.75.2 <none> 8080/TCP 3m52s
gateway NodePort 10.96.7.237 <none> 8080:31518/TCP 3m52s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 38h
mongodb ClusterIP 10.108.198.233 <none> 27017/TCP 3m53s
organization ClusterIP 10.107.102.26 <none> 8080/TCP 3m52sLet’s obtain the Minikube IP address on our local machine:
$ minikube ipNow, we can use that IP address to access e.g. Spring Boot Admin Server on the target port. For me its 31368. Spring Boot Admin should successfully discover all three microservices and connect to the /actuator endpoints exposed by that apps.
We can go to the details of each Spring Boot app. As you depatment-service is running on my local Minikube.
Once you stop the skaffold dev command, all the apps and configured will be removed from your Kubernetes cluster.
Final Thoughts
If you are running only Spring Boot apps on your Kubernetes cluster, Spring Cloud Kubernetes is an interesting option. It allows us to easily integrate with Kubernetes discovery, config maps, and secrets. Thanks to that we can take advantage of other Spring Cloud components like load balancer, circuit breaker, etc. However, if you are running apps written in different languages and frameworks, and using language-agnostic tools like service mesh (Istio, Linkerd), Spring Cloud Kubernetes may not be the best choice.
20 COMMENTS