Microservices with Spring Boot, Spring Cloud Gateway and Consul Cluster

Microservices with Spring Boot, Spring Cloud Gateway and Consul Cluster

The Spring Cloud Consul project provides integration for Consul and Spring Boot applications through auto-configuration. By using the well-known Spring Framework annotation style, we may enable and configure common patterns within microservice-based environments. These patterns include service discovery using Consul agent, distributed configuration using Consul key/value store, distributed events with Spring Cloud Bus, and Consul Events. The project also supports a client-side load balancer based on Netflix’s Ribbon and an API gateway based on Spring Cloud Gateway.
In this article I will cover the following topics:

  • Integrating Spring Boot application with Consul discovery
  • Integrating Spring Cloud Gateway with Consul discovery
  • Using Consul KV for distributing configuration across Spring Boot applications
  • Running Consul in a clustered mode
  • Defining virtual zones for microservices and gateway

Microservices Architecture with Spring Cloud Consul cluster

Let’s proceed to the example system built with Spring Cloud Consul cluster support. It consists of four independent microservices. Some of them may call endpoints exposed by the others. The application source code is available on GitHub here: https://github.com/piomin/sample-spring-cloud-consul.git.
In the current example, we will try to develop a simple order system where customers may buy products. If a customer decides to confirm a selected list of products to buy, the POST request is sent to the order-service. It is processed by the Order prepare(@RequestBody Order order) method inside REST controller. This method is responsible for order preparation. First, it calculates the final price, considering the price of each product from the list, customer order history, and their category in the system by calling the proper API method from the customer-service. Then, it verifies if the customer’s account balance is enough to execute the order by calling the account-service, and finally, it returns the calculated price. If the customer confirms the action, the PUT /{id} method is called. The request is processed by the method Order accept(@PathVariable Long id) inside REST controller. It changes the order status and withdraws money from the customer’s account. The system architecture is broken down into the individual microservices hidden behind API gateway as shown here:

spring-cloud-consul-arch

The description created above should give you a big picture of our example system. However, business logic plays a supporting role, technically we have four Spring Boot applications using Consul discovery and KV store communicating with each other through REST APIs. The whole system is hidden for the external client behind the API gateway built on top of the Spring Cloud Gateway. Let’s proceed to the implementation.

1. Building Spring Cloud Consul Microservices

Let’s begin from dependencies. We use the currently newest stable version of Spring Boot – 2.2.0.RELEASE together with Spring Cloud Release Train Hoxton.RC1. The minimal set of required dependencies is to have Spring Web, Actuator (optionally), and Spring Cloud Consul (discovery + config).

<parent>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-parent</artifactId>
   <version>2.2.0.RELEASE</version>
</parent>

<dependencyManagement>
   <dependencies>
      <dependency>
         <groupId>org.springframework.cloud</groupId>
         <artifactId>spring-cloud-dependencies</artifactId>
         <version>Hoxton.RC1</version>
         <type>pom</type>
         <scope>import</scope>
      </dependency>
   </dependencies>
</dependencyManagement>

<dependencies>
   <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-consul-all</artifactId>
   </dependency>
   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
   </dependency>
   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
   </dependency>
</dependencies>

When running the application we will use dynamic listen port number generation feature by setting property server.port to 0. Because we will run more than instance of every service we also need to override default value of spring.cloud.consul.discovery.instance-id which is based on port number that is not applicable when it is set to 0. Here’s our application.yml file for account-service.

spring:  
  cloud:
    consul:
      discovery:
        instance-id: "${spring.cloud.client.hostname}:${spring.application.name}:${random.int[1,999999]}"

server:
  port: 0

The configuration is deployed on Consul, which means we are only having bootstrap.yml file on classpath. If you have both Spring Cloud Consul Discovery and Config dependencies distributed configuration is enabled by default. You only have to override the address of the Consul server if required.

spring:  
  application:
    name: account-service
  cloud:
    consul:
      host: 192.168.99.100
      port: 8500

In the current version of Spring Cloud we don’t have to enable anything, so just need to declare the main class:

@SpringBootApplication
public class AccountApplication {
   
   public static void main(String[] args) {
      SpringApplication.run(AccountApplication.class);
   }
}

2. Running Consul Cluster using Docker

In this section how to set up the local environment similar to the production mode. Therefore, we would like to have a scalable, production-grade service discovery infrastructure, consisting of some nodes working together inside the cluster. Consul provides support for clustering based on a gossip protocol used for communication between members and a Raft consensus protocol for a leadership election. I wouldn’t like to go into the details of that process, but some basics about Consul architecture should be clarified.
The first important element is the Consul agent. An agent is a long-running daemon on every member of the Consul cluster. It may be run in either client or server mode. All agents are responsible for running checks and keeping services registered, in different nodes and in sync, globally. Our main goal in this section is to set up and configure the Consul cluster using its Docker image. First, we will start the container, which acts as a leader of the cluster. There is only one difference in the currently used Docker command than for the standalone Consul server. We have set the environment variable CONSUL_BIND_INTERFACE=eth0 in order to change the network address of the cluster agent from 127.0.0.1 to the one available for the other member containers. My Consul server is now running at the internal address 172.17.0.2. To check out what your address is (it should be the same) you may run the command docker logs consul. The appropriate information is logged just after the container startup. Here’s the command that starts the first Consul node:


$ docker run -d --name consul-1 -p 8500:8500 -e CONSUL_BIND_INTERFACE=eth0 consul

Knowledge of that address is very important since now we have to pass it to every member container startup command as a cluster join parameter. We also bind it to all
interfaces by setting 0.0.0.0 as a client address. Now, we may easily expose the client agent API outside the container using the -p parameter:


$ docker run -d --name consul-2 -e CONSUL_BIND_INTERFACE=eth0 -p 8501:8500 consul agent -dev -join=172.17.0.2
$ docker run -d --name consul-3 -e CONSUL_BIND_INTERFACE=eth0 -p 8502:8500 consul agent -dev -join=172.17.0.2

After running two containers with Consul agent, you may check out the full list of cluster members by executing the following command on the leader’s container:

spring-cloud-consul-logs

We can always get the same information using the Consul Web Console.

spring-cloud-consul-ui

We may easily change the default Consul node address for the Spring Boot application by changing configuration properties. Spring Cloud Consul cluster support allows you to define only a single host address and port number of Consul agent. It is worth noting that in normal production mode with multiple machines you would install only a Consul agent on every machine, which is connected with a cluster of Consul servers.

spring:
  application:
    name: customer-service
  cloud:
    consul:
      host: 192.168.99.100
      port: 8501

3. Inter-service Communication

An inter-service communication is performed using OpenFeign declarative REST client. We can also include Spring Cloud Sleuth dependency for propagating correlationId between subsequent calls.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>

The OpenFeign client is auto-integrated with service discovery. To use it we need to declare an interface with required methods for communication. The interface has to be annotated with @FeignClient that points to the service using its discovery name.

@FeignClient(name = "account-service")
public interface AccountClient {

   @GetMapping("/customer/{customerId}")
   List<Account> findByCustomer(@PathVariable("customerId") Long customerId);
   
}

Finally, OpenFeign client needs to be enabled for the whole application.


@SpringBootApplication
@EnableFeignClients
public class CustomerApplication {
   
   public static void main(String[] args) {
      SpringApplication.run(CustomerApplication.class, args);
   }
   
}

4. Enable Zone Affinity Mechanism

When using Spring Cloud Discovery we may take advantage of zones affinity mechanism. If your microservices has been deployed to multiple zones, you may prefer that those services communicate with other services within the same zone before trying to access them in another zone. The same rule applies to API gateway that prefers communication with microservices within the same zone as gateway.
For testing purposes we run two instances of every microservice distributed across two zones: zone1 and zone2. The same with gateway-service. The current architecture of our system looks as shown below.
microservices-consul-2.png
The whole mechanism is enabled through the configuration. We need to set the default zone name for our microservice using property spring.cloud.consul.discovery.instanceZone. I defined two profiles for each application that may be set during startup with --spring.profiles.active command-line argument.

---
spring:
  profiles: zone1
  cloud:
    consul:
      discovery:
        instanceZone: zone1

---
spring:
  profiles: zone2
  cloud:
    consul:
      discovery:
        instanceZone: zone2

Spring Cloud provides a zone affinity mechanism based on Consul tags. If you set spring.cloud.consul.discovery.instanceZone property, Spring Cloud Consul tags a registered instance of service with zone metadata. The name of that tag may be overridden with spring.cloud.consul.discovery.defaultZoneMetadataName property. Assuming you have run two instances of each microservice divided into two zones using the command, for example java -jar --spring.profiles.active=zone1 target/order-service-1.1.jar, you should see the following list of registered services on your Consul instance.

microservices-consul-3

Here’s a more detailed view is the Nodes section that prints all tags and a listen port number for every instance of microservice.

microservices-consul-5

We can also display all running instances of a single service. In the following picture you can see instances of account-service.

microservices-consul-4

5. Building API Gateway with Spring Cloud

Since now, we have succesfully run all the microservices in two instances distributed across two different zones. Because they are all listening on dynamically generated ports we need an API gateway which is exposed on a static port to an external client. Here’s the list of dependencies used for building gateway-service:

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-consul-all</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

Because we would like to run a gateway on a static port the configuration of Maven profiles is slightly larger than for microservices. We also don’t need to register gateway in Consul discovery, because it is not accessed internally. We will run two instances of gateway, first available under port 8080 in zone1, and second available under port 9080 in zone2. Because it is not registered in Consul discovery we have to manually set a value for zone tag.

---
spring:
  profiles: zone1
  cloud:
    consul:
      discovery:
        instanceZone: zone1
        register: false
        registerHealthCheck: false
        tags: zone=zone1
server:  
  port: ${PORT:8080}

---
spring:
  profiles: zone2
  cloud:
    consul:
      discovery:
        instanceZone: zone2
        register: false
        registerHealthCheck: false
        tags: zone=zone2
server:  
  port: ${PORT:9080}

To enable integration with Consul discovery we need to set property spring.cloud.gateway.discovery.locator.enabled to true. In order to expose service under custom path we should define Path predicate and RewritePath filter for each service. In that case account-service is available under address http://localhost:8080/account/, customer-service under http://localhost:8080/customer/ etc.

spring:
  cloud:
    gateway:
      discovery:
        locator:
          enabled: true
      routes:
        - id: account-service
          uri: lb://account-service
          predicates:
            - Path=/account/**
          filters:
            - RewritePath=/account/(?<path>.*), /$\{path}
        - id: customer-service
          uri: lb://customer-service
          predicates:
            - Path=/customer/**
          filters:
            - RewritePath=/customer/(?<path>.*), /$\{path}
        - id: order-service
          uri: lb://order-service
          predicates:
            - Path=/order/**
          filters:
            - RewritePath=/order/(?<path>.*), /$\{path}
        - id: product-service
          uri: lb://product-service
          predicates:
            - Path=/product/**
          filters:
            - RewritePath=/product/(?<path>.*), /$\{path}

Now you can be sure that each request incoming to gateway-service started in zone1 would be forwarded to in the first place to microservice also started in zone1. And the same for zone2.

6. Distributed Configuration

Consul Config is automatically enabled for the application just after including dependency spring-cloud-starter-consul-config. Of course it is included together with spring-cloud-starter-consul-all also. Configuration is stored in the /config folder by default. We can create the configuration per all applications or just for a single application in a dedicated folder. Assuming we have four microservices and API gateway deployed in two zones we would have to define ten configuration folders. We have different options for storing application properties, but I chose YAML format. YAML must be set in the appropriate data key in consul. So the Consul folders structure for all our sample applications looks as shown below.


config/account-service,zone1/data
config/account-service,zone2/data
config/customer-service,zone1/data
config/customer-service,zone2/data
config/order-service,zone1/data
config/order-service,zone2/data
config/product-service,zone1/data
config/product-service,zone2/data
config/gateway-service,zone1/data
config/gateway-service,zone2/data

Here’s the typical configuration for one our sample microservice running in zone1 zone.

spring:  
  cloud:
    consul:
      discovery:
        instanceId: "${spring.cloud.client.hostname}:${spring.application.name}:${random.int[1,999999]}"
        instanceZone: zone1      
server.port: 0

And the same configuration created on Consul for account-service with active zone1 profile.

microservices-consul-1

In case we use Consul Config for our application the only file that should be available on classpath is bootstrap.yml. Except overriding Consul IP address or port if required we have to set the format of configuration properties to YAML. Here’s bootstrap.yml file for account-service.

spring:  
  application:
    name: account-service
  cloud:
    consul:
      host: 192.168.99.100
      port: 8500
        config:
          format: YAML

Summary

In this article I show you how to run microservices using Spring Consul Discovery and Config in the local environment similar to production. The applications use an affinity mechanism for inter-service communication and integrate with a cluster of Consul nodes. The configuration may be stored on the classpath or externalized to be stored on Consul with division into different active profiles. I think that Consul is a future of Spring Cloud microservices in the post-Netflix era, so it is definitely worth to know Spring Cloud Consul cluster support better. Before starting with Spring Cloud Consul cluster it worth knowing the basics: Quick Guide to Microservices with Spring Boot 2.0 and Spring Cloud.

19 COMMENTS

comments user
Liviu

Congratulations Piotr a nice article. I have read several of your articles in my opinion you are a pioneer in microservice architecture. I would have a question about launching microservices in docker using consul as a service dicovery. How do you map the container door to the port assigned by consul to the microservice server to make communication between microservices possible? I give an example if inside the docker file of a microservice I expose the port 8080 with the command EXPOSE 8080 and I set the property server.port = 0 how do you start the container starting from the image? They use the docker run command -P service1 the docker container is started by mapping to 8080 a port dynamically assigned by the docker but the integrated spring-boot web server is run on a port assigned by consul making sure that the service is not reached by the others since the other services call service1 on the port assigned by consul. Thanks in advance
Liviu

    comments user
    Piotr Mińkowski

    Thanks. About your question. There is no sense to set dynamic port for Spring Boot application, since each Docker container is running on diferent internal IP address. Therefore, you may run many instances of application listening on the same port.

comments user
felipe ramos da silva

Hello Piotr!

Thanks for the article, it helped me a lot! Also, I took your article and made a branch on my local to be able
to deploy that in a Kubernetes cluster, using Consul and Vault. I’m planing to create a article, may I use your article as a sample? Thanks a lot!

    comments user
    Piotr Mińkowski

    Hello,
    Thanks! Yes, you can use this article.

comments user
Phong Nguyen

Hello Piotr!

Thanks for your article. I have learned more from your articles
But I would have a question about this article regarding to consul cluster

In my yml,I config like this

spring:
cloud:
consul:
host: 10.17.70.20
port: 8500

but now I want make my program to know the cluster of consul.
I have 10.17.70.20 (master), 10.17.70.21, 10.17.70.22
how can I config yml for consul cluster

comments user
Phong Nguyen

Hi Piotr,
This description is consul agent. It’s way to consuls agent connect together to guarantee HA. My question is in Spring cloud side. How to configure in bootstrap.yml?

    comments user
    Piotr Mińkowski

    Hi, Well generally there is such option. But first, we were starting Consul on Docker in dev mode, what means that agent is run only in server mode. You can run it in client mode also. More details here: https://hub.docker.com/_/consul in section Running Consul Agent in Client Mode

      comments user
      Chowdhury S Masood

      Piotr,

      Thanks for the wonderful and informative blog. I have a follow-up question. I want to connect my spring boot services to a consul cluster. I know that I need to use consul client. Do I start the consul agent as client on the box that hosts the spring boot service or the box that has the consul server?

comments user
felipemeriga

Hello! I really like your article, also I took it and created a repo based on that but I added more kubernetes features. May I use this article as a base to a course that I’m currently doing?

Kind regards!

    comments user
    Piotr Mińkowski

    Hello. Yes, it will be a part of my video course in slightly modified form 🙂

comments user
Ash

Hi Piotr,
Really nice article! Glad that I found my way here.
Are you able to clarify if running multiple instances of Consul requires an Enterprise licence, or can be done so with the open-source version?

Many Thanks

comments user
amitojhanspal

Hi Piotr,
Nice article and tech blog! Glad to find my way here.
Are you able to tell me if I can create a Consul cluster in production using the open-source version?
Or, do I need the enterprise version?

Kind Regards

    comments user
    Piotr Mińkowski

    Hi. You don’t need enterprise version.

comments user
Christian

Thank you so much Piotr, it’s what i was looking for.

comments user
Roman

Hello Piotr!
I have a question regarding multizoned microservices.
Say we have Zone A and Zone B, like you described in your cool article.
But say there is one extra microservice in Zone B, that is unique, and non-transferrable to zone A.
And I, as a user of Zone A want to call this particular service in Zone B. What is the best way to do it?

Cheers, Roman

    comments user
    Piotr Mińkowski

    Hello Roman!
    You mean that we have a service that may be run only in a single instance?
    First, the zone affinity mechanism is just a option, so if a service would not be found in the same zone, it should be searched in another zones. You are also setting zones per service, what means that you can set no zone for a particular service. I didn’t test that solution by myself, but in theory it should work property – situation where service, which is in zone1 is communicating with service which is not assigned to any zone.

comments user
amir

Excelent blog,
I have a question about this regex: /customer/(?.*), /$\{path}

What does it mean ?

    comments user
    piotr.minkowski

    It is RewritePath rule, so for request come to the gateway as /account/accounts/1, I call /accounts/1 on the downstrea service only.

Leave a Reply