Part 3: Creating Microservices: Circuit Breaker, Fallback and Load Balancing with Spring Cloud

Part 3: Creating Microservices: Circuit Breaker, Fallback and Load Balancing with Spring Cloud

Probably you read some articles about Hystrix and you know in what purpose it is used for. Today I would like to show you an example of exactly how to use it, which gives you the ability to combine with other tools from Netflix OSS stack like Feign and Ribbon. I assume that you have basic knowledge on topics such as microservices, load balancing, service discovery. If not I suggest you read some articles about it, for example, my short introduction to microservices architecture available here: Part 1: Creating microservice using Spring Cloud, Eureka and Zuul. The code sample used in that article is also used now. There is also sample source code available on GitHub. For the sample described now see hystrix branch, for basic sample master branch.

Let’s look at some scenarios for using fallback and circuit breaker. We have Customer Service which calls API method from Account Service. There two running instances of Account Service. The requests to Account Service instances are load balanced by Ribbon client 50/50.

micro-details-1

Scenario 1

Hystrix is disabled for Feign client (1), auto retries mechanism is disabled for Ribbon client on local instance (2) and other instances (3). Ribbon read timeout is shorter than request max process time (4). This scenario also occurs with the default Spring Cloud configuration without Hystrix. When you call customer test method you sometimes receive full response and sometimes 500 HTTP error code (50/50).


ribbon:
  eureka:
    enabled: true
    MaxAutoRetries: 0 #(2)
    MaxAutoRetriesNextServer: 0 #(3)
    ReadTimeout: 1000 #(4)

feign:
  hystrix:
    enabled: false #(1)
 

Scenario 2

Hystrix is still disabled for Feign client (1), auto retries mechanism is disabled for Ribbon client on the local instance (2) but enabled on other instances once (3). You always receive a full response. If your request is received by instance with the delayed response it is timed out after 1 second and then Ribbon calls another instance – in that case not delayed. You can always change MaxAutoRetries to positive value but gives us nothing in that sample.


ribbon:
  eureka:
    enabled: true
    MaxAutoRetries: 0 #(2)
    MaxAutoRetriesNextServer: 1 #(3)
    ReadTimeout: 1000 #(4)

feign:
  hystrix:
    enabled: false #(1)
 

Scenario 3

Here is not a very elegant solution to the problem. We set ReadTimeout on value bigger than delay inside API method (5000 ms).


ribbon:
  eureka:
    enabled: true
    MaxAutoRetries: 0
    MaxAutoRetriesNextServer: 0
    ReadTimeout: 10000

feign:
  hystrix:
    enabled: false
 

Generally configuration from Scenario 2 and 3 is right, you always get the full response. But in some cases, you will wait for more than 1 second (Scenario 2) or more than 5 seconds (Scenario 3), and delayed instance receives 50% requests from Ribbon client. But fortunately, there is Hystrix – circuit breaker.

Scenario 4

Let’s enable Hystrix just by removing feign property. There are no auto retries for Ribbon client (1) and its read timeout (2) is bigger than Hystrix’s timeout (3). 1000ms is also default value for Hystrix timeoutInMilliseconds property. Hystrix circuit breaker and fallback will work for delayed instance of account service. For some first requests, you receive a fallback response from Hystrix. Then delayed instance will be cut off from requests, most of them will be directed to the not delayed instance.


ribbon:
  eureka:
    enabled: true
    MaxAutoRetries: 0 #(1)
    MaxAutoRetriesNextServer: 0
    ReadTimeout: 2000 #(2)

hystrix:
  command:
    default:
      execution:
        isolation:
          thread:
            timeoutInMilliseconds: 1000 #(3)
 

Scenario 5

This scenario is a more advanced development of Scenario 4. Now Ribbon timeout (2) is lower than Hystrix timeout (3) and also auto retries mechanism is enabled (1) for local instance and for other instances (4). The result is same as for Scenario 2 and 3 – you receive full response, but Hystrix is enabled and it cuts off delayed instance from future requests.


ribbon:
  eureka:
    enabled: true
    MaxAutoRetries: 3 #(1)
    MaxAutoRetriesNextServer: 1 #(4)
    ReadTimeout: 1000 #(2)

hystrix:
  command:
    default:
      execution:
        isolation:
          thread:
            timeoutInMilliseconds: 10000 #(3)
 

I could imagine a few other scenarios. But the idea was just a show difference in circuit breaker and fallback when modifying configuration properties for Feign, Ribbon, and Hystrix in application.yml.

Hystrix

Let’s take a closer look at standard Hystrix circuit breaker and usage described in Scenario 4. To enable Hystrix in your Spring Boot application you have to add the following dependencies to pom.xml. Second step is to add annotation @EnableCircuitBreaker to main application class and also @EnableHystrixDashboard if you would like to have UI dashboard available.


<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-hystrix</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-hystrix-dashboard</artifactId>
</dependency>
 

Hystrix fallback is set on Feign client inside customer service.


@FeignClient(value = "account-service", fallback = AccountFallback.class)
public interface AccountClient {

   @RequestMapping(method = RequestMethod.GET, value = "/accounts/customer/{customerId}")
   List<Account> getAccounts(@PathVariable("customerId") Integer customerId);

}
 

Fallback implementation is really simple. In this case, I just return an empty list instead of the customer’s account list received from the account-service.


@Component
public class AccountFallback implements AccountClient {

   @Override
   public List<Account> getAccounts(Integer customerId) {
      List<Account> acc = new ArrayList<Account>();
      return acc;
   }

}
 

Now, we can perform some tests. Let’s start discovery service, two instances of account service on different ports (-DPORT VM argument during startup) and customer service. Endpoint for tests is /customers/{id}. There is also JUnit test class which sends multiple requests to this enpoint available in customer-service module pl.piomin.microservices.customer.ApiTest.


@RequestMapping("/customers/{id}")
public Customer findById(@PathVariable("id") Integer id) {
   logger.info(String.format("Customer.findById(%s)", id));
   Customer customer = customers.stream().filter(it -> it.getId().intValue()==id.intValue()).findFirst().get();
   List<Account> accounts =  accountClient.getAccounts(id);
   customer.setAccounts(accounts);
   return customer;
}
 

I enabled Hystrix Dashboard on account-service main class. If you would like to access it call from your web browser http://localhost:2222/hystrix address and then type Hystrix’s stream address from customer-service http://localhost:3333/hystrix.stream. When I run test that sends 1000 requests to customer service ;about 20 (2%) of them were forwarder to delayed instance of account service, remaining to not delayed instance. Hystrix dashboard during that test is visible below. For more advanced Hystrix configuration refer to its documentation available ;here.

hystrix-1

0 COMMENTS

comments user
Shaan

Hi Piotr,

I have done the set up as explained in the article above, but i am getting following error when i use the hystrix dashboard to monitor hystrix stream – http://localhost:3333/hystrix.stream
Any help would be greatly appreciated.

2018-01-04 00:51:50.178 ERROR [account-service,b6efcafa1087ff38,b6efcafa1087ff38,true] 12792 — [nio-2222-exec-2] ashboardConfiguration$ProxyStreamServlet : Error proxying request: http://localhost:3333/hystrix.stream?delay=0

java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[na:1.8.0_102]
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) ~[na:1.8.0_102]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_102]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_102]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_102]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[na:1.8.0_102]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_102]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_102]
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.3.jar:4.5.3]

    comments user
    Piotr Mińkowski

    Hi Shaan,
    Did you check out the application logs just after startup if hystrix.stream is available for the application? Did you try to download that file just by calling this URL http://localhost:3333/hystrix.stream from your web browser?

      comments user
      Shaan

      Hi Piotr

      When I try this url, i keep getting ping: in the browser, nothing else.

      In the logs i see –
      Mapped “{[/hystrix.stream/**]}” onto public org.springframework.web.servlet.ModelAndView org.springframework.cloud.netflix.endpoint.ServletWrappingEndpoint.handle(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.lang.Exception

        comments user
        Piotr Mińkowski

        Did you try to send some test requests to the service?

comments user
Shaan

Hi Piotr,

I have done the set up as explained in the article above, but i am getting following error when i use the hystrix dashboard to monitor hystrix stream – http://localhost:3333/hystrix.stream
Any help would be greatly appreciated.

2018-01-04 00:51:50.178 ERROR [account-service,b6efcafa1087ff38,b6efcafa1087ff38,true] 12792 — [nio-2222-exec-2] ashboardConfiguration$ProxyStreamServlet : Error proxying request: http://localhost:3333/hystrix.stream?delay=0

java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[na:1.8.0_102]
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) ~[na:1.8.0_102]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_102]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_102]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_102]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[na:1.8.0_102]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_102]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_102]
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835) ~[httpclient-4.5.3.jar:4.5.3]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.3.jar:4.5.3]

    comments user
    Piotr Mińkowski

    Hi Shaan,
    Did you check out the application logs just after startup if hystrix.stream is available for the application? Did you try to download that file just by calling this URL http://localhost:3333/hystrix.stream from your web browser?

      comments user
      Shaan

      Hi Piotr

      When I try this url, i keep getting ping: in the browser, nothing else.

      In the logs i see –
      Mapped “{[/hystrix.stream/**]}” onto public org.springframework.web.servlet.ModelAndView org.springframework.cloud.netflix.endpoint.ServletWrappingEndpoint.handle(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.lang.Exception

        comments user
        Piotr Mińkowski

        Did you try to send some test requests to the service?

comments user
Shaan

Sorry, for my late reply.. I was able to solve the issue and it worked for me.

One more question – when the circuit is open due to an issue in the following service call, how long it is kept open before the hystrix tries again to see if the service is working or not. Is there a setting to control it?

comments user
Shaan

Sorry, for my late reply.. I was able to solve the issue and it worked for me.

One more question – when the circuit is open due to an issue in the following service call, how long it is kept open before the hystrix tries again to see if the service is working or not. Is there a setting to control it?

Leave a Reply