If you need a solution for querying and visualizing time series and metrics probably your first choice will be Grafana. Grafana is a visualization dashboard and it can collect data from some different databases like MySQL, Elasticsearch, and InfluxDB. At present, it is becoming very popular to integrate with InfluxDB as a data source. This is a solution specifically designed for storing real-time metrics and events and is very fast and scalable for time-based data. Today, I’m going to show an example Spring Boot application of metrics visualization based on Grafana, InfluxDB, and alerts using Slack communicator.

Spring Boot Actuator exposes some endpoints useful for monitoring and interacting with the application. It also includes a metrics service with gauge and counters support. A gauge records a single value, counter records incremented or decremented value in all previous steps. The full list of basic metrics is available in Spring Boot documentation here and these are for example free memory, heap usage, datasource pool usage, or thread information. We can also define our own custom metrics. To allow exporting such values into InfluxDB we need to declare bean @ExportMetricWriter. Spring Boot has not build-in metrics exporter for InfluxDB, so we have add influxdb-java library into pom.xml dependencies and define connection properties.

@Bean
@ExportMetricWriter
GaugeWriter influxMetricsWriter() {
   InfluxDB influxDB = InfluxDBFactory.connect("http://192.168.99.100:8086", "root", "root");
   String dbName = "grafana";
   influxDB.setDatabase(dbName);
   influxDB.setRetentionPolicy("one_day");
   influxDB.enableBatch(10, 1000, TimeUnit.MILLISECONDS);

   return new GaugeWriter() {

      @Override
      public void set(Metric<?> value) {
         Point point = Point.measurement(value.getName()).time(value.getTimestamp().getTime(), TimeUnit.MILLISECONDS)
            .addField("value", value.getValue()).build();
         influxDB.write(point);
         logger.info("write(" + value.getName() + "): " + value.getValue());
      }
   };
}

The metrics should be read from Actuator endpoint, so we should declare MetricsEndpointMetricReader bean.

@Bean
public MetricsEndpointMetricReader metricsEndpointMetricReader(final MetricsEndpoint metricsEndpoint) {
   return new MetricsEndpointMetricReader(metricsEndpoint);
}

We can customize exporting process by declaring properties inside application.yml file. In the code fragment below there are two parameters: delay-millis which set metrics export interval to 5 seconds and includes, where we can define which metric should be exported.

spring:
  metrics:
    export:
      delay-millis: 5000
    includes: heap.used,heap.committed,mem,mem.free,threads,datasource.primary.active,datasource.primary.usage,gauge.response.persons,gauge.response.persons.id,gauge.response.persons.remove

To easily run Grafana and InfluxDB let’s use docker.

$ docker run -d --name grafana -p 3000:3000 grafana/grafana
$ docker run -d --name influxdb -p 8086:8086 influxdb

Grafana is available under default security credentials admin/admin. The first step is to create InfluxDB data source.

Now, we can create our new dashboard and add some graphs. Before it run Spring Boot sample application to export metrics some data into InfluxDB. Grafana has user friendly support for InfluxDB queries, where you can click the entire configuration and have a hint of syntax. Of course there is also a possibility of writing text queries, but not all of query language features are available.

Here’s the picture with my Grafana dashboard for metrics passed in includes property. On the second picture below you can see enlarged graph with average REST methods processing time.

We can always implement our custom service which generates metrics sent to InfluxDB. Spring Boot Actuator provides two classes for that purpose: CounterService and GaugeService. Below, there is example of GaugeService usage, where the random value between 0 and 100 is generated in 100ms intervals.

@Service
public class FirstService {

   private final GaugeService gaugeService;

   @Autowired
   public FirstService(GaugeService gaugeService) {
   this.gaugeService = gaugeService;
   }

   public void exampleMethod() {
      Random r = new Random();
      for (int i = 0; i < 1000000; i++) {
         this.gaugeService.submit("firstservice", r.nextDouble()*100);
         try {
            Thread.sleep(100);
         } catch (InterruptedException e) {
            e.printStackTrace();
         }
      }
   }

}

The sample bean FirstService is starting after application startup.

@Component
public class Start implements ApplicationListener<ContextRefreshedEvent> {

   @Autowired
   private FirstService service1;

   @Override
   public void onApplicationEvent(ContextRefreshedEvent contextRefreshedEvent) {
      service1.exampleMethod();
   }

}

Now, let’s configure alert notification using Grafana dashboard and Slack. This feature is available from 4.0 version. I’m going to define a threshold for statistics sent by FirstService bean. If you have already created graph for gauge.firstservice (you need to add this metric name into includes property inside application.yml) go to edit section and then to Alert tab. There you can define the alerting condition by selecting the aggregating function (for example avg, min, max), evaluation interval, and the threshold value. For my sample visible in the picture below I selected alerting when the maximum value is bigger than 95 and conditions should be evaluated in 5-minute intervals.

After creating an alert configuration we should define the notification channel. There are some interesting supported notification types like email, Hip Chat, webhook, or Slack. When configuring Slack notification we need to pass the recipient’s address or channel name and incoming webhook URL. Then, add new notification for your alert sent to Slack in Notifications section.

I created dedicated channel #grafana for Grafana notification on my Slack account and attached incoming webhook to this channel by searching it in Channel Settings -> Add app or integration.

Finally, run my sample application, and don’t forget to logout from Grafana Dashboard in case you would like to receive an alert on Slack.