Piotr's TechBlog

Java, Spring, Kotlin, microservices, Kubernetes, containers

Observability on Kubernetes with OpenTelemetry, Camel and Quarkus

Kubernetes Performance Quarkus

Observability on Kubernetes with OpenTelemetry, Camel and Quarkus

By piotr.minkowski 0

Observability is becoming essential in modern applications, especially when they run in distributed environments like Kubernetes. The platform gives us a great place to build a shared observability stack across multiple applications, instead of configuring everything separately for each one.

In this article, we will build an observability stack using OpenTelemetry and Grafana Tempo, with all examples running on OpenShift. We will then connect a Quarkus application using Apache Camel and see how easily Camel can integrate with OpenTelemetry to provide useful distributed tracing information. Camel is especially interesting here because observability can be integrated into Camel routes in a very natural way. A Camel route represents the flow of a message through different steps, so being able to trace that flow gives us a pretty good picture of what our application is actually doing.

If you enjoy this kind of content, you may also want to check out some of my other articles covering similar topics, including Arconia for Spring Boot Dev Services and Observability and Quarkus REST with Apache Camel and Keycloak. And if you’re interested in running Java applications on Kubernetes, you can find a lot more hands-on examples in my book “Hands-On Java with Kubernetes”.

Source Code

Feel free to use my source code if you’d like to try the exercise yourself. To do that, you must clone my sample GitHub repository. Then you should only follow my instructions. You will find the sample application in the meteo-public directory. We’ll dive into the sample application later. I’ll explain its architecture and show how it connects to the observability stack.

Install Observability Stack in Kubernetes

You can build the observability stack on Kubernetes using various tools and configure it in different ways. With OpenShift, it’s a little different. The recommended installation tools are Kubernetes operators, and the recommended tracing stack is Grafana Tempo and OpenTelemetry. Quarkus can also automatically run the Grafana and OTEL stack locally in containers using the Dev Services solution. But I’ll show you that later in this article.

Install Operators on OpenShift

Therefore, let’s start with a list of required Kubernetes operators for the observability stack. You can see them in the screenshot below from my OpenShift instance. Red Hat provides pre-built operators for both OpenTelemetry and Tempo. Of course, installing the operator is just the first step. Next, we need to configure them by creating a few fairly simple Kubernetes CR objects. We then use the Cluster Observability Operator to visualize traces and metrics in the OpenShift web GUI.

kubernetes-observability-ocp-operators

Configure Azure Storage

I’ll be honest—setting up observability on OpenShift is a bit of a pain. But that’s mainly because the Tempo operator requires S3-compatible storage. There are various solutions you can use for this. You’ll find everything in the documentation here. Personally, I run OpenShift on Azure. Therefore, I use Azure storage. First, you should create the Azure storage account and blob container as shown below. My container’s name is tempo-ocp, and the storage account name is tempoocp.

kubernetes-observability-s3

In the next step, we need to run a few commands using the Azure CLI. So you’ll need to have the CLI installed on your computer. You should also sign in to your account using the az login command. Then, create an Azure managed identity by running the following command:

az identity create \
  --name tempo-ocp \
  --resource-group <your_resource_group> \
  --location centralus \
  --subscription <your_subscription_id>
ShellSession

Before running the next az command, obtain the URL of the OpenID Connect (OIDC) issuer for your cluster by running the following command:

$ oc get authentication cluster -o json | jq -r .spec.serviceAccountIssuer
https://centralus.oic.aro.azure.com/64dc69e4-d083-49fc-9569-ebece1dd1408/5bf45fd9-a89a-4b1c-862d-1a6844abfe54
ShellSession

Then, you can proceed with the creation of a federated identity credential for the distributed tracing service account. The name of my service account in OpenShift is tempo-basic. I will install all the Tempo components in the tempo namespace. All of these elements must be included in the subject field, as shown in the command below.

az identity federated-credential create \
  --name tempo-ocp \
  --identity-name tempo-ocp \
  --resource-group <your_resource_group> \
  --issuer https://centralus.oic.aro.azure.com/64dc69e4-d083-49fc-9569-ebece1dd1408/5bf45fd9-a89a-4b1c-862d-1a6844abfe54 \
  --subject system:serviceaccount:tempo:tempo-basic \
  --audiences api://AzureADTokenExchange
ShellSession

Let’s go ahead and create the correct namespace and service account on OpenShift right now.

$ oc create ns tempo
$ oc create sa tempo-basic -n tempo
ShellSession

You can retrieve the assignee ID using the CLI or by going to the managed identity tab in the portal.

Then, assign the Storage Blob Data Contributor role to the Azure service principal identity of the created Azure managed identity.

az role assignment create \
  --assignee e0796797-b94d-4937-be7f-14445ef91665 \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/<your_subscription_id>"
ShellSession

This is what it looks like on my portal.

Run Tempo and OTEL Collector in Kubernetes

As you can see, unfortunately, I had to run a few commands to configure Azure for observability. Finally, we can create a Kubernetes Secret with all parameters required to use the Azure account by the Tempo instance.

oc create -n tempo secret generic azure-secret \
  --from-literal=container=tempo-ocp \
  --from-literal=account_name=tempoocp \
  --from-literal=client_id=e0796797-b94d-4937-be7f-14445ef91665 \
  --from-literal=audience=api://AzureADTokenExchange \
  --from-literal=tenant_id=redhat0.onmicrosoft.com
ShellSession

Then, we can move on to creating instances of Tempo and the collector’s OTEL. As I mentioned earlier, the hardest part is already behind us. Now all we need to do is create two CR objects. This TempoStack custom resource defines a basic Grafana Tempo deployment in the tempo namespace. A few things worth clarifying. At the beginning, we need to provide the name of a previously created Secret that contains the credentials for Azure Storage. You must specify the tenants, but you can copy them using the same names and IDs I used. We can also enable the gateway, but for the purposes of this exercise, we’ll be using Tempo directly through a Kubernetes Service.

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: basic
  namespace: tempo
spec:
  resources:
    total:
      limits:
        cpu: 2000m
        memory: 2Gi
  tenants:
    authentication:
      - tenantId: 1610b0c3-c509-4592-a256-a1871353dbfa
        tenantName: dev
      - tenantId: 1610b0c3-c509-4592-a256-a1871353dbfb
        tenantName: prod
    mode: openshift
  managementState: Managed
  template:
    gateway:
      enabled: true
    queryFrontend:
      jaegerQuery:
        enabled: true
  storage:
    secret:
      name: azure-secret
      type: azure
  storageSize: 10Gi
YAML

Here’s a list of pods running in the tempo namespace after creating the Tempo object.

$ oc get po -n tempo
NAME                                        READY   STATUS    RESTARTS   AGE
tempo-basic-compactor-d5c965bb5-sqxrk       1/1     Running   0          1m
tempo-basic-distributor-84bd886697-ztq69    1/1     Running   0          1m
tempo-basic-gateway-5d4bf67676-5mxqh        2/2     Running   0          1m
tempo-basic-ingester-0                      1/1     Running   0          1m
tempo-basic-querier-85d77b7446-hv6bw        1/1     Running   0          1m
tempo-basic-query-frontend-8cddc7d4-cv6cw   3/3     Running   0          1m
ShellSession

There are also several Kubernetes Services. However, what interests us most is the tempo-basic-gateway Service, which the OTEL collector will use to send data.

$ oc get svc tempo-basic-gateway
NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
tempo-basic-gateway   ClusterIP   172.30.217.147   <none>        8090/TCP,8081/TCP,8080/TCP   1m
ShellSession

With OTEL, things are a bit more complicated—at least when it comes to the number of YAML lines we need to configure in our OpenTelemetryCollector object. The most important part is the exporters, where the collector will send the data it receives from the application. To connect to the Tempo instance we created earlier, we need to use bearer token-based authentication. For receivers, gRPC alone would suffice, since our example application uses this protocol by default. The processor, on the other hand, isn’t essential at this point. We’ll use it later for more advanced applications.

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: cluster-collector
  namespace: otel
spec:
  mode: deployment
  serviceAccount: otel-collector
  config:
    extensions:
      bearertokenauth:
        filename: "/var/run/secrets/kubernetes.io/serviceaccount/token"
    exporters:
      debug: {}
      otlp_grpc/traces:
        endpoint: 'tempo-basic-gateway.tempo.svc.cluster.local:8090'
        tls:
          insecure: false
          ca_file: '/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt'
        auth:
          authenticator: bearertokenauth
        headers:
          X-Scope-OrgID: "dev"
      otlp_http/traces:
        endpoint: 'https://tempo-basic-gateway.tempo.svc.cluster.local:8080/api/traces/v1/dev'
        tls:
          insecure: false
          ca_file: '/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt'
        auth:
          authenticator: bearertokenauth
        headers:
          X-Scope-OrgID: "dev"
    processors:
      batch:
        timeout: 5s
        send_batch_max_size: 10000
    receivers:
      otlp:
        protocols:
          grpc: {}
          http: {}
    service:
      extensions:
        - bearertokenauth
      pipelines:
        traces:
          exporters:
            - debug
            - otlp_grpc/traces
            - otlp_http/traces
          processors:
            - batch
          receivers:
            - otlp
      telemetry:
        metrics:
          readers:
            - pull:
                exporter:
                  prometheus:
                    host: 0.0.0.0
                    port: 8889
YAML

You should see a single running pod in the otel namespace.

$ oc get pod -n otel
NAME                                           READY   STATUS    RESTARTS   AGE
cluster-collector-collector-64798f4fc6-qzhq4   1/1     Running   0          2m
ShellSession

The cluster-collector-collector Service exposes gRPC (4317) and HTTP (4318) to receive traces from the apps running internally on the OpenShift cluster.

$ oc get svc -n otel
NAME                                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
cluster-collector-collector              ClusterIP   172.30.173.51   <none>        4317/TCP,4318/TCP   2m
cluster-collector-collector-headless     ClusterIP   None            <none>        4317/TCP,4318/TCP   2m
cluster-collector-collector-monitoring   ClusterIP   172.30.2.194    <none>        8889/TCP            2m
ShellSession

Enable UI Plugins Using Cluster Observability Operator

To view traces or create your own dashboards with metrics, enable two UI plugins. After installing them, additional tabs appear in the “Observe” section of the OpenShift console. This is where the Cluster Observability Operator comes in handy, as it provides the necessary mechanisms. You can create both objects in the OpenShift console or simply add them using the YAML below. Note: The object names must match exactly those in the manifest below.

apiVersion: observability.openshift.io/v1alpha1
kind: UIPlugin
metadata:
  name: monitoring
spec:
  monitoring:
    perses:
      enabled: true
  type: Monitoring
---
apiVersion: observability.openshift.io/v1alpha1
kind: UIPlugin
metadata:
  name: distributed-tracing
spec:
  type: DistributedTracing
YAML

This is what it looks like in my OpenShift console after I create the UIPlugin objects and log in.

But that’s not all. To create dashboards using Perses, you need to add a datasource in the namespace containing your application (or create a global object—but I’m deliberately leaving that out). Here, too, we’ll use the CRD provided by the Kubernetes observability operator.

apiVersion: perses.dev/v1alpha2
kind: PersesDatasource
metadata:
  labels:
    app.kubernetes.io/managed-by: observability-operator
    app.kubernetes.io/name: thanos-querier-datasource
    app.kubernetes.io/part-of: monitoring
  name: thanos-querier-datasource
  namespace: apps-dev
spec:
  client:
    tls:
      caCert:
        certPath: /ca/service-ca.crt
        type: file
      enable: true
  config:
    default: true
    display:
      name: Default Datasource
    plugin:
      kind: PrometheusDatasource
      spec:
        proxy:
          kind: HTTPProxy
          spec:
            secret: thanos-querier-datasource-secret
            url: 'https://thanos-querier.openshift-monitoring.svc.cluster.local:9091'
YAML

Observability Architecture in Kubernetes

Before we move on to our sample Quarkus application, let’s take a look at the solution’s architecture. The diagram below illustrates it. All components except for the Quarkus application are already running in the cluster. Our application must connect to the OTEL collector on port 4317 using the service cluster-collector-collector.otel.svc.cluster.local. The OTEL collector sends the received traces to the Tempo instance. In addition, Prometheus will automatically query the metrics provided by the application. You’ll see how easy it is to implement this using just Quarkus.

kubernetes-observability-arch

Now let’s move on to the application itself. Its task is to expose a REST endpoint that determines the conditions for running in a specific location. To do this, it must communicate with several publicly available APIs within the Open Meteo service. First, it must determine the geolocation of the city specified in the REST endpoint’s input. Then, using the geolocation obtained in this way, it queries the weather conditions and air quality. Based on the collected data, it rates the running conditions on a scale from A to E. Obviously, we want to capture an accurate trace for such an endpoint call, which in turn calls several different APIs behind the scenes.

kubernetes-observability-apis

Quarkus Camel Application with OTEL Support

The meteo-public app uses Apache Camel to aggregate data from three public Open-Meteo APIs and rate outdoor running conditions for a given city. It exposes a single POST /rating endpoint that accepts a city name and country code, resolves coordinates, and fetches the air quality and the current weather forecast. Finally, it returns a letter-grade score from A+ to E with a short description.

Dependencies

Let’s begin with dependencies. Apache Camel will allow us to structure our tracing neatly, as you’ll see later. That’s basically the main reason I wanted to use it in this article. We include quite a few Camel dependencies. The REST layer uses camel-quarkus-rest with camel-quarkus-platform-http as the underlying component. For JSON, camel-quarkus-jackson handles marshaling inside Camel routes and quarkus-rest-jackson covers the Quarkus REST binding. Outbound HTTP calls go through camel-quarkus-http. Route-to-route calls use camel-quarkus-direct.

Finally, the observability part. Tracing needs two extensions. The quarkus-opentelemetry extension wires up the OTLP exporter and W3C Trace Context propagation and auto-instruments inbound HTTP. The camel-quarkus-opentelemetry2 extension takes that further into the Camel pipeline, turning each direct: route into its own named span. Quarkus quarkus-micrometer-registry-prometheus extension drops a Prometheus scrape endpoint at /q/metrics. With the quarkus-openshift dependency, you can simply run the build where Quarkus handles OpenShift deployment automatically. It generates the Deployment, Service, and Route or even ServiceMonitor for metrics exposure at build time, and then applies them to the cluster.

    <dependencies>
        <!-- Quarkus REST server -->
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-rest-jackson</artifactId>
        </dependency>
        <!-- Camel HTTP component for calling external APIs -->
        <dependency>
            <groupId>org.apache.camel.quarkus</groupId>
            <artifactId>camel-quarkus-http</artifactId>
        </dependency>
        <!-- Camel REST DSL -->
        <dependency>
            <groupId>org.apache.camel.quarkus</groupId>
            <artifactId>camel-quarkus-rest</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel.quarkus</groupId>
            <artifactId>camel-quarkus-platform-http</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel.quarkus</groupId>
            <artifactId>camel-quarkus-jackson</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel.quarkus</groupId>
            <artifactId>camel-quarkus-direct</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel.quarkus</groupId>
            <artifactId>camel-quarkus-bean</artifactId>
        </dependency>
        <!-- OpenAPI / Swagger -->
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-smallrye-openapi</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel.quarkus</groupId>
            <artifactId>camel-quarkus-openapi-java</artifactId>
        </dependency>
        <!-- Observability -->
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-opentelemetry</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.camel.quarkus</groupId>
            <artifactId>camel-quarkus-opentelemetry2</artifactId>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-micrometer-registry-prometheus</artifactId>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-openshift</artifactId>
        </dependency>
        <!-- Tests -->
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-junit</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>io.rest-assured</groupId>
            <artifactId>rest-assured</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>
XML

Camel Routes Implementation

Every request goes through an Apache Camel route pipeline that fans out to three external APIs and scores the result through a CDI bean (MeteoService). The orchestrator route get-running-rating calls three direct: sub-routes in sequence — geocode, airQuality, and weather. Each sub-route hits one API and passes the result forward as an exchange property:

from("direct:getRating")
    .routeId("get-running-rating")
    .setProperty("request", simple("${body}"))
    .to("direct:geocode")
    .to("direct:airQuality")
    .to("direct:weather")
    .bean(meteoService, "buildResponse");
Java

Here’s the implementation of the three direct routes called from the root direct:getRating route. The direct:geocode resolves the city to coordinates. On the other hand, the direct:airQuality route picks up those coordinates and fetches the AQI (Air Quality Index). Finally, the direct:weather gets a weather forecast.

from("direct:geocode")
    .removeHeaders("*")
    .setHeader(Exchange.HTTP_METHOD, constant("GET"))
    .setHeader(Exchange.HTTP_QUERY, simple(
        "name=${exchangeProperty.request.city}"
        + "&country_code=${exchangeProperty.request.country}"
        + "&count=1&language=en&format=json"))
    .setBody(constant(""))
    .to("https://geocoding-api.open-meteo.com/v1/search?bridgeEndpoint=true")
    .unmarshal(new JacksonDataFormat(GeocodingResponse.class))
    .choice()
        .when(simple("${body.results} == null"))
            .throwException(new IllegalArgumentException("City not found"))
    .end()
    .setProperty("lat", simple("${body.results[0].latitude}"))
    .setProperty("lon", simple("${body.results[0].longitude}"))
    .setProperty("cityName", simple("${body.results[0].name}"));

from("direct:airQuality")
    .removeHeaders("*")
    .setHeader(Exchange.HTTP_METHOD, constant("GET"))
    .setHeader(Exchange.HTTP_QUERY, simple(
        "latitude=${exchangeProperty.lat}"
        + "&longitude=${exchangeProperty.lon}"
        + "&current=european_aqi,pm2_5,pm10"))
    .setBody(constant(""))
    .to("https://air-quality-api.open-meteo.com/v1/air-quality?bridgeEndpoint=true")
    .unmarshal(new JacksonDataFormat(AirQualityResponse.class))
    .setProperty("aqi", simple("${body.current.europeanAqi}"));

from("direct:weather")
    .removeHeaders("*")
    .setHeader(Exchange.HTTP_METHOD, constant("GET"))
    .setHeader(Exchange.HTTP_QUERY, simple(
        "latitude=${exchangeProperty.lat}"
        + "&longitude=${exchangeProperty.lon}"
        + "&current=temperature_2m,apparent_temperature,wind_speed_10m,precipitation,weather_code,cloud_cover"))
    .setBody(constant(""))
    .to("https://api.open-meteo.com/v1/forecast?bridgeEndpoint=true")
    .unmarshal(new JacksonDataFormat(WeatherResponse.class));
Java

We must provide OTEL collector endpoint URL in the application.properties and enable OpenTelemetry for Camel. There are a few additional settings, like this one quarkus.swagger-ui.always-include, to enable Swagger UI not only in dev mode.

quarkus.swagger-ui.always-include = true

quarkus.otel.exporter.otlp.endpoint = http://cluster-collector-collector.otel.svc.cluster.local:4317
quarkus.otel.enabled = true
quarkus.otel.traces.enabled = true
quarkus.otel.propagators = tracecontext,baggage
camel.opentelemetry2.enabled = true

quarkus.openshift.route.expose = true
Plaintext

Deploy and Test the App on OpenShift

In this section of the article, you’ll deploy the application using Quarkus Kubernetes and test how it interacts with the observability stack. To build and deploy the app on OpenShift, run the following Maven command:

mvn clean package -DskipTests -Dquarkus.kubernetes.deploy=true
ShellSession

Assuming you’re logged in to the OpenShift cluster, you should see this command output. As you can see, the application built and deployed successfully to OpenShift. For me, it’s accessible at http://meteo-public-apps-dev.apps.piomin.centralus.aroapp.io. Your address will be different. Quarkus automatically generated all the necessary YAML files, including a ServiceMonitor for Prometheus scraping.

Let’s call our service using an OpenShift route in the following way:

curl -X 'POST' \
  'http://meteo-public-apps-dev.apps.piomin.centralus.aroapp.io/rating' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "city": "Warsaw",
  "country": "Poland"
}'
ShellSession

Here’s the response:

After that, go to the OpenShift Console to verify the list of available traces. Next, select the Tempo instance and tenant. In my case, these are “tempo/basic” and “dev”. You should see the trace for your most recent call, as shown below.

kubernetes-observability-traces

Then, select the trace you’re interested in to verify its details. You’ll see a detailed call histogram broken down by Camel routes.

kubernetes-observability-histogram

You can select a specific span to, for example, view a single call to the target API.

kubernetes-observability-timeline

Observability on Kubernetes: Advanced Use Case

Inject Delay in Camel

Let’s try to implement a slightly more advanced scenario than before. While we’re at it, I’ll also show you a few interesting features that might come in handy later. Let’s start by introducing an artificial delay into an outgoing call to one of the target APIs. To avoid modifying the Camel route itself, we can use the interceptSendToEndpoint() method. It will intercept the request sent to another HTTP endpoint. Here’s the method that injects a random delay between 200 ms and 2s to the air quality API call.

        interceptSendToEndpoint("https://air-quality-api.open-meteo.com/*")
                .process(exchange -> {
                    Thread.sleep(new Random().nextInt(200, 2000));
                });
Java

Run and Test Grafana Stack Locally

We can verify the changes locally before deploying the app to OpenShift. Quarkus provides a dedicated extension that allows us to run the Grafana stack, including Tempo and the OTel collector, in Docker in dev mode.

        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-observability-devservices-lgtm</artifactId>
        </dependency>
XML

Then, we can run the application using the following command:

mvn quarkus:dev
ShellSession

After starting the app, you should see the same fragment in the logs:

Now, let’s send some test calls. Our application is available at http://localhost:8080.

curl -X 'POST' \
  'http://localhost:8080/rating' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "city": "Warsaw",
  "country": "Poland"
}'
ShellSession

All LGTM containers listen on dynamic ports. On the application side, this is handled automatically by Testcontainers. To find the Grafana link, open the Quarkus dev console (http://localhost:8080/q/dev-ui), as shown below.

Next, in Grafana, under the Drilldown -> Traces section, you’ll find the traces for your test calls. As you can clearly see here, the response time for the air quality API is significantly longer than for other APIs, due to random latency.

Configure OTEL Collector to Filter Incoming Traces

Until now, our Kubernetes observability stack has captured all traces sent by the application. This is, of course, not an optimal approach. The goal of this part of the exercise is to filter incoming traces so that we retain only those that may be related to a problematic situation. The best place to perform this operation is the OTEL collector. In its definition, we’ll add an appropriate processor that will filter incoming traces based on latency and errors.

To do this, we will use the “Tail Sampling” processor. It samples traces according to user-defined policies when all of the spans are completed. Tail-based sampling lets you filter traces of interest and reduce data ingestion and storage costs. Let’s take a look at the new definition of the OTEL collector. The most important parts of the facility are highlighted. The new processor forwards only packets with errors or with a latency greater than 2.5 seconds.

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: cluster-collector
  namespace: otel
spec:
  mode: deployment
  serviceAccount: otel-collector
  config:
    extensions:
      bearertokenauth:
        filename: "/var/run/secrets/kubernetes.io/serviceaccount/token"
    exporters:
      debug: {}
      otlp_grpc/traces:
        endpoint: 'tempo-basic-gateway.tempo.svc.cluster.local:8090'
        tls:
          insecure: false
          ca_file: '/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt'
        auth:
          authenticator: bearertokenauth
        headers:
          X-Scope-OrgID: "dev"
      otlp_http/traces:
        endpoint: 'https://tempo-basic-gateway.tempo.svc.cluster.local:8080/api/traces/v1/dev'
        tls:
          insecure: false
          ca_file: '/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt'
        auth:
          authenticator: bearertokenauth
        headers:
          X-Scope-OrgID: "dev"
    processors:
      tail_sampling:
        policies:
          - name: errors
            status_code:
              status_codes:
                - ERROR
            type: status_code
          - latency:
              threshold_ms: 2500
            name: slow
            type: latency
    receivers:
      otlp:
        protocols:
          grpc: {}
          http: {}
    service:
      extensions:
        - bearertokenauth
      pipelines:
        traces:
          exporters:
            - debug
            - otlp_grpc/traces
            - otlp_http/traces
          processors:
            - tail_sampling
          receivers:
            - otlp
YAML

After that, let’s redeploy our app and repeat test calls several times. Below are a few of my requests, along with the response times.

As you can see, Tempo stores only traces with latency greater than 2.5 seconds.

Create Dashbaord with Perses

It’s time for the final stage of our exercise. We’ll create dashboards to visualize several metrics related to our application using the Perses solution. It’s actually very simple. As you can see below, you can click through the entire process from start to finish. The chart below illustrates the average processing time for incoming requests.

Here’s my Perses dashboard.

Conclusion

In this article, we built a complete observability stack for a Quarkus application running on Kubernetes and OpenShift, using OpenTelemetry, Grafana Tempo, Prometheus, and Perses. We also integrated Apache Camel with OpenTelemetry to trace individual Camel routes and identify slow external API calls.

Our example also demonstrated more advanced observability features, such as tail-based sampling, which lets us keep only traces related to errors or slow requests. Finally, we used application metrics to create a dashboard showing request processing times.

The main takeaway is that Kubernetes provides a solid foundation for building a shared observability platform. With OpenTelemetry, Quarkus, and Apache Camel, we can add observability to applications in a standardized way while keeping enough flexibility to investigate real production problems.

Leave a Reply