In this article, you will learn how to install, configure and use Loki to collect logs from apps running on Kubernetes. Together with Loki, we will use the Promtail agent to ship the logs and the Grafana dashboard to display them in graphical form. We will also create a simple app written in Quarkus that prints the logs in JSON format. Of course, Loki will collect the logs from the whole cluster. If you are interested in other approaches for integrating your apps with Loki you can read my article. It shows how to send the Spring Boot app logs to Loki using Loki4j Logback appended. You can also find the article about Grafana Agent used to send logs from the Spring Boot app to Loki on Grafana Cloud here.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then you should just follow my instructions.

Install Loki Stack on Kubernetes

In the first step, we will install Loki Stack on Kubernetes. The most convenient way to do it is through the Helm chart. Fortunately, there is a single Helm chart that installs and configures all the tools required in our exercise: Loki, Promtail, and Grafana. Let’s add the following Helm repository:

$ helm repo add grafana https://grafana.github.io/helm-charts

Then, we can install the loki-stack chart. By default, it does not install Grafana. In order to enable Grafana we need to set the grafana.enabled parameter to true. Our Loki Stack is installed in the loki-stack namespace:

$ helm install loki grafana/loki-stack \
  -n loki-stack \
  --set grafana.enabled=true \
  --create-namespace

Here’s a list of running pods in the loki-stack namespace:

$ kubectl get po -n loki-stack
NAME                           READY   STATUS    RESTARTS   AGE
loki-0                         1/1     Running   0          78s
loki-grafana-bf598db67-czcds   2/2     Running   0          93s
loki-promtail-vt25p            1/1     Running   0          30s

Let’s enable port forwarding to access the Grafana dashboard on the local port:

$ kubectl port-forward svc/loki-grafana 3000:80 -n loki-stack

Helm chart automatically generates a password for the admin user. We can obtain it with the following command:

$ kubectl get secret -n loki-stack loki-grafana \
    -o jsonpath="{.data.admin-password}" | \
    base64 --decode ; echo

Once we login into the dashboard we will see the auto-configured Loki datasource. We can use it to get the latest logs from the Kubernetes cluster:

Create and Deploy Quarkus App on Kubernetes

In the next step, we will install our sample Quarkus app on Kubernetes. It connects to the Postgres database. Therefore, we will also install Postgres with the Bitnami Helm chart:

$ helm install person-db bitnami/postgresql -n sample-quarkus \
  --set auth.username=quarkus  \
  --set auth.database=quarkus  \
  --set fullnameOverride=person-db \
  --create-namespace

With Quarkus we can easily change the logs format to JSON. We just need to include the following Maven dependency:

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-logging-json</artifactId>
</dependency>

And also enable JSON logging in the application properties:

quarkus.log.console.json = true

Besides the static logging fields, we will include a single dynamic field. We will use the MDC mechanism for that (1) (2). That field indicates the id of the person for whom we make the GET or POST request. Here’s the code of the REST controller:

@Path("/persons")
public class PersonResource {

    private PersonRepository repository;
    private Logger logger;

    public PersonResource(PersonRepository repository, Logger logger) {
        this.repository = repository;
        this.logger = logger;
    }

    @POST
    @Transactional
    public Person add(Person person) {
        repository.persist(person);
        MDC.put("personId", person.id); // (1)
        logger.infof("IN -> add(%s)", person);
        return person;
    }

    @GET
    @APIResponseSchema(Person.class)
    public List<Person> findAll() {
        logger.info("IN -> findAll");
        return repository.findAll()
                .list();
    }

    @GET
    @Path("/{id}")
    public Person findById(@PathParam("id") Long id) {
        MDC.put("personId", id); // (2)
        logger.infof("IN -> findById(%d)", id);
        return repository.findById(id);
    }
}

Here’s the sample log for the GET endpoint. Now, our goal is to parse and index it properly in Loki with Promtail.

Now, we need to deploy our sample app on Kubernetes. Fortunately, with Quarkus we can build and deploy the app using the single Maven command. We just need to activate the following custom profile which includes quarkus-kubernetes dependency and enables deployment with the quarkus.kubernetes.deploy property. It also activates image build using the Jib Maven Plugin.

<profile>
  <id>kubernetes</id>
  <activation>
    <property>
      <name>kubernetes</name>
    </property>
  </activation>
  <dependencies>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-container-image-jib</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-kubernetes</artifactId>
    </dependency>
  </dependencies>
  <properties>
    <quarkus.kubernetes.deploy>true</quarkus.kubernetes.deploy>
  </properties>
</profile>

Let’s build and deploy the app:

$ mvn clean package -DskipTests -Pkubernetes

Here’s the list of running pods (database and app):

$ kubectl get po -n sample-quarkus
NAME                             READY   STATUS    RESTARTS   AGE
person-db-0                      1/1     Running   0          48s
person-service-9f67b6d57-gvbs6   1/1     Running   0          18s

Configure Promptail to Parse JSON Logs

Let’s take a look at the Promtail configuration. We can find it inside the loki-promtail Secret. As you see it uses only the cri component.

server:
  log_level: info
  http_listen_port: 3101


clients:
  - url: http://loki:3100/loki/api/v1/push

positions:
  filename: /run/promtail/positions.yaml

scrape_configs:
  - job_name: kubernetes-pods
    pipeline_stages:
      - cri: {}
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels:
          - __meta_kubernetes_pod_controller_name
        regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
        action: replace
        target_label: __tmp_controller_name
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_name
          - __meta_kubernetes_pod_label_app
          - __tmp_controller_name
          - __meta_kubernetes_pod_name
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: app
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_instance
          - __meta_kubernetes_pod_label_release
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: instance
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_component
          - __meta_kubernetes_pod_label_component
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: component
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_node_name
        target_label: node_name
      - action: replace
        source_labels:
          - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        replacement: $1
        separator: /
        source_labels:
          - namespace
          - app
        target_label: job
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_container_name
        target_label: container
      - action: replace
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
        target_label: __path__
      - action: replace
        regex: true/(.*)
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
          - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
          - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
          - __meta_kubernetes_pod_container_name
        target_label: __path__

The result for our app is quite inadequate. Loki stores the full Kubernetes pods’ log lines and doesn’t recognize our logging fields.

In order to change that behavior we will parse data using the json component. This action will be limited just to our sample application (1). We will label the log records with level, sequence, and the personId MDC field (2) after extracting them from the Kubernetes log line. The mdc field contains a list of objects, so we need to perform additional JSON parsing (3) to extract the personId field. As the output, Promtail should return the log message field (4). Here’s the required transformation in the configuration file:

- job_name: kubernetes-pods
  pipeline_stages:
    - cri: {}
    - match:
        selector: '{app="person-service"}' # (1)
        stages:
          - json:
              expressions:
                log:
          - json: # (2)
              expressions:
                sequence: sequence
                message: message
                level: level
                mdc:
              source: log
          - json: # (3)
              expressions:
                personId: personId
              source: mdc
          - labels:
              sequence:
              level:
              personId:
          - output: # (4)
              source: message

After setting a new value of the loki-promtail Secret we should restart the Promtail pod. Let’s also restart our app and perform some test calls of the REST API:

$ curl http://localhost:8080/persons/1

$ curl http://localhost:8080/persons/6

$ curl -X 'POST' http://localhost:8080/persons \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "John Wick",
  "age": 18,
  "gender": "MALE",
  "externalId": 100,
  "address": {
    "street": "Test Street",
    "city": "Warsaw",
    "flatNo": 18,
    "buildingNo": 100
  }
}'

Let’s see how it looks in Grafana:

As you see, the log record for the GET request is labeled with level, sequence and the personId MDC field. That’s what we exactly wanted to achieve!

Now, we are able to filter results using the fields from our JSON log line:

Distributed Installation of Loki Stack

In the previously described installation method, we run a single instance of Loki. In order to use a more cloud-native and scalable approach we should switch to the loki-distributed Helm chart. It decides a single Loki instance into several independent components. That division also separates read and write streams. Let’s install it in the loki-distributed namespace with the following command:

$ helm install loki grafana/loki-distributed \
  -n loki-distributed --create-namespace

When installing Promtail we should modify the default address of the write endpoint. We use the Loki gateway component for that. In our case the name of the gateway Service is loki-loki-distributed-gateway. That component listens on the 80 port.

config:
  clients:
  - url: http://loki-loki-distributed-gateway/loki/api/v1/push

Let’s install Promtail using the following command:

$ helm install promtail grafana/promtail -n loki-distributed \
  -f values.yml

Finally, we should install Grafana. The same as before we will use a dedicated Helm chart:

$ helm install grafana grafana/grafana -n loki-distributed

Here’s a list of running pods:

$ kubectl get pod -n loki-distributed
NAME                                                    READY   STATUS    RESTARTS   AGE
grafana-6cd56666b9-6hvqg                                1/1     Running   0          42m
loki-loki-distributed-distributor-59767b5445-n59bq      1/1     Running   0          48m
loki-loki-distributed-gateway-7867bc8ddb-kgdfk          1/1     Running   0          48m
loki-loki-distributed-ingester-0                        1/1     Running   0          48m
loki-loki-distributed-querier-0                         1/1     Running   0          48m
loki-loki-distributed-query-frontend-86c944647c-vl2bz   1/1     Running   0          48m
promtail-c6dxj                                          1/1     Running   0          37m

After logging in to Grafana, we should add the Loki data source (we could also do it during the installation with Helm values). This time we have to connect to the query-frontend component available under the address loki-loki-distributed-query-frontend:3100.

Final Thoughts

Loki Stack is an interesting alternative to Elastic Stack for collecting and aggregating logs on Kubernetes. Loki has been designed to be very cost-effective and easy to operate. Since it does not index the contents of the logs, the usage of such resources as disk space or RAM memory is lower than for Elasticsearch. In this article, I showed you how to install Loki Stack on Kubernetes and how to configure it to analyze app logs in practice.