This article will teach you how to use the Secrets Store CSI Driver to integrate your app with HashiCorp Vault on Kubernetes. The main goal of that project is to integrate the secrets store with Kubernetes via a Container Storage Interface (CSI) volume. It allows mounting multiple secrets or keys retrieved from secure external providers like AWS Secrets Manager, Google Secret Manager, or HashiCorp Vault. In order to test the solution, we will create a simple Spring Boot app that reads the content of a file on a mounted volume. We will also use Terraform with the Helm provider to install and configure both Secrets Store CSI Driver and HashiCorp Vault. Finally, we are going to consider a secret rotation scenario.

The solution presented in this article is not the only way how we can deal with HashiCorp Vault on Kubernetes. If you are interested in other approaches you may refer to some of my previous articles. In that article, you can find a guide on how to integrate Vault secrets with Argo CD through the plugin. If you are running Spring Boot apps on Kubernetes you can also be interested in Spring Cloud Vault support. In that case please refer to the following article.

How it works

I guess you may not be very familiar with the Container Storage Interface (CSI) pattern. At the high-level CSI is a standard for exposing block or file storage to the containers. It is implemented by different storage providers.

The Secrets Store CSI Driver is running on Kubernetes as a DeamonSet. It is interacting with every instance of Kubelet on the Kubernetes nodes. Once the pod is starting, the Secrets Store CSI Driver communicates with the external secrets provider to retrieve the secret content. The following diagram illustrates how Secrets Store CSI Driver works on Kubernetes.

It provides the SecretProviderClass CRD to manage that process. In this provider class, we need to set the secure vault address and the location of the secret keys. Here’s the SecretProviderClass for our scenario. We will use HashiCorp Vault running on Kubernetes.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database
  namespace: default
spec:
  parameters:
    objects: |-
      - objectName: "db-password"
        secretPath: "secret/data/db-pass"
        secretKey: "password"
    roleName: webapp
    vaultAddress: 'http://vault.vault.svc:8200'
  provider: vault

Here’s the location of our secret in the HashiCorp Vault. As you see the current value of the password entry is test1.

Source Code

As usual – if you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my two GitHub repositories. The first of them contains Terraform scripts for installing Vault and Secrets Store CSI. After cloning it you go to the vault-ocp directory. The second repository contains a simple Spring Boot app for a test scenario. Once you clone it, go to the spring-util-app directory. Then you should just follow my instructions 🙂

Install Vault and Secrets Store CSI Driver with Terraform

As I mentioned before, we will use Terraform to set up almost the whole test scenario today. We will just leverage Skaffold, in the last step, to deploy the Spring Boot app on the Kubernetes cluster.

In order to install both Vault and Secrets Store CSI Driver we will use Helm charts. To do that part of the exercise, we need kubernetes (1) and helm (2) as the Terraform providers. The third step (3) is required only if you run the scenario on OpenShift. It changes the default service account access restrictions and security context constraints (SCCs) to ensure that a pod has sufficient permissions to start on OpenShift. Then we may proceed to the Helm charts installation. For the Secrets Store CSI Driver chart (4), it is important to enable secrets rotation since we will test this feature at the end of the article. Finally, we can install the HashiCorp Vault chart (5).

# (1)
provider "kubernetes" {
  config_path = "~/.kube/config"
  config_context = var.cluster-context
}

# (2)
provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
    config_context = var.cluster-context
  }
}

# (3)
resource "kubernetes_cluster_role_binding" "privileged" {
  metadata {
    name = "system:openshift:scc:privileged"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "system:openshift:scc:privileged"
  }
  subject {
    kind      = "ServiceAccount"
    name      = "secrets-store-csi-driver"
    namespace = "k8s-secrets-store-csi"
  }
  subject {
    kind      = "ServiceAccount"
    name      = "vault-csi-provider"
    namespace = "vault"
  }
}

resource "kubernetes_namespace" "vault" {
  metadata {
    name = "vault"
  }
}

resource "kubernetes_service_account" "vault-sa" {
  depends_on = [kubernetes_namespace.vault]
  metadata {
    name      = "vault"
    namespace = "vault"
  }
}

resource "kubernetes_secret_v1" "vault-secret" {
  depends_on = [kubernetes_namespace.vault]
  metadata {
    name = "vault-token"
    namespace = "vault"
    annotations = {
      "kubernetes.io/service-account.name" = "vault"
    }
  }

  type = "kubernetes.io/service-account-token"
}

# (4)
resource "helm_release" "secrets-store-csi-driver" {
  chart            = "secrets-store-csi-driver"
  name             = "csi-secrets-store"
  namespace        = "k8s-secrets-store-csi"
  create_namespace = true
  repository       = "https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts"

  set {
    name  = "linux.providersDir"
    value = "/var/run/secrets-store-csi-providers"
  }

  set {
    name  = "syncSecret.enabled"
    value = "true"
  }

  set {
    name  = "enableSecretRotation"
    value = "true"
  }
}

# (5)
resource "helm_release" "vault" {
  chart            = "vault"
  name             = "vault"
  namespace        = "vault"
  create_namespace = true
  repository       = "https://helm.releases.hashicorp.com"

  values = [
    file("values.yaml")
  ]
}

The Helm values.yaml file used by the Vault chart is visible below. In order to simplify deployment, we will enable the development mode (1). It generates a root token automatically and runs a single instance of Vault. We can enable Route for OpenShift (2) and use the image supported by Red Hat (3). Of course, we also need to enable global OpenShift configuration (4). You can omit all three steps (2) (3) (4) when running the scenario on vanilla Kubernetes. Finally, we need to enable CSI support (5) and disable the Vault sidecar injector which is not needed in that exercise (7). The path in the csi.deamonSet.providersDir property should be the same as the linux.providersDir in the Halm chart params (6).

server:
  dev:
    enabled: true # (1)
  route: # (2)
    enabled: true
    host: ""
    tls: null
  image: # (3)
    repository: "registry.connect.redhat.com/hashicorp/vault"
    tag: "1.12.4-ubi"
  serviceAccount:
    name: vault
    create: false
global: # (4)
  openshift: true
csi: 
  debug: true
  enabled: true # (5)
  daemonSet:
    providersDir: /var/run/secrets-store-csi-providers # (6)
    securityContext:
      container:
        privileged: true
injector: # (7)
  enabled: false

Finally, let’s apply the configuration to the target cluster.

$ terraform apply -auto-approve -compact-warnings

Here’s the result of the terraform apply command for my cluster:

In order to verify if everything has been installed successfully we can display the details of the vault-csi-provider DaemonSet in the vault namespace.

$ kubectl describe ds vault-csi-provider -n vault

Then, we can do a very similar thing for the Secrets Store CSI driver. We need to display the details of the csi-secrets-store-secrets-store-csi-driver DeamonSet.

$ kubectl describe ds csi-secrets-store-secrets-store-csi-driver \
  -n k8s-secrets-store-csi

Configure Vault with Terraform

The big advantage of using Terraform in our scenario is its integration with Vault. There is a dedicated Terraform provider for interacting with HashiCorp Vault. In order to set up the provider, we need to pass the Vault token (root in dev mode) and address. We will still need the kubernetes provider in that part of the exercise.

provider "kubernetes" {
  config_path = "~/.kube/config"
  config_context = var.cluster-context
}

provider "vault" {
  token = "root"
  address = var.vault-addr
}

We need to set the Kubernetes context path name and Vault API address in the variables.tf file. Here’s my Terraform variables.tf file:

variable "cluster-context" {
  type    = string
  default = "default/api-cluster-6sccr-6sccr-sandbox1544-opentlc-com:6443/opentlc-mgr"
}

variable "vault-addr" {
  type = string
  default = "http://vault-vault.apps.cluster-6sccr.6sccr.sandbox1544.opentlc.com"
}

The Terraform script responsible for configuring Vault is visible below. There are several things we need to do before deploying a sample Spring Boot app. Here’s a list of the required steps:

1. We need to enable Kubernetes authentication in Vault. The Secrets Store CSI Driver will use it to authenticate against the instance of Vault
2. In the second step, we will create a test secret. Its name is password and the value is test1. It is stored in Vault under the /secret/data/db-pass path.
3. Then, we have to Configure Kubernetes authenticate method.
4. In the fourth step, we are creating the policy for our app. It has read access to the secret created in step 2
5. We are creating the ServicerAccount for our sample Spring Boot app in the default namespace. The name of the ServiceAccount object is webapp-sa.
6. Finally, we can proceed to the last step in the Vault configuration – the authentication role required to access the secret. The name of the role is webapp and is then used by the Secrets Store CSI SecretProviderClass CR. The authentication role refers to the already created policy and ServiceAccount webapp-sa in the default namespace.
7. Once the Vault backend is configured properly we create the Secrets Store CSI SecretProviderClass CR.

# (1)
resource "vault_auth_backend" "kubernetes" {
  type = "kubernetes"
}

# (2)
resource "vault_kv_secret_v2" "secret" {
  mount = "secret"
  name = "db-pass"
  data_json = jsonencode(
    {
      password = "test1"
    }
  )
}

data "kubernetes_secret" "vault-token" {
  metadata {
    name      = "vault-token"
    namespace = "vault"
  }
}

# (3)
resource "vault_kubernetes_auth_backend_config" "example" {
  backend                = vault_auth_backend.kubernetes.path
  kubernetes_host        = "https://172.30.0.1:443"
  kubernetes_ca_cert     = data.kubernetes_secret.vault-token.data["ca.crt"]
  token_reviewer_jwt     = data.kubernetes_secret.vault-token.data.token
}

# (4)
resource "vault_policy" "internal-app" {
  name = "internal-app"

  policy = <<EOT
path "secret/data/db-pass" {
  capabilities = ["read"]
}
EOT
}

# (5)
resource "kubernetes_service_account" "webapp-sa" {
  metadata {
    name      = "webapp-sa"
    namespace = "default"
  }
}

# (6)
resource "vault_kubernetes_auth_backend_role" "internal-role" {
  backend                          = vault_auth_backend.kubernetes.path
  role_name                        = "webapp"
  bound_service_account_names      = ["webapp-sa"]
  bound_service_account_namespaces = ["default"]
  token_ttl                        = 3600
  token_policies                   = ["internal-app"]
}

# (7)
resource "kubernetes_manifest" "vault-database" {
  manifest = {
    "apiVersion" = "secrets-store.csi.x-k8s.io/v1alpha1"
    "kind"       = "SecretProviderClass"
    "metadata" = {
      "name"      = "vault-database"
      "namespace" = "default"
    }
    "spec" = {
      "provider"   = "vault"
      "parameters" = {
        "vaultAddress" = "http://vault.vault.svc:8200"
        "roleName"     = "webapp"
        "objects"      = "- objectName: \"db-password\"\n  secretPath: \"secret/data/db-pass\"\n  secretKey: \"password\""
      }
    }
  }
}

Once again, to apply the configuration we need to execute the terraform apply command.

Of course, we could apply the whole configuration visible above using Vault CLI or UI. However, we can verify it with Vault UI. In order to log in there we must use the root token. After login, we need to go to the Access tab and then to the Auth Methods menu. As you see, there is a webapp method defined in Terraform scripts.

Let’s switch to the Policies tab. Then we can check out if the internal-app policy exists.

Run the App with Mounted Secrets

Once we applied the whole configuration with Terraform we may proceed to the sample Spring Boot app. The idea is pretty simple. Our app just reads the data from the file and exposes it as the REST endpoint. Here’s the REST @Controller implementation:

@RestController
@RequestMapping("/api")
class SampleUtilController {

    @GetMapping("/db-password")
    fun resourceString(): String {
        val file = File("/mnt/secrets-store/db-password")
        return if(file.exists()) file.readText()
        else "none"
    }
}

Here’s the app Deployment manifest. As you see we are using the secrets-store.csi.k8s.io implementation of the CSI driver for mounted volume. It refers the vault-database SecretProviderClass object created with the Terraform script. The volume is containing the file with the value of our secret. We are mounting it under the path /mnt/secrets-store, which is accessed by the Spring Boot application.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-util-app
spec:
  selector:
    matchLabels:
      app: sample-util-app
  template:
    metadata:
      labels:
        app: sample-util-app
    spec:
      serviceAccountName: webapp-sa
      containers:
        - name: sample-util-app
          image: piomin/sample-util-app
          ports:
            - containerPort: 8080
          volumeMounts:
            - mountPath: /mnt/secrets-store
              name: secrets-store
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "vault-database"

Here’s the Kubernetes Service manifest:

apiVersion: v1
kind: Service
metadata:
  name: sample-util-app
spec:
  type: ClusterIP
  selector:
    app: sample-util-app
  ports:
    - port: 8080
      targetPort: 8080

We can easily build and deploy the app with Skaffold. It also allows exposing a port outside the cluster as a local port with the port-forward option.

$ skaffold dev --port-forward

Finally, we can our test endpoint GET /api/db-password. It returns the value we have already set in Vault for the db-pass/password secret.

$ curl http://localhost:8080/api/db-password
test2

Now, let’s test the secret rotation feature. In order to do that we need to change the value of the db-pass/password secret. We can do it using Vault UI. We can set the test2 value:

Secrets Store CSI Driver periodically queries managed secrets to detect changes. So, after the change, we would probably wait a moment until our app refreshes that value. The default poll interval is 2 minutes. We can override it using the rotation-poll-interval parameter (e.g. on the Helm chart). However, the most important thing is, that everything happens without restarting the pod. The only trace of change is in the events:

Now, let’s query for the latest value of the key used by the app. As you see the value has been refreshed.

Final Thoughts

If you are looking for a solution that injects Vault secrets into your app without creating Kubernetes Secret, Secrets Store CSI Driver is the solution for you. It is able to refresh the value of the secret in your app without restarting the container. In this article, I’m presenting how to install and configure it with Terraform to simplify the installation and configuration process.