HashiDays One conference. Three cities. Find a city near you
Vault
Vault

Install Vault to Red Hat OpenShift

  • 14min
  • |
  • Vault

Red Hat's OpenShift is a distribution of the Kubernetes platform that provides a number of usability and security enhancements

In this tutorial, you login to an OpenShift cluster, install Vault via the Helm chart and then configure the authentication between Vault and the cluster. Then you deploy two web applications. One that authenticates and requests secrets directly from the Vault server. The other that employs deployment annotations that enable it to remain Vault unaware.

Prerequisites

This tutorial was last tested 14 Jul 2020 on a macOS 10.15.5 using this configuration.

  1. Verify the RedHat OpenShift Local version.

    $ crc versionCRC version: 2.19.0+a71226OpenShift version: 4.12.13Podman version: 4.4.4

  2. Verify the Helm version.

    $ helm versionversion.BuildInfo{Version:"v3.11.2", GitCommit:"912ebc1cd10d38d340f048efaf0abda047c3468e", GitTreeState:"clean", GoVersion:"go1.20.2"}

    These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.

  3. Retrieve the web application and additional configuration by cloning the hashicorp/learn-vault-redhat-openshift repository from GitHub.

    $ git clone https://github.com/hashicorp-education/learn-vault-redhat-openshift.git

    This repository contains the supporting content for this tutorial.

  4. Go to the learn-vault-redhat-openshift directory.

    $ cd learn-vault-redhat-openshift

    Working directory

    This tutorial assumes that the remainder of commands are executed within this directory.

Configure and start the OpenShift cluster

RedHat OpenShift Local provisions and manages the lifecycle of OpenShift clusters running on your local system.

  1. Configure RedHat OpenShift Local.

    $ crc setup

  2. Start the OpenShift cluster and enter the pull secret.

    $ crc startINFO Checking if running as non-rootINFO Checking if crc-admin-helper executable is cachedINFO Checking if running on a supported CPU architecture...snip...Started the OpenShift cluster.The server is accessible via web console at:  https://console-openshift-console.apps-crc.testingLog in as administrator:  Username: kubeadmin  Password: A2c4e-dK6MJ-mTf37-4gn3TLog in as user:  Username: developer  Password: developerUse the 'oc' command line interface:  $ eval $(crc oc-env)  $ oc login -u developer https://api.crc.testing:6443

    Image Pull Secret

    This secret is generated and stored in your Red Hat account.

    The cluster starts and describes how to setup the environment and login as an administrator.

  3. Apply the oc-env into the current shell session.

    $ eval $(crc oc-env)

    The OpenShift CLI is accessed using the command oc. From here, you can administrate the entire OpenShift cluster and deploy new applications. The CLI exposes the underlying Kubernetes orchestration system with the enhancements made by OpenShift.

  4. Login to the OpenShift cluster with as the user admin with the command provided by the crc start command. Replace the user (-u), password (-p), and URL with the values from the crc start` command.

    Example:

    $ oc login -u kubeadmin -p fq66o-KsVBU-cnKBU-xLpqd https://api.crc.testing:6443##...Login successful.You have access to 57 projects, the list has been suppressed. You can list all projects with 'oc projects'Using project "default".

    The output displays that you are logged in as an admin within the default project.

Install the Vault Helm chart

The recommended way to run Vault on OpenShift is via the Helm chart. Helm is a package manager that installs and configures all the necessary components to run Vault in several different modes. To install Vault via the Helm chart in the next step requires that you are logged in as administrator within a project.

  1. Add the Hashicorp Helm repository.

    $ helm repo add hashicorp https://helm.releases.hashicorp.com

  2. Update all the repositories to ensure helm is aware of the latest versions.

    $ helm repo updateHang tight while we grab the latest from your chart repositories......Successfully got an update from the "hashicorp" chart repositoryUpdate Complete. ⎈Happy Helming!⎈

  3. Install the latest version of the Vault server running in development mode configured to work with OpenShift.

    $ helm install vault hashicorp/vault \    --set "global.openshift=true" \    --set "server.dev.enabled=true" \    --set "server.image.repository=docker.io/hashicorp/vault" \    --set "injector.image.repository=docker.io/hashicorp/vault-k8s"

    Example output:

    NAME: vaultLAST DEPLOYED: Fri May 19 09:46:57 2023NAMESPACE: defaultSTATUS: deployedREVISION: 1NOTES:Thank you for installing HashiCorp Vault!Now that you have deployed Vault, you should look over the docs on usingVault with Kubernetes available here:https://www.vaultproject.io/docs/Your release is named vault. To learn more about the release, try:  $ helm status vault  $ helm get manifest vault

    The Vault pod and Vault Agent Injector pod are deployed in the default namespace.

  4. Display all the pods within the default namespace.

    $ oc get podsNAME                                    READY   STATUS    RESTARTS   AGEvault-0                                 1/1     Running   0          45svault-agent-injector-777b86fbbd-cxrgb   1/1     Running   0          45s

    The vault-0 pod runs a Vault server in development mode. The vault-agent-injector pod performs the injection based on the annotations present or patched on a deployment.

    Wait until the vault-0 pod and vault-agent-injector pod are running and ready (1/1).

Configure Kubernetes authentication

Vault provides a Kubernetes authentication method that enables clients to authenticate with Vault within an OpenShift cluster. This authentication method configuration requires the location of the Kubernetes API, which is available in environment variables within the pod.

  1. Start an interactive shell session on the vault-0 pod.

    $ oc exec -it vault-0 -- /bin/sh

    Your system prompt is replaced with a new prompt / #. Commands issued at this prompt are executed on the vault-0 container.

  2. Enable the Kubernetes authentication method.

    $ vault auth enable kubernetesSuccess! Enabled kubernetes auth method at: kubernetes/

  3. Configure the Kubernetes authentication method to use the location of the Kubernetes host. It will automatically use the pod's own identity to authenticate with Kubernetes when querying the token review API.

    $ vault write auth/kubernetes/config \    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

    Example output:

    Success! Data written to: auth/kubernetes/config

    The authentication method is now configured.

  4. Exit the vault-0 pod.

    $ exit

Deployment: Request secrets directly from Vault

Applications on pods can directly communicate with Vault to authenticate and request secrets. An application needs a:

  • Service account
  • Vault secret
  • Vault policy to read the secret
  • Kubernetes authentication role

Create the service account

  1. Display the service account defined in service-account-webapp.yml.

    $ cat service-account-webapp.ymlapiVersion: v1kind: ServiceAccountmetadata:  name: webapp

    This definition of the service account creates the account with the name webapp.

  2. Apply the service account.

    $ oc apply --filename service-account-webapp.ymlserviceaccount/webapp created

  3. Get all the service accounts within the default namespace.

    $ oc get serviceaccountsNAME                   SECRETS   AGEbuilder                2         24ddefault                2         24ddeployer               2         24dvault                  2         2m54svault-agent-injector   2         2m54swebapp                 2         10s

    The webapp service account is displayed.

Create the secret

  1. Start an interactive shell session on the vault-0 pod.

    $ oc exec -it vault-0 -- /bin/sh

    Your system prompt is replaced with a new prompt / #. Commands issued at this prompt are executed on the vault-0 container.

  2. Create a secret at path secret/webapp/config with a username and password.

    $ vault kv put secret/webapp/config username="static-user" \    password="static-password"

    Example output:

    ====== Secret Path ======secret/data/webapp/config======= Metadata =======Key                Value---                -----created_time       2023-05-19T15:22:31.33887648Zcustom_metadata    <nil>deletion_time      n/adestroyed          falseversion            1

  3. Get the secret at path secret/webapp/config.

    $ vault kv get secret/webapp/config====== Secret Path ======secret/data/webapp/config======= Metadata =======Key                Value---                -----created_time       2023-05-19T15:22:31.33887648Zcustom_metadata    <nil>deletion_time      n/adestroyed          falseversion            1====== Data ======Key         Value---         -----password    static-passwordusername    static-user

    The secret with the username and password is displayed.

Define the read policy

  1. Write out the policy named webapp that enables the read capability for secrets at path secret/data/webapp/config.

    $ vault policy write webapp - <<EOFpath "secret/data/webapp/config" {  capabilities = ["read"]}EOF

    Example output:

    Success! Uploaded policy: webapp

    The policy webapp is used in the Kubernetes authentication role definition.

Create a Kubernetes authentication role

  1. Create a Kubernetes authentication role, named webapp, that connects the Kubernetes service account name and webapp policy.

    $ vault write auth/kubernetes/role/webapp \    bound_service_account_names=webapp \    bound_service_account_namespaces=default \    policies=webapp \    ttl=24h

    Example output:

    Success! Data written to: auth/kubernetes/role/webapp

    The role connects the Kubernetes service account, webapp, the namespace, default, with the Vault policy, webapp. The tokens returned are valid for 24 hours.

  2. Exit the vault-0 pod.

    $ exit

Deploy the application

  1. Display the webapp deployment defined in deployment-webapp.yml.

    $ cat deployment-webapp.yml

    deployment-webapp.yml

    ---apiVersion: apps/v1kind: Deploymentmetadata:  name: webapp  labels:    app: webappspec:  replicas: 1  selector:    matchLabels:      app: webapp  template:    metadata:      labels:        app: webapp    spec:      serviceAccountName: webapp      containers:        - name: app          image: hashieducation/simple-vault-client:latest          imagePullPolicy: Always          env:            - name: VAULT_ADDR              value: 'http://vault:8200'            - name: JWT_PATH              value: '/var/run/secrets/kubernetes.io/serviceaccount/token'            - name: SERVICE_PORT              value: '8080'

    The deployment deploys a pod with a web application running under the webapp service account that talks directly to the Vault service created by the Vault Helm chart http://vault:8200.

  2. Apply the webapp deployment.

    $ oc apply --filename deployment-webapp.ymldeployment.apps/webapp created

  3. Display all the pods within the default namespace.

    $ oc get podsNAME                                    READY   STATUS    RESTARTS   AGEvault-0                                 1/1     Running   0          7m16svault-agent-injector-777b86fbbd-cxrgb   1/1     Running   0          7m16swebapp-7b5c8d8ddd-x6lwz                 1/1     Running   0          2m55s

    Wait until the webapp pod is running and ready (1/1).

    This web application runs an HTTP service that listens on port 8080.

  4. Perform a curl request at http://localhost:8080 on the webapp pod.

    $ oc exec \    $(oc get pod --selector='app=webapp' --output='jsonpath={.items[0].metadata.name}') \    --container app -- curl -s http://localhost:8080 ; echo

    Example output:

    {"password"=>"static-password", "username"=>"static-user"}

    The web application running on port 8080 in the webapp pod:

    • authenticates with the Kubernetes service account token
    • receives a Vault token with the read capability at the secret/data/webapp/config path
    • retrieves the secrets from secret/data/webapp/config path
    • displays the secrets as JSON

Deployment: Secrets through annotations

Applications on pods can remain Vault unaware if they provide deployment annotations that the Vault Agent Injector detects. This injector service leverages the Kubernetes mutating admission webhook to intercept pods that define specific annotations and inject a Vault Agent container to manage these secrets. An application needs a:

  • service account
  • Vault secret
  • Vault policy to read the secret
  • Kubernetes authentication role
  • Deployment with Vault Agent Injector annotations

Create the service account

  1. Display the service account defined in service-account-issues.yml.

    $ cat service-account-issues.ymlapiVersion: v1kind: ServiceAccountmetadata:  name: issues

    This definition of the service account creates the account with the name issues.

  2. Apply the service account.

    $ oc apply --filename service-account-issues.ymlserviceaccount/issues created

  3. Get all the service accounts within the default namespace.

    $ oc get serviceaccountsNAME                   SECRETS   AGEbuilder                2         24ddefault                2         24ddeployer               2         24dissues                 2         8svault                  2         7m56svault-agent-injector   2         7m56swebapp                 2         5m12s

    The issues service account is displayed.

Create the secret

  1. Start an interactive shell session on the vault-0 pod.

    $ oc exec -it vault-0 -- /bin/sh

    Your system prompt is replaced with a new prompt / #. Commands issued at this prompt are executed on the vault-0 container.

  2. Create a secret at path secret/issues/config with a username and password.

    $ vault kv put secret/issues/config username="annotation-user" \    password="annotation-password"

    Example output:

    ====== Secret Path ======secret/data/issues/config======= Metadata =======Key                Value---                -----created_time       2023-05-19T15:32:11.619831856Zcustom_metadata    <nil>deletion_time      n/adestroyed          falseversion            1

  3. Get the secret at path secret/issues/config.

    $ vault kv get secret/issues/config====== Secret Path ======secret/data/issues/config======= Metadata =======Key                Value---                -----created_time       2023-05-19T15:32:11.619831856Zcustom_metadata    <nil>deletion_time      n/adestroyed          falseversion            1====== Data ======Key         Value---         -----password    annotation-passwordusername    annotation-user

    The secret with the username and password is displayed.

Define the read policy

  1. Write the policy named issues that enables the read capability for secrets at path secret/data/issues/config.

    $ vault policy write issues - <<EOFpath "secret/data/issues/config" {  capabilities = ["read"]}EOF

    Example output:

    Success! Uploaded policy: issues

    The policy issues is used in the Kubernetes authentication role definition.

Create a Kubernetes authentication role

  1. Create a Kubernetes authentication role, named issues, that connects the Kubernetes service account name and issues policy.

    $ vault write auth/kubernetes/role/issues \    bound_service_account_names=issues \    bound_service_account_namespaces=default \    policies=issues \    ttl=24h

    Example output:

    Success! Data written to: auth/kubernetes/role/issues

    The role connects the Kubernetes service account, issues, the namespace, default, with the Vault policy, issues. The tokens returned are valid for 24 hours.

  2. Exit the vault-0 pod.

    $ exit

Deploy the application

  1. Display the issues deployment defined in deployment-issues.yml.

    $ cat deployment-issues.yml

    deployment-issues.yaml

    apiVersion: apps/v1kind: Deploymentmetadata:  name: issues  labels:    app: issuesspec:  selector:    matchLabels:      app: issues  replicas: 1  template:    metadata:      annotations:        vault.hashicorp.com/agent-image: "docker.io/hashicorp/vault"        vault.hashicorp.com/agent-inject: 'true'        vault.hashicorp.com/role: 'issues'        vault.hashicorp.com/agent-inject-secret-issues-config.txt: 'secret/data/issues/config'        vault.hashicorp.com/agent-inject-template-issues-config.txt: |          {{- with secret "secret/data/issues/config" -}}          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard          {{- end -}}      labels:        app: issues    spec:      serviceAccountName: issues      containers:        - name: issues          image: jweissig/app:0.0.1

    The Vault Agent Injector service reads the metadata annotations prefixed with vault.hashicorp.com.

  2. Apply the issues deployment.

    $ oc apply --filename deployment-issues.ymldeployment.apps/issues created

  3. Display all the pods within the default namespace.

    $ oc get podsNAME                                    READY   STATUS    RESTARTS   AGEissues-75d794c744-x9gnf                 2/2     Running   0          17svault-0                                 1/1     Running   0          9m53svault-agent-injector-777b86fbbd-cxrgb   1/1     Running   0          9m53swebapp-7b5c8d8ddd-x6lwz                 1/1     Running   0          5m32s

    Wait until the issues pod is running and ready (2/2).

    This new pod now launches two containers. The application container, named issues, and the Vault Agent container, named vault-agent.

  4. Display the logs of the vault-agent container in the issues pod.

    $ oc logs \    $(oc get pod -l app=issues -o jsonpath="{.items[0].metadata.name}") \    --container vault-agent

    Example output:

    ==> Vault Agent started! Log data will stream in below:==> Vault Agent configuration:           Api Address 1: http://bufconn                     Cgo: disabled               Log Level: info                 Version: Vault v1.13.2, built 2023-04-25T13:02:50Z             Version Sha: b9b773f1628260423e6cc9745531fd903cae853f2023-05-19T16:40:07.041Z [INFO]  agent.sink.file: creating file sink2023-05-19T16:40:07.041Z [INFO]  agent.sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----2023-05-19T16:40:07.041Z [INFO]  agent.template.server: starting template server2023-05-19T16:40:07.041Z [INFO] (runner) creating new runner (dry: false, once: false)2023-05-19T16:40:07.041Z [INFO]  agent.auth.handler: starting auth handler2023-05-19T16:40:07.041Z [INFO]  agent.sink.server: starting sink server2023-05-19T16:40:07.041Z [INFO]  agent.auth.handler: authenticating2023-05-19T16:40:07.042Z [INFO] (runner) creating watcher2023-05-19T16:40:07.047Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks2023-05-19T16:40:07.047Z [INFO]  agent.auth.handler: starting renewal process2023-05-19T16:40:07.048Z [INFO]  agent.template.server: template server received new token2023-05-19T16:40:07.048Z [INFO] (runner) stopping2023-05-19T16:40:07.048Z [INFO] (runner) creating new runner (dry: false, once: false)2023-05-19T16:40:07.048Z [INFO] (runner) creating watcher2023-05-19T16:40:07.048Z [INFO] (runner) starting2023-05-19T16:40:07.048Z [INFO]  agent.sink.file: token written: path=/home/vault/.vault-token2023-05-19T16:40:07.052Z [INFO]  agent.auth.handler: renewed auth token

  5. Display the secret written to the issues container.

    $ oc exec \  $(oc get pod -l app=issues -o jsonpath="{.items[0].metadata.name}") \  --container issues -- cat /vault/secrets/issues-config.txt ; echo

    Example output:

    postgresql://annotation-user:annotation-password@postgres:5432/wizard

    The secrets are rendered in a PostgreSQL connection string is present on the container.

Next steps

You launched Vault within OpenShift with a Helm chart. Learn more about the Vault Helm chart by reading the documentation or exploring the project source code.

Then you deployed a web application that authenticated and requested a secret directly from Vault. And finally, deployed a web application that injected secrets based on deployment annotations supported by the Vault Agent Injector service. Learn more by reading the blog post announcing the Injecting Vault Secrets into Kubernetes Pods via a Sidecar, or the documentation for Vault Agent Injector service.

Previous

Vault installation to Amazon EKS via Helm

 Next

Vault installation to GKE via Helm