Vault Agent - secrets as environment variables

Consul Template and Envconsul tools have been widely used by the Vault practitioners to help integrate Vault in their existing solutions. Vault 1.3.0 introduced the Vault Agent Template feature which provides the workflow that Consul Template provides.

Vault 1.14 introduced the process supervisor mode to retrieve secrets from Vault as environment variables using Consul Template markup.

Prerequisites

To complete this tutorial, you will need:

• Vault binary version 1.14.0 or later

Lab setup

Setup test secrets

1. In your terminal, clone the learn-vault-agent-envconsul repository which contains the example configuration used in this tutorial.

   $ git clone https://github.com/hashicorp-education/learn-vault-agent-envconsul.git

2. You can explore this repository by changing directories.

   $ cd learn-vault-agent-envconsul

   The folder includes the following files:

   .├── README.md├── kv-demo.sh├── pki-agent-config.hcl├── pki-demo.sh└── setup-secrets.sh0 directories, 5 files

3. Run the setup-secrets.sh script. This script enables kv-v2 secrets engine at web-team/ and dev-app/ paths and creates initial test data. Also, it configures PKI secrets engine to generate certificate which you will leverage in later section.

   $ ./setup-secrets.sh

4. Read the secrets created at web-team/data/api-keys.

   $ vault kv get -mount=web-team api-keys===== Secret Path =====web-team/data/api-keys======= Metadata =======Key                Value---                -----created_time       2023-06-13T21:05:24.965199Zcustom_metadata    <nil>deletion_time      n/adestroyed          falseversion            1==== Data ====Key     Value---     -----key1    4984h33sdkey2    o48r3eqf

5. Read the secrets created at dev-app/data/creds.

   $ vault kv get -mount=dev-app creds=== Secret Path ===dev-app/data/creds======= Metadata =======Key                Value---                -----created_time       2023-06-13T21:05:25.042033Zcustom_metadata    <nil>deletion_time      n/adestroyed          falseversion            1====== Data ======Key         Value---         -----password    my-long-passworduser        tester1

6. Read the secrets created at dev-app/data/creds/database/db-admin.

   $ vault kv get -mount=dev-app creds/database/db-admin============ Secret Path ============dev-app/data/creds/database/db-admin======= Metadata =======Key                Value---                -----created_time       2023-06-13T21:05:25.10932Zcustom_metadata    <nil>deletion_time      n/adestroyed          falseversion            1====== Data ======Key         Value---         -----password    o3ir23irusername    admin

Generate a Vault Agent config file

Vault Agent introduced the generate-config subcommand to auto generate the agent configuration file based on the user input.

Usage:

$ vault agent generate-config [options] <path_to_config>

Run the generate-config subcommand

Generates a Vault Agent configuration file named agent-config.hcl at the current working directory. If your application needs to pull secrets from multiple paths, you can provide multiple -path options as well as the wildcards (*).

1. Generate a Vault Agent configuration file named agent-config.hcl. Pass the kv-demo.sh as the command to execute.

   $ vault agent generate-config -type="env-template" \    -exec="./kv-demo.sh" \    -path="web-team/api-keys" \    -path="dev-app/*" \    agent-config.hcl

   Output:

   Successfully generated "agent-config.hcl" configuration file!Warning: the generated file uses 'token_file' authentication method, which isnot suitable for production environments.

   In the absence of the output file name, the command will generate a configuration file named agent.hcl.

2. Open the generated configuration file with your preferred text editor to examine.

   VaultHCP Vault Dedicated

   agent-config.hcl

   auto_auth {   method {      type = "token_file"      config {         token_file_path = "/Users/username/.vault-token"      }   }}template_config {   static_secret_render_interval = "5m"   exit_on_retry_failure         = true}vault {   address = "http://127.0.0.1:8200"}env_template "API_KEYS_KEY1" {   contents             = "{{ with secret \"web-team/data/api-keys\" }}{{ .Data.data.key1 }}{{ end }}"   error_on_missing_key = true}env_template "API_KEYS_KEY2" {   contents             = "{{ with secret \"web-team/data/api-keys\" }}{{ .Data.data.key2 }}{{ end }}"   error_on_missing_key = true}env_template "CREDS_PASSWORD" {   contents             = "{{ with secret \"dev-app/data/creds\" }}{{ .Data.data.password }}{{ end }}"   error_on_missing_key = true}env_template "CREDS_USER" {   contents             = "{{ with secret \"dev-app/data/creds\" }}{{ .Data.data.user }}{{ end }}"   error_on_missing_key = true}env_template "DB_ADMIN_PASSWORD" {   contents             = "{{ with secret \"dev-app/data/creds/database/db-admin\" }}{{ .Data.data.password }}{{ end }}"   error_on_missing_key = true}env_template "DB_ADMIN_USERNAME" {   contents             = "{{ with secret \"dev-app/data/creds/database/db-admin\" }}{{ .Data.data.username }}{{ end }}"   error_on_missing_key = true}exec {   command                   = ["./kv-demo.sh"]   restart_on_secret_changes = "always"   restart_stop_signal       = "SIGTERM"}

   agent-config.hcl

   auto_auth {   method {      type = "token_file"      config {         token_file_path = "/Users/username/.vault-token"      }   }}template_config {   static_secret_render_interval = "5m"   exit_on_retry_failure         = true}vault {   address = "https://vault-cluster-public-vault-XXXXXXXXX.XXXXXXXXX.XX.hashicorp.cloud:8200"}env_template "API_KEYS_KEY1" {   contents             = "{{ with secret \"web-team/data/api-keys\" }}{{ .Data.data.key1 }}{{ end }}"   error_on_missing_key = true}env_template "API_KEYS_KEY2" {   contents             = "{{ with secret \"web-team/data/api-keys\" }}{{ .Data.data.key2 }}{{ end }}"   error_on_missing_key = true}env_template "CREDS_PASSWORD" {   contents             = "{{ with secret \"dev-app/data/creds\" }}{{ .Data.data.password }}{{ end }}"   error_on_missing_key = true}env_template "CREDS_USER" {   contents             = "{{ with secret \"dev-app/data/creds\" }}{{ .Data.data.user }}{{ end }}"   error_on_missing_key = true}env_template "DB_ADMIN_PASSWORD" {   contents             = "{{ with secret \"dev-app/data/creds/database/db-admin\" }}{{ .Data.data.password }}{{ end }}"   error_on_missing_key = true}env_template "DB_ADMIN_USERNAME" {   contents             = "{{ with secret \"dev-app/data/creds/database/db-admin\" }}{{ .Data.data.username }}{{ end }}"   error_on_missing_key = true}exec {   command                   = ["./kv-demo.sh"]   restart_on_secret_changes = "always"   restart_stop_signal       = "SIGTERM"}

   The auto_auth stanza is using token_file as a placeholder. You can change the stanza to use desired auth method. See the following tutorials for other auto auth examples:

   • Vault Agent with AWS demonstrates AWS auth method
   • Vault Agent with Kubernetes demonstrates Kubernetes auth method

Process supervisor mode

Each generated env_template block maps to a secret key of a KV path that you passed in the command.

The exec block sets the kv-demo.sh script as the commands that Vault Agent to execute.

Open and examine the kv-demo.sh file. It reads secrets as environment variables.

#!/bin/shechoecho "==== KV DEMO APP ===="echo "API_KEYS_KEY1     = ${API_KEYS_KEY1}"echo "API_KEYS_KEY2     = ${API_KEYS_KEY2}"echo "CREDS_USER        = ${CREDS_USER}"echo "CREDS_PASSWORD    = ${CREDS_PASSWORD}"echo "DB_ADMIN_USERNAME = ${DB_ADMIN_USERNAME}"echo "DB_ADMIN_PASSWORD = ${DB_ADMIN_PASSWORD}"

This application does not need to know about Vault.

Start a Vault Agent

Start a Vault Agent instance that connects to the Vault server running at VAULT_ADDR.

1. Start a Vault Agent.

   $ vault agent -config=agent-config.hcl -log-level=error

   Example output:

   ==> Vault Agent started! Log data will stream in below:==> Vault Agent configuration:         Api Address 1: http://bufconn                     Cgo: disabled               Log Level: error               Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z            Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a==== KV DEMO APP ====API_KEYS_KEY1     = 4984h33sdAPI_KEYS_KEY2     = o48r3eqfCREDS_USER        = tester1CREDS_PASSWORD    = my-long-passwordDB_ADMIN_USERNAME = adminDB_ADMIN_PASSWORD = o3ir23ir

Restarts on secrets change

The secret values could change while your application is running. Examine the behavior when the secret values change.

1. Add sleep 100 in the kv-demo.sh to mock a long running application.

   kv-demo.sh

   #!/bin/shechoecho "==== DEMO APP ===="echo "API_KEYS_KEY1     = ${API_KEYS_KEY1}"echo "API_KEYS_KEY2     = ${API_KEYS_KEY2}"echo "CREDS_USER        = ${CREDS_USER}"echo "CREDS_PASSWORD    = ${CREDS_PASSWORD}"echo "DB_ADMIN_USERNAME = ${DB_ADMIN_USERNAME}"echo "DB_ADMIN_PASSWORD = ${DB_ADMIN_PASSWORD}"sleep 100

2. Edit the agent-config.sh file to set the static_secret_render_interval to 10 seconds for the purpose of demonstration. This makes Vault Agent to pull the secrets every 10 seconds.

   agent-config.hcl

   auto_auth {   method {      type = "token_file"      config {         token_file_path = "/Users/username/.vault-token"      }   }}template_config {   static_secret_render_interval = "10s"   exit_on_retry_failure         = true}vault {   address = "$VAULT_ADDR"}...snip...

3. Start the Vault Agent again.

   $ vault agent -config=agent-config.hcl -log-level=error==> Vault Agent started! Log data will stream in below:==> Vault Agent configuration:         Api Address 1: http://bufconn                     Cgo: disabled               Log Level: error               Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z            Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a==== KV DEMO APP ====API_KEYS_KEY1     = 4984h33sdAPI_KEYS_KEY2     = o48r3eqfCREDS_USER        = tester1CREDS_PASSWORD    = my-long-passwordDB_ADMIN_USERNAME     = adminDB_ADMIN_PASSWORD = o3ir23ir

4. Open another terminal, and connect to the target Vault server.

   Tip

   Refer back to the Lab setup section.

   VaultHCP Vault Dedicated

   Set the VAULT_ADDR environment variable.

   $ export VAULT_ADDR=http://127.0.0.1:8200

   Authenticate with Vault.

   $ vault login root

   Set the VAULT_ADDR environment variable

   $ export VAULT_ADDR=<Public_Cluster_URL>

   Set the VAULT_TOKEN environment variable.

   $ export VAULT_NAMESPACE=admin

   Authenticate with Vault Dedicated using the admin token.

   $ vault login <admin_token>

5. Update the secret values at web-team/data/api-keys.

   $ vault kv put web-team/api-keys \   key1="new-key-value-1" \   key2="new-key-value-2"

   Return to the terminal where Vault Agent is running and watch the output.

   ==> Vault Agent started! Log data will stream in below:==> Vault Agent configuration:         Api Address 1: http://bufconn                     Cgo: disabled               Log Level: error               Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z            Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a==== KV DEMO APP ====API_KEYS_KEY1     = 4984h33sdAPI_KEYS_KEY2     = o48r3eqfCREDS_USER        = tester1CREDS_PASSWORD    = my-long-passwordDB_ADMIN_USERNAME = adminDB_ADMIN_PASSWORD = o3ir23ir==== KV DEMO APP ====API_KEYS_KEY1     = new-key-value-1API_KEYS_KEY2     = new-key-value-2CREDS_USER        = tester1CREDS_PASSWORD    = my-long-passwordDB_ADMIN_USERNAME = adminDB_ADMIN_PASSWORD = o3ir23ir

6. Press Ctrl + C to stop the running Vault Agent.

Work with dynamic secrets

Currently, the generate-config subcommand only supports kv-v1 and kv-v2 secrets engines; however, the Vault Agent's process supervisor mode supports dynamic secrets that Envconsul supports today. The difference is that you would have to manually define the env_template blocks.

1. The setup-secrets.sh script configured the PKI secrets engine and it is ready to serve.

   $ vault secrets list

   Example output:

   Path          Type         Accessor              Description----          ----         --------              -----------cubbyhole/    cubbyhole    cubbyhole_595f8b0f    per-token private secret storagedev-app/      kv           kv_f16c3aac           n/aidentity/     identity     identity_cd1c4e99     identity storepki/          pki          pki_ca9e7398          n/apki_int/      pki          pki_331d3c63          n/asecret/       kv           kv_1bb31236           key/value secret storagesys/          system       system_a6bb36e2       system endpoints used for control, policy and debuggingweb-team/     kv           kv_c5dd184d           n/a

2. (Optional) Request a certificate.

   $ vault write -field=certificate pki_int/issue/example-dot-com \   common_name="test.example.com" ttl="24h"

   Example output:

   -----BEGIN CERTIFICATE-----MIIDZjCCAk6gAwIBAgIUZl818nTOMzvFdlhcnE5YYZ/WC5UwDQYJKoZIhvcNAQELBQAwLTErMCkGA1UEAxMiZXhhbXBsZS5jb20gSW50ZXJtZWRpYXRlIEF1dGhvcml0eTAeFw0yMzA2MTMyMzE3MjdaFw0yMzA2MTQyMzE3NTdaMBsxGTAXBgNVBAMTEHRlc3QuZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVIrsKREu2+lQbNbdYgYzMYAidevxVV3K4kWe9MYQWW4xLus/DtuBZXwqPkPsl3UGsNRjv4gE9hvb3lGAcs5Ze4f27sK/FvXgcOXy/p1wJ8JQV+rMHq/cg3dcEvYvY/f2cIMeR0vMJb58POjM4iWSgvYfvQtO+4r0hPydGGafubhorRvlvEMfiNfghDG3wSoeJZ0OmwQenBY6ASsJZ32qJbuiVszSCHIgOP5s6GfF6KKkKj3ojr/aH++ff+ckDzFbW/uz0IGV2hhdGEnnSjlWn0uaPR6xFGqprp1csAAZtdY50q6PYZ7Ev8Oe3zFPNM4sjuXETsXcjRUUusC1vGxYrAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgOoMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUcu+At+HNbaGOD8CVvoKRxf9Q4JMwHwYDVR0jBBgwFoAUOiYGxwIjDoTz389uuIGb9VWmEKUwGwYDVR0RBBQwEoIQdGVzdC5leGFtcGxlLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEAQbHfB/gcpG9n2Pb9wfw9wek+Ko1abmf2rgkqD44z6lvS+kpRi1SIwY2oUEhXvILdl098M4D7zRhWfCYyBuN+jVuGpZjqVlc3nS976JzBil3MzlfvUSbscoao0oh6unaBfYlGlgR+STNjjeQDx+bVWL3os8u3WqY+XlyVVHnBAQME+9Hgiamzv0gKE2vEb93KZ33rozjfwSTNzEfvQOHTAjkMZ2rwAu2prhLq4e8jUM8BA4QxFiw9R+vx1Dnu3Kz3W0eLhQ2WXG9YH57A234vOc/dcW6teNq8QzYLuqzA4oNKpBWiWT7uLt2hP0L3OtUliqDZAzFZpxt9m7E2V9qQcQ==-----END CERTIFICATE-----

3. Open the pki-agent-config.hcl to examine the env_template block it defines.

   pki-agent-config.hcl

   env_template "CERT" {   contents = "{{ with pkiCert \"pki_int/issue/example-dot-com\" \"common_name=test.example.com\" \"ttl=24h\"}}{{ .Data.Cert }}{{ end }}"   error_on_missing_key = true}exec {   command                   = ["./pki-demo.sh"]   restart_on_secret_changes = "always"   restart_stop_signal       = "SIGTERM"}

4. Open the pki-demo.sh file to review.

   #!/bin/shechoecho "==== CERT DEMO APP ===="echo "CERT = ${CERT}"

   This mock application gets certificate stored in the CERT environment variable. Similar to the kv-demo.sh, this application does not need to know anything about Vault.

5. Run a Vault Agent with additional env_template definition.

   $ vault agent -config=agent-config.hcl \   -config=pki-agent-config.hcl \   -log-level=error

   Example output:

   ==> Vault Agent started! Log data will stream in below:==> Vault Agent configuration:         Api Address 1: http://bufconn                     Cgo: disabled               Log Level: error               Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z            Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a==== CERT DEMO APP ====CERT = -----BEGIN CERTIFICATE-----MIIDZjCCAk6gAwIBAgIUdTJEGBZtcnxlo8f0mMLVzKvXbtgwDQYJKoZIhvcNAQELBQAwLTErMCkGA1UEAxMiZXhhbXBsZS5jb20gSW50ZXJtZWRpYXRlIEF1dGhvcml0eTAeFw0yMzA2MTMyMzQ3MDNaFw0yMzA2MTQyMzQ3MzNaMBsxGTAXBgNVBAMTEHRlc3QuZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC3GXTT5UlMnH5OJKZbx7n9DJs4K4lgrKdirpD4mVL7O4ba3KUA/mam9M6LAqn5YzB6UX6y1+zIyNaakqltxLVxiyLWodqix3fjN9g6jzis5JAQM3UIgVYPBpg5ptKzpVzHASvuDonnPflBYQG609rD56RK8ycyhq0bcUVCa6pEQI3EtJaqj2woOTVWyauo+r7gjrwr49BL0ye0T1IoTSs3l0E5Pk6ZcTDEbK3rNZ+YWSzwcg6bAL1U8kOW3B3jINwIGYXPKVVLM2v6TTYpeYB/fZ6FeQeDasVU3iy3N16HdIHbI2vAQqUkTZcv/LM1ALgfDbEYOUvAcdNCZKifhDTdAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgOoMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUBIsJvc4reYPkq1Vytuf3NMAHvD0wHwYDVR0jBBgwFoAUOiYGxwIjDoTz389uuIGb9VWmEKUwGwYDVR0RBBQwEoIQdGVzdC5leGFtcGxlLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEAZyN5dSKo9E8G6kaArX6Jfq8CGdX1jhaqFH8SljYZ5/WngzJwuQNkvCU42TpAbvHsBfen93T/VfGgPPh6GohwhvLbVQjTLzGBlFTq+oQJs0ntQxjL/CMEfqg0aLH1OnZuVULc9J6zLaOHhsV4EHx7c5iSuwwun7NOJgysadboS8EE4BOqkKehaEz/ZXVVM5sJhRpi8KfRI9EA+FNG6URKllW72OM2kbg9HVQsxsIPdG+o/JDsCz48yMwxJDyLAxMH/YtOVrbFj55N7ooFPw/h3BhMP+0zzDrQrl9GPYRCe//10zmTA+4/q2vQ3nPMVeInEa+QdTBihxz5k4LuDjhuoA==-----END CERTIFICATE-----

   Similar to what you observed in the restarts on secrets change section, if the certificate expires and the application is still running, Vault Agent can fetch a new certificate.

Clean up

Summary

You learned the use of the generate-config subcommand and the process supervisor mode to take advantage of secrets presented as environment variables.

Vault Agent can handle the authentication and secrets retrieval so that your application can remain Vault unaware. This reduces the barrier to adopting Vault and keep your applications secure.

There are several tutorials demonstrates the use of Vault Agent. See the available Vault Agent tutorials.

Help and reference

• Vault Agent overview
• Vault Agent generate-config
• Vault Agent's Process Supervisor Mode
• Vault Agent Auto-Auth