Secrets sync

In certain circumstances, fetching secrets directly from Vault is impossible or impractical. To help with this challenge, Vault can maintain a one-way sync for KVv2 secrets into various destinations that are easier to access for some clients. With this, Vault remains the system of records but can cache a subset of secrets on various external systems acting as trusted last-mile delivery systems.

A secret that is associated from a Vault KVv2 Secrets Engine into an external destination is actively managed by a continuous process. If the secret value is updated in Vault, the secret is updated in the destination as well. If the secret is deleted from Vault, it is deleted on the external system as well. This process is asynchronous and event-based. Vault propagates modifications into the proper destinations automatically in a handful of seconds.

Activating the feature

The secrets sync feature requires manual activation through a one-time trigger. If a sync-related endpoint is called prior to activation, an error response will be received indicating that the feature has not been activated yet. Be sure to understand the potential client count impacts of using secrets sync before proceeding.

Activating the feature can be done through one of several methods:

1. Activation directly through the UI.

2. Acitvation through the CLI:

   vault write -f sys/activation-flags/secrets-sync/activate
   

   Copy

3. Activation through a POST or PUT request:

   $ curl \
     --request PUT \
     --header "X-Vault-Token: ..." \
     http://127.0.0.1:8200/v1/sys/activation-flags/secrets-sync/activate
   

   Copy

Destinations

Secrets can be synced into various external systems, called destinations. The supported destinations are:

• AWS Secrets Manager
• Azure Key Vault
• GCP Secret Manager
• GitHub Repository Actions
• Vercel Projects

Associations

Syncing a secret into one of the external systems is done by creating a connection between it and a destination, which is called an association. These associations are created via Vault's API by adding a KVv2 secret target to one of the configured destinations. Each association keeps track of that secret's current sync status, the timestamp of its last status change, and the error code of the last sync or unsync operation if it failed. Each destination can have any number of secret associations.

Sync statuses

There are several sync statuses which relay information about the outcome of the latest sync operation to have occurred on that secret. The status information is stored inside each association object returned by the endpoint and, upon failure, includes an error code describing the cause of the failure.

Name template

By default, the name of synced secrets follows this format: vault/<accessor>/<secret-path>. The casing and delimiters may change as they are normalized according to the valid character set of each destination type. This pattern was chosen to prevent accidental name collisions and to clearly identify where the secret is coming from.

Every destination allows you to customize this name pattern by configuring a secret_name_template field to best suit individual use cases. The templates use a subset of the go-template syntax for extra flexibility.

The following placeholders are available:

Let's assume we want to sync the following secret:

$ VAULT_NAMESPACE=ns1/ns2 vault kv get -mount=path/to/kv1 path/to/secret1

========== Secret Path ==========
path/to/kv1/data/path/to/secret1

======= Metadata =======
(...)

=== Data ===
Key    Value
---    -----
foo    bar

Let's look at some name template examples and the resulting secret name at the sync destination.

Name templates can be updated. The new template is only effective for new secrets associated with the destination and does not affect the secrets synced with the previous template. It is possible to update an association to force a recreate operation. The secret synced with the old template will be deleted and a new secret using the new template version will be synced.

Custom tags

A destination can also have custom tags so that every secret associated to it that is synced will share that same set of tags. Additionally, a default tag value of hashicorp:vault is used to denote any secret that is synced via Vault Enterprise. Similar to secret names, tag keys and values are normalized according to the valid character set of each destination type.

Granularity

Vault KV-v2 secrets are multi-value and their data is represented in JSON. Multi-value secrets are useful to bundle closely related information together like a username & password pair. However, most secret management systems only support single-value entries. Secrets sync allows you to choose the granularity that best suits your use case for each destination by specifying a granularity field.

The secret-path granularity syncs the entire JSON content of the Vault secret as a single entry at the destination. If the destination does not support multi-value secret the JSON is encoded as a single-value JSON-string.

The secret-key granularity syncs each Vault key-value pair as a distinct entry at the destination. If the value itself is a list or map it is encoded as a JSON blob.

Granularity can be updated. The new granularity only affects secrets newly associated with the destination and does not modify the previously synced secrets. It is possible to update an association to force a recreate operation. The secret synced with the old granularity will be deleted and new secrets will be synced according to the new granularity.

Security

Vault access requirements

Vault verifies the client has read access on the secret before syncing it with any destination. This additional check is there to prevent users from maliciously or unintentionally leveraging elevated permissions on an external system to access secrets they normally wouldn't be able to.

Let's assume we have a secret located at path/to/data/secret1 and a user with write access to the sync feature, but no read access to that secret. This scenario is equivalent to this ACL policy:

# Allow full access to the sync feature
path "sys/sync/*" {
  capabilities = ["read", "list", "create", "update", "delete"]
}

# Allow read access to the secret mount path/to
path "path/to/*" {
  capabilities = ["read"]
}

# Deny access to a specific secret
path "path/to/data/my-secret-1" {
  capabilities = ["deny"]
}

If a client with this policy tries to read this secret they will receive an unauthorized error:

$ vault kv get -mount=path/to my-secret-1

Error reading path/to/data/my-secret-1: Error making API request.

URL: GET http://127.0.0.1:8200/v1/path/to/data/my-secret-1
Code: 403. Errors:

* 1 error occurred:
  * permission denied

Likewise, if the client tries to sync this secret to any destination they will receive a similar unauthorized error:

$ vault write sys/sync/destinations/$TYPE/$NAME/associations/set \
mount="path/to" \
secret_name="my-secret-1"

Error writing data to sys/sync/destinations/$TYPE/$NAME/associations/set: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/sync/destinations/$TYPE/$NAME/associations/set
Code: 403. Errors:

* permission denied to read the content of the secret my-secret-1 in mount path/to

This read access verification is only done when creating or updating an association. Once the association is created, revoking read access to the policy that was used to sync the secret has no effect.

Collisions and overwrites

Secrets Sync operates with a last-write-wins strategy. If a secret with the same name already exists at the destination, Vault overwrites it when syncing a secret. There are also no automatic mechanisms to prevent a principal with sufficient privileges at the destination from overwriting a secret synced by Vault.

To prevent Vault from accidentally overwriting existing secrets, it is recommended to use either a name pattern or built-in tags as an extra policy condition on the role used to configure a Vault sync destination. A negative condition on other policies may be used to prevent out-of-band overwrites to Vault secrets from non-Vault roles.

To see examples of policies that provide this type of restriction, refer to the access management section of the documentation for each destination type below:

• AWS Access Management
• GCP Access Management

Reconciliation

Vault Secrets Sync is designed to automatically recover from transient failures in two ways: operation retries and reconciliation scans.

Operation retries happen when a sync operation fails. Vault automatically retries the operation with exponential backoff. Operation retries help in situations where your network becomes unreliable or overwhelmed.

Reconciliation scans happen periodically in a background thread. Vault scans all secrets currently managed by the sync system to identify and update out-of-date secrets, and to ensure that any configured destinations are up-to-date. Reconciliation scans help in situations where there are external service downtimes that are outside of your control and provide a way to automatically recover and self-heal.

Operation retries and reconciliation scans are both enabled by default.

Note that reconciliation process do not protect from out-of-band updates that occur directly in the external service. The secrets sync system is designed to be one-way and does not support bidirectional sync at this time.

Client counts

Each secret that is synced with one or more destinations is counted as a distinct client in Vault's client counting. See entity assignments with secret sync for more information.

API

Please see the secrets sync API for more details.