Overview

This page contains the list of deprecations and important or breaking changes for Vault 1.10.x compared to 1.9. Please read it carefully.

SSH secrets engine

The new default value of algorithm_signer for SSH CA roles has been changed to rsa-sha2-256 from ssh-rsa. Existing roles will be migrated to explicitly specify the algorithm_signer=ssh-rsa for RSA keys if they used the implicit (empty) default, but newly created roles will use the new default value (preferring a literal default which presently uses rsa-sha2-256).

Etcd v2 API no longer supported

Support for the Etcd v2 API is removed in Vault 1.10. The Etcd v2 API was deprecated with the release of Etcd v3.5, and will be decommissioned in a forthcoming Etcd release.

Users of the etcd storage backend with the etcdv2 API that are upgrading to Vault 1.10 should migrate Vault storage to an Etcd v3 cluster prior to upgrading to Vault 1.10. All storage migrations should have backups taken prior to migration.

OTP generation process

Customers passing in OTPs during the process of generating root tokens must modify the OTP generation to include an additional 2 characters before upgrading so that the OTP can be xor-ed with the encoded root token. This change was implemented as a result of the change in the prefix from hvs. to s. for service tokens.

New error response for requests to perf standbys lagging behind active node

The introduction of Server Side Consistent Tokens means that when issuing a request to a perf standby right after having obtained a token (e.g. via login), if the token and its lease haven't yet been replicated to the perf standby, an HTTP 412 error will be returned. Before 1.10.0 this typically would've resulted in a 400, 403, or 50x response.

Token format change

Token prefixes were updated to be more easily identifiable.

Service tokens previously started with s. now start with hvs.
Batch tokens previously started with b. now start with hvb.
Recovery tokens previously started with r. now start with hvr.

Additionally, non-root service tokens are now longer than before. Previously, service tokens were 26 characters; they now have a minimum of 95 characters. However, existing tokens will still work.

Refer to the Server Side Consistent Token FAQ for details.

OIDC provider built-in resources

In Vault 1.9, the OIDC identity provider feature was released as a tech preview. In Vault 1.10, built-in resources were introduced to the OIDC provider system to reduce configuration steps and enhance usability.

The following built-in resources are included in each Vault namespace starting with Vault 1.10:

A default OIDC provider that's usable by all client applications
A default key for signing and verification of ID tokens
An allow_all assignment which authorizes all Vault entities to authenticate via a client application

If you created an OIDC provider with the name default, key with the name default, or assignment with the name allow_all using the Vault 1.9 tech preview, the installation of these built-in resources will be skipped. We strongly recommend that you delete any existing resources that have naming collisions before upgrading to Vault 1.10. Failing to delete resources with naming collisions could result unexpected default behavior. Additionally, we recommend reading the corresponding details in the OIDC provider concepts document to understand how the built-in resources are used in the system.

Known issues

Cluster initialization hangs with `retry_join`

The retry_join feature no longer successfully attempts to rejoin the raft cluster every 2 seconds following a join failure.

The error occurs when attempting to initialize non-leader nodes with a retry_join stanza. This affects multi-node raft clusters on impacted versions.

The bug was introduced by commit https://github.com/hashicorp/vault/commit/cc6409222ce246ed72d067debe6ffeb8f62f9dad and first reported in https://github.com/hashicorp/vault/issues/16486.

Impacted versions

Affects versions 1.11.1 and 1.10.5. Versions prior to these are unaffected.

NOTE: This error does not extend to version 1.9.8, which is slightly different in this portion of the code and does not exhibit the same behavior.

New releases addressing this bug are coming soon.

Raft in Vault uses its own set of TLS certificates, independent of those that the user controls to protect the API port and those used for replication and clustering. These certs get rotated daily, but to ensure that nodes which were down or behind on Raft log replication don't lose the ability to speak with other nodes, the newly generated daily TLS cert only starts being used once we see that all nodes have received it.

A recent security audit related change results in this rotation code getting a panic when the current cert is more than 24h old. This can happen if the cluster as a whole is down for a day or more. It can also happen if a single node is unreachable 24h, or sufficiently backlogged in applying raft logs that it's more than a day behind.

Impacted versions: 1.10.1, 1.9.5, 1.8.10. Versions prior to these are unaffected.

New releases addressing this panic are coming soon.

Rotation configuration persistence issue could lose transform tokenization key versions

A rotation performed manually or via automatic time based rotation after restarting or leader change of Vault, where configuration of rotation was changed since the initial configuration of the tokenization transform can result in the loss of intermediate key versions. Tokenized values from these versions would not be decodeable. It is recommended that customers who have enabled automatic rotation disable it, and other customers avoid key rotation until the upcoming fix.

Affected versions

This issue affects Vault Enterprise with ADP versions 1.10.x and higher. A fix will be released in Vault 1.11.9, 1.12.5, and 1.13.1.

Errors returned by perf standbys lagging behind active node with consul storage

The introduction of Server Side Consistent Tokens means that when issuing a request to a perf standby right after having obtained a token (e.g. via login), if the token and its lease haven't yet been replicated to the perf standby, an HTTP 412 error will be returned. Before 1.10.0 this wouldn't have resulted in the client seeing errors with Consul storage.

Single Vault follower restart causes election even with established quorum

We now support Server Side Consistent Tokens (See Replication and Vault Eventual Consistency), which introduces a new token format that can only be used on nodes of 1.10 or higher version. This new format is enabled by default upon upgrading to the new version. Old format tokens can be read by Vault 1.10, but the new format Vault 1.10 tokens cannot be read by older Vault versions.

For more details, see the Server Side Consistent Tokens FAQ.

Since service tokens are always created on the leader, as long as the leader is not upgraded before performance standbys, service tokens will be of the old format and still be usable during the upgrade process. However, the usual upgrade process we recommend can't be relied upon to always upgrade the leader last. Due to this known issue, a Vault cluster using Integrated Storage may result in a leader not being upgraded last, and this can trigger a re-election. This re-election can cause the upgraded node to become the leader, resulting in the newly created tokens on the leader to be unusable on nodes that have not yet been upgraded. Note that this issue does not impact Vault OSS users.

We will have a fix for this issue in Vault 1.10.3. Until this issue is fixed, you may be at risk of having performance standbys unable to service requests until all nodes are upgraded. We recommended that you plan for a maintenance window to upgrade.

Limited policy shows unhelpful message in UI after mounting a secret engine

When a user has a policy that allows creating a secret engine but not reading it, after successful creation, the user sees a message n is undefined instead of a permissions error. We will have a fix for this issue in an upcoming minor release.

Adding/Modifying Duo MFA method for enterprise MFA triggers a panic error

When adding or modifying a Duo MFA method for step-up Enterprise MFA using the sys/mfa/method/duo endpoint, a panic gets triggered due to a missing schema field. We will have a fix for this in Vault 1.10.1. Until this issue is fixed, avoid making any changes to your Duo configuration if you are upgrading Vault to v1.10.0.

Signing in to the Vault UI using an OIDC auth mount listed in the "tabs" of the form will result in the following error: "Authentication failed: role with oidc role_type is not allowed". The auth mounts listed in the "tabs" of the form are those that have listing_visibility set to unauth.

There is a workaround for this error that will allow you to sign in to Vault using the OIDC auth method. Select the "Other" tab instead of selecting the specific OIDC auth mount tab. From there, select "OIDC" from the "Method" select box and proceed to sign in to Vault.

A serious bug was identified in the Login MFA feature introduced in 1.10.0: #15108. Upon restart, Vault is not populating its in-memory MFA data structures based on what is found in storage. Although Vault is persisting to storage MFA methods and login enforcement configs populated via /identity/mfa, they will effectively disappear after the process is restarted.

We plan to issue a new 1.10.3 release to address this soon. We recommend delaying any rollouts of Login MFA until that release.