Workspace Best Practices

An HCP Terraform workspace manages a single state file and the lifecycle of its resources. It is the smallest collection of HCP Terraform-managed infrastructure. Any operation on a resource can potentially affect other resources managed in the same state file, so it is best to keep the potential blast radius of your operations small. To do so, manage resources in separate workspaces when possible, grouping together only necessary and logically-related resources. For example, even though your application may require both compute resources and a database, these resources can operate independently and should be in their own workspaces.

Scoping your configuration and planning your workspace strategy early in your adoption of HCP Terraform and Terraform Enterprise will simplify your operations and make them safer.

Group by volatility

Volatility refers to the rate of change of the resources in a workspace. Infrastructure such as databases, VPCs, and subnets change much less frequently than infrastructure such as your web servers. By exposing your long-living infrastructure to unnecessary volatility, you introduce more opportunities for accidental changes. When planning your workspace organization, group resources by volatility.

The above example groups together tightly-coupled resources like networking, security, and identity in a shared workspace. Compute, storage, and databases have separate workspaces, since they change at different frequencies. You may scale compute instances multiple times a day, but your database instances probably change far less frequently. By grouping these parts of your infrastructure into separate workspaces, you decouple unrelated resources and reduce the risk of unexpected changes.

Determine stateful vs stateless infrastructure

Stateful resources are ones that you cannot delete and recreate because they persist data, such as databases and object storage. By managing stateful resources independently of stateless ones, such as separating databases from compute instances, you limit the blast radius of operations that cause the resource recreation and help protect against accidental data loss.

Consider the workspace structure in the Volatility section. You could potentially manage filestore and database resources together, as they are both stateful resources. Your compute resources are stateless and should still have a separate workspace.

Separate privileges and responsibilities

A best practice is to split up workspaces based on team responsibilities and required privileges. For example, consider an application that requires separate developer and production environments, each with special networking and application infrastructure. One approach is to create four different workspaces, two for the developer environment and two for production. Only the networking team has access to the networking workspaces.

In this setup, only the networking team needs permissions to manage the resources in the networking workspaces, and others cannot manage those workspace resources. If a workspace's scope is too large, a user might need more permissions than appropriate in order to perform operations the workspace.

Splitting your workspaces by team also helps limit the responsibility per workspace and allows teams to maintain distinct areas of ownership. If you need to reference attributes of resources managed in other workspaces, you can share the outputs using the tfe_outputs data source. By limiting the scope of each workspace and sharing just the required outputs with others, you reduce the risk of leaking potentially sensitive information in a workspace's state. To share outputs from a workspace, you must explicitly enable remote state sharing in the workspace settings.

Avoid large Terraform plans and applies

HCP Terraform and Terraform Enterprise execute workloads using agents. Every time an agent refreshes a workspace's state, it builds a dependency graph of the resources to determine how to sequence operations in the workspace. As the number of resources your workspace manages grows, these graphs become larger and more complex. As these graphs grow, they require more worker RAM to build them. If your agent's performance degrades or workloads take longer to complete, we suggest exploring ways to split up the workspace to reduce the size of the dependency graph.

Determine workspace concurrency vs Terraform parallelism

Concurrency refers to the number of plan and apply operations HCP Terraform or Terraform Enterprise can run simultaneously. In HCP Terraform, your subscription limits the maximum concurrency. Terraform Enterprise lets you configure the concurrency, but defaults to 10 concurrent runs. As you increase concurrency, the amount of memory your Terraform Enterprise installation requires increases as well. Refer to the Capacity and performance documentation for more information.

Parallelism refers to the number of tasks the Terraform CLI performs simultaneously in a single workload. By default, Terraform performs a maximum of 10 operations in parallel. When running a terraform apply command, Terraform refreshes each resource in the state file and compares to the remote object. Every resource refresh, creation, update, or destruction is an individual operation. If your workload creates 11 resources, Terraform starts by creating the first 10 resources in its dependency graph, and will begin creating the 11th once it finishes creating one of the first 10 resources.

You can increase the parallelism of Terraform, but this increases a run's CPU usage. We recommend that you instead break down large Terraform configurations into smaller ones with fewer resources when possible. Long-running Terraform workloads are an early sign of a bloated workspace scope.

Next steps

This article introduces some considerations to keep in mind as your organization matures their workspace usage. Being deliberate about how you use these to organize your infrastructure will ensure smoother and safer operations. HCP Terraform provides a place to try these concepts hands-on, and you can get started for free.

To learn more about HCP Terraform and Terraform Enterprise best practices, refer to Project Best Practices. To learn best practices for writing Terraform configuration, refer to the Terraform Style Guide.