Fault Injection

This topic describes the process to inject different types of faults into the services in Consul's service mesh to help you test your network's resilience.

This feature is available in Consul Enterprise.

Introduction

Consul allows you to configure fault injection filters to alter the responses from an upstream service. By injecting HTTP and gRPC statuses, bandwidth limits, or delays, users can test the resilience of their system to different unexpected issues.

Consul applies fault injection filters to an upstream service as part of its service defaults. For example, to simulate that an upstream returned a 500 HTTP status code 50 percent of the time, add an Abort configuration to the Envoy extensions block in the service default configuration for the upstream service.

The fault injection filters may be used individually, or in a combination of the three types by adding multiple blocks to the Arguments.Config section of the configuration.

Requirements

Consul Enterprise v1.18.0 or later

Inject delays

Specify the fault injection behavior in the service defaults configuration entry.

To inject a delay fault when Consul runs on virtual machines, configure the following parameters:

Arguments.Config.Delay.Duration: The delay in milliseconds to use when injecting a delay type fault.
Arguments.Config.Delay.Percentage: The percentage of responses to inject the delay behavior into.

The following example configures the default behavior for a service named billing. This configuration injects a 100 ms delay into 50 percent of responses from the billing service.

Kind = "service-defaults"
Name = "billing"
EnvoyExtensions = [
{
  Name = "builtin/fault-injection"
  Arguments = {
    Config = {
      Delay = {
        Duration = 100
        Percentage = 50
      }
    }
  }
}
]

To inject a delay fault when Consul runs on Kubernetes, configure the following parameters:

spec.arguments.config.delay.duration: The delay in milliseconds to use when injecting a delay type fault.
spec.arguments.config.delay.percentage: The percentage of responses to inject the delay behavior into.

The following example configures the default behavior for a service named billing. This configuration injects a 100 ms delay into 50 percent of responses from the billing service.

kind: ServiceDefaults
metadata:
  name: billing
spec:
  envoyExtensions:
  - name: "builtin/fault-injection"
  arguments:
    config:
      delay:
        duration: 100
        percentage: 50

Refer to the service defaults configuration entry reference for additional specifications and example configurations.

Inject statuses

To inject a status code when aborting a response, configure the following parameters:

Arguments.Config.Abort.HttpStatus: The HTTP status code to inject into responses when injecting an abort type fault. You may specify either a HTTP status or a gRPC status but not both.
Arguments.Config.Abort.GrpcStatus: The gRPC status to inject into responses when injecting an abort type fault. You may specify either a HTTP status or a gRPC status but not both.
Arguments.Config.Abort.Percentage: The percentage of responses to inject the abort behavior into.

The following example configures the default behavior for a service named billing. This configuration injects an HTTP status code of 503 into 5 percent of responses from the billing service.

Kind = "service-defaults"
Name = "billing"
EnvoyExtensions = [
{
  Name = "builtin/fault-injection"
  Arguments = {
    Config = {
      Abort = {
        HttpStatus = 503
        Percentage = 5
      }
    }
  }
}
]

To inject a status code when aborting a response, configure the following parameters:

spec.arguments.config.abort.httpStatus: The HTTP status code to inject into responses when injecting an abort type fault. You may specify either a HTTP status or a gRPC status but not both.
spec.arguments.config.abort.grpcStatus: The gRPC status to inject into responses when injecting an abort type fault. You may specify either a HTTP status or a gRPC status but not both.
spec.arguments.config.abort.percentage: The percentage of responses to inject the abort behavior into.

The following example configures the default behavior for a service named billing. This configuration injects an HTTP status code of 503 into 5 percent of responses from the billing service.

kind: ServiceDefaults
metadata:
  name: billing
spec:
  envoyExtensions:
  - name: "builtin/fault-injection"
  arguments:
    config:
      abort:
        httpStatus: 503
        percentage: 5

Refer to the service defaults configuration entry reference for additional specifications and example configurations.

Inject bandwidth limit

To inject a rate limit into a response, configure the following parameters:

Arguments.Config.Bandwidth.Limit: The amount of data allowed through the filter when applied. This value is specified in KiB/s. When the limit is exceeded, requests, operations, or connections will be subjected to rate limiting.
Arguments.Config.Bandwidth.Percentage: The percent of responses to inject the bandwidth limit behavior into.

The following example configures the default behavior for a service named billing. This configuration limits the bandwidth of outgoing requests to 100 KiB/s for 50 percent of responses from the billing service.

Kind = "service-defaults"
Name = "billing"
EnvoyExtensions = [
{
  Name = "builtin/fault-injection"
  Arguments = {
    Config = {
      Bandwidth = {
        Limit = 100
        Percentage = 50
      }
    }
  }
}
]

To inject a rate limit into a response, configure the following parameters:

spec.arguments.config.bandwidth.limit: The amount of data allowed through the filter when applied. This value is specified in KiB/s. When the limit is exceeded, requests, operations, or connections will be subjected to rate limiting.
spec.arguments.config.bandwidth.percentage: The percent of responses to inject the bandwidth limit behavior into.

kind: ServiceDefaults
metadata:
  name: billing
spec:
  envoyExtensions:
  - name: "builtin/fault-injection"
  arguments:
  config:
    bandwidth:
      limit: 100
      percentage: 50

Refer to the service defaults configuration entry reference for additional specifications and example configurations.