containerd Task Driver

Name: containerd-driver

Homepage: https://github.com/Roblox/nomad-driver-containerd

containerd (containerd.io) is a lightweight container daemon for running and managing container lifecycle. Docker daemon also uses containerd.

dockerd (docker daemon) --> containerd --> containerd-shim --> runc

nomad-driver-containerd enables Nomad clients to launch containers directly using containerd, without Docker. The Docker daemon is therefore not required on the host system.

See the project's homepage for more details.

Client Requirements

The containerd task driver is not built into Nomad. It must be downloaded onto the client host in the configured plugin directory.

Linux (Ubuntu >=16.04) with containerd (>=1.3) installed.
containerd-driver binary in Nomad's plugin_dir.

Capabilities

The containerd-driver implements the following capabilities.

Feature	Implementation
send signals	true
exec	true
filesystem isolation	image
network isolation	host, group, task, none
volume mounting	true

For sending signals, one can use nomad alloc signal command.
For exec'ing into the container, one can use nomad alloc exec command.

Task Configuration

Since Docker also relies on containerd for managing container lifecycle, the example job created by nomad init -short can easily be adapted to use containerd-driver instead:

job "redis" {
  datacenters = ["dc1"]

  group "redis-group" {
    task "redis-task" {
      driver = "containerd-driver"

      config {
        image = "docker.io/library/redis:alpine"
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

The containerd task driver supports the following parameters:

image - (Required) OCI image (Docker is also OCI compatible) for your container.

config {
  image = "docker.io/library/redis:alpine"
}

command - (Optional) Command to override command defined in the image.

config {
  command = "some-command"
}

args - (Optional) Arguments to the command.

config {
  args = [
    "arg1",
    "arg2",
  ]
}

cwd - (Optional) Specify the current working directory (cwd) for your container process. If the directory does not exist, one will be created for you.
privileged - (Optional) true or false (default) Run container in privileged mode. Your container will have all Linux capabilities when running in privileged mode.

config {
  privileged = true
}

host_dns - (Optional) true (default) or false By default, a container launched using containerd-driver will use host /etc/resolv.conf. This is similar to Docker's behavior. However, if you don't want to use host DNS, you can turn off this flag by setting host_dns=false.
seccomp - (Optional) Enable default seccomp profile. List of allowed syscalls.
seccomp_profile - (Optional) Path to custom seccomp profile. seccomp must be set to true in order to use seccomp_profile.
The default docker seccomp profile found in the Moby repository can be downloaded, and modified (by removing/adding syscalls) to create a custom seccomp profile. The custom seccomp profile can then be saved under /opt/seccomp/seccomp.json on the Nomad client nodes.

config {
  seccomp         = true
  seccomp_profile = "/opt/seccomp/seccomp.json"
}

readonly_rootfs - (Optional) true or false (default) Container root filesystem will be read-only.

config {
  readonly_rootfs = true
}

host_network - (Optional) true or false (default) Enable host network. This is equivalent to --net=host in docker.

config {
  host_network = true
}

cap_add - (Optional) Add individual capabilities.

config {
  cap_add = [
    "CAP_SYS_ADMIN",
    "CAP_CHOWN",
    "CAP_SYS_CHROOT"
  ]
}

cap_drop - (Optional) Drop individual capabilities.

config {
  cap_drop = [
    "CAP_SYS_ADMIN",
    "CAP_CHOWN",
    "CAP_SYS_CHROOT"
  ]
}

devices - (Optional) A list of devices to be exposed to the container.

config {
  devices = [
    "/dev/loop0",
    "/dev/loop1"
  ]
}

mounts - (Optional) A list of mounts to be mounted in the container. Volume, bind and tmpfs type mounts are supported. fstab style mount options are supported.
- type - (Optional) Supported values are volume, bind or tmpfs. Default: volume.
- target - (Required) Target path in the container.
- source - (Optional) Source path on the host.
- options - (Optional) fstab style mount options. NOTE: For bind mounts, at least rbind and ro are required.

config {
  mounts = [
    {
      type = "bind"
      target = "/tmp/t1"
      source = "/tmp/s1"
      options = ["rbind", "ro"]
    }
  ]
}

Networking

nomad-driver-containerd supports host and bridge networks.

NOTE: host and bridge are mutually exclusive options, and only one of them should be used at a time.

Host network can be enabled by setting host_network to true in task config of the job spec (see host_network under Task Configuration).
Bridge network can be enabled by setting the network stanza in the task group section of the job spec.

network {
  mode = "bridge"
}

You need to install CNI plugins on Nomad client nodes under /opt/cni/bin before you can use bridge networks.

Instructions for installing CNI plugins.

 $ curl -L -o cni-plugins.tgz "https://github.com/containernetworking/plugins/releases/download/v0.9.0/cni-plugins-linux-$( [ $(uname -m) = aarch64 ] && echo arm64 || echo amd64)"-v0.9.0.tgz
 $ sudo mkdir -p /opt/cni/bin
 $ sudo tar -C /opt/cni/bin -xzf cni-plugins.tgz

Also, ensure your Linux operating system distribution has been configured to allow container traffic through the bridge network to be routed via iptables. These tunables can be set as follows:

 $ echo 1 > /proc/sys/net/bridge/bridge-nf-call-arptables
 $ echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
 $ echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

To preserve these settings on startup of a Nomad client node, add a file including the following to /etc/sysctl.d/ or remove the file your Linux distribution puts in that directory.

 net.bridge.bridge-nf-call-arptables = 1
 net.bridge.bridge-nf-call-ip6tables = 1
 net.bridge.bridge-nf-call-iptables = 1

Port Forwarding

Nomad supports both static and dynamic port mapping.

Static ports

Static port mapping can be added in the network stanza.

network {
  mode = "bridge"
  port "lb" {
    static = 8889
    to     = 8889
  }
}

Here, host port 8889 is mapped to container port 8889.
NOTE: static ports are usually not recommended, except for system or specialized jobs like load balancers.

Dynamic ports

Dynamic port mapping is also enabled in the network stanza.

network {
  mode = "bridge"
  port "http" {
    to = 8080
  }
}

Here, nomad will allocate a dynamic port on the host and that port will be mapped to 8080 in the container.

You can read more about configuring networking under the network stanza documentation.

Service discovery

Nomad schedules workloads of various types across a cluster of generic hosts. Because of this, placement is not known in advance and you will need to use service discovery to connect tasks to other services deployed across your cluster. Nomad integrates with Consul to provide service discovery and monitoring.

A service block can be added to your job spec, to enable service discovery.

The service stanza instructs Nomad to register a service with Consul.

Plugin Options

enabled - (Optional) The containerd driver may be disabled on hosts by setting this option to false (defaults to true).
containerd_runtime - (Required) Runtime for containerd e.g. io.containerd.runc.v1 or io.containerd.runc.v2
stats_interval - (Optional) This value defines how frequently you want to send TaskStats to nomad client. (defaults to 1 second).

An example of using these plugin options with the new plugin syntax is shown below:

plugin "containerd-driver" {
  config {
    enabled = true
    containerd_runtime = "io.containerd.runc.v2"
    stats_interval = "5s"
  }
}

Please note the plugin name should match whatever name you have specified for the external driver in the plugin_dir directory.