Exec2 Task Driver

Name: exec2

The exec2 driver is used to execute a command for a task. It offers a security model optimized for running 'ordinary' processes with very low startup times and minimal overhead in terms of CPU, disk, and memory utilization. The exec2 driver leverages kernel features such as the Landlock LSM, cgroups v2, and the unshare system utility.

Source is on GitHub

Installation

$ sudo apt-get update && \
  sudo apt-get install wget gpg coreutils

$ wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) test" | sudo tee /etc/apt/sources.list.d/hashicorp.list

$ sudo apt-get update && sudo apt-get install nomad-driver-exec2

$ sudo yum install -y yum-utils

$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo

$ sudo yum -y install nomad-driver-exec2

Usage

The example below is a short demonstration of what a task making use of the exec2 task driver looks like:

job "http" {
  group "web" {
    task "python" {
      driver = "exec2"
      config {
        command = "python3"
        args    = ["-m", "http.server", "8080", "--directory", "${NOMAD_TASK_DIR}"]
        unveil  = ["r:/etc/mime.types"]
      }
    }
  }
}

Client Requirements

The exec2 task driver is not built into Nomad. It must be downloaded onto the client host in the configured plugin directory.

• Linux 5.15+
• Cgroups v2 enabled
• Landlock LSM enabled
• Commands nsenter and unshare
• Nomad client running as root

Capabilities

The exec2 task driver implements the following driver capabilities

Concepts

Filesystem Isolation

landlock

The exec2 task driver makes use of go-landlock for providing filesystem isolation, making the host filesystem unreachable except where explicitly allowed.

By default a task is enabled to access its task directory and its shared alloc directory. The paths to these directories are accessible by reading the environment variables $NOMAD_TASK_DIR and $NOMAD_ALLOC_DIR respectively.

A file access mode must also be specified when granting additional filesystem access to a path. This is done by prefix the path with 'r', 'w', 'x', and/or 'c' indicating read, write, executable, and create permissions, respectively. e.g.,

• r:/srv/www - read-only access to `/srv/www`
• rwc:/tmp - read, write, and create files in `/tmp`
• rx:/opt/bin/application - read and execute a specific application
• wc:/var/log - write and create files in `/var/log`

This style of permission control is modeled after the unveil system call pioneered by the OpenBSD project. In configuration parameters we refer to "unveil"-ing of filesystem paths as exec2 is leveraging landlock to emulate the semantics of unveil.

dynamic workload users

While landlock prevents tasks from accessing the host filesystem, Nomad 1.8 introduces dynamic workload users which enable tasks to be run as a UID/GID that is not assigned to any user. This provides protection from non-root users getting access inside the task and allocation directories created for the task.

To make use of a dynamic workload user, simply leave the user field blank in the task definition of an exec2 task.

Resource Isolation

Similar to exec and other container runtimes, exec2 makes use of cgroups for limiting the amount of CPU and RAM a task may consume.

Configuration

Plugin Configuration

The default plugin configuration is shown below. System default paths are enabled, but nothing else. These default paths enable basic functionality like reading system TLS certificates, executing programs in /bin, /usr/bin, and accessing system shared object files. The exact set of paths is system dependent, and can be disabled or customized in plugin config.

The default set of paths are listed below. These paths are enabled only if they are found to exist at the time of the task launching.

shared objects

rx:/lib
rx:/lib64
rx:/usr/lib
rx:/usr/libexec
rx:/usr/local/lib
rx:/usr/local/lib64
r:/etc/ld.so.conf
r:/etc/ld.so.cache
r:/etc/ld.so.conf.d

io, common

rwc:/tmp
rw:/dev/null
r:/dev/zero
r:/dev/fd
rw:/dev/stdin
rw:/dev/stdout
r:/dev/urandom
w:/dev/log
r:/usr/share/locale
r:/proc/self/cmdline
r:/usr/share/zoneinfo
r:/usr/share/common-licenses
r:/proc/sys/kernel/ngroups_max
r:/proc/sys/kernel/cap_last_cap
r:/proc/sys/vm/overcommit_memory

dns

r:/etc/hosts
r:/hostname
r:/etc/services
r:/etc/protocols
r:/etc/resolv.conf

certificates

r:/etc/ssl/certs
r:/etc/pki/tls/certs
r:/etc/ssl/ca-bundle.pem
r:/etc/pki/tls/cacert.pem
r:/etc/pki/ca-trust-extracted/pem/tls-ca-bundle.pem
r:/etc/ssl/cert.pem

Additional allowable paths can be specified at the plugin level, which applies to all tasks making use of the exec2 task driver, or at the task level, which will apply specifically to each task.

plugin "nomad-driver-exec2" {
  config {
    unveil_defaults = true
    unveil_paths    = []
    unveil_by_task  = false
  }
}

• unveil_defaults - (default: true) - enable or disable default system paths useful for running basic commands
• unveil_paths - (default: []) - a list of filesystem paths with permissions to grant all tasks

  unveil_paths = ["rx:/opt/bin", "r:/srv/certs"]
  

  Copy
• unveil_by_task - (default: false) - enable or disable job submitters to specify additional filesystem path access within task config

Task Configuration

config

Task configuration for an exec2 task includes setting a command, args for the command, and additional unveil filesystem paths if unveil_by_task is enabled in plugin configuration.

config {
  command       = "/usr/bin/cat"
  args          = ["/etc/os-release"]
  unveil        = ["r:/etc/os-release"]
  oom_score_adj = 500
}

• command - (string: required) - command to run
• args - (list(string): optional) - list of arguments to provide to command
• unveil - (list(string): optional) - list of additional filesystem paths to provide access to the task (requires unveil_by_task in plugin config).
• oom_score_adj - (optional) - The likelihood of the task being OOM killed, must be a positive integer. Defaults to 0.

cpu

Tasks are limited in CPU resources by setting the cpu or cores values in the task resources block.

• cpu - (default: 100) - limits the CPU bandwidth allowable for the task to make use of in MHz, may not be used with cores.
• cores - (int: optional) - specifies the number of CPU cores to reserve specifically for the task, may not be used with cpu

memory

Tasks are limited in memory resources by setting memory and optionally the memory_max values in the task resources block.

Node Attributes

When installed, the exec2 plugin provides the following node attributes which can be used as constraints when authoring jobs.

driver.exec2.unveil.defaults    = true
driver.exec2.unveil.tasks       = true