Express inter-job dependencies with init tasks

8min
|
Nomad
Consul

Nomad task dependencies provide the ability to define prestart tasks.

Prestart tasks have two patterns: init tasks and sidecar tasks. Init tasks are tasks that must run to completion before the main workload is started. They are commonly used to download assets or to create necessary tables for an extract-transform-load (ETL) job. Sidecar tasks are started before main workload starts and run for the lifetime of the main workload. Typical sidecars tasks are log forwarders, proxies, and for platform abstractions. This tutorial demonstrates an init task.

You can create an init task by adding a lifecycle stanza with hook set to prestart and sidecar to false as below.

      lifecycle {
        hook    = "prestart"
        sidecar = false
      }

You can model complex job dependency trees by using one or more init tasks to delay the job's main tasks from running until a condition is met. In this case, until a service is available and advertised in Consul.

In this tutorial you will work with several Nomad objects:

mock-app - a job file that contains two tasks
- await-mock-service - an init task that will wait infinitely for a service named "mock-service" to be advertised over the Consul DNS API. Once found, it will exit successfully.
- mock-app-container - the main workload task that is dependent on the "mock-service" service.
mock-service - a job that contains one task which advertises a service named "mock-service". This is provided as a Nomad job as a convenience, but could be replaced by any means of registering a service named "mock-service" in Consul, like the CLI or API.

In this guide, you will complete the following actions:

Deploy the "mock-app" job.
Verify that the "mock-app-container" task remains in pending and unstarted.
Start the "mock-service" job.
Verify that the "await-mock-service" container completes successfully and that the "mock-app-container" task starts.

Prerequisites

To complete this tutorial you will need:

a Nomad cluster and at least one Consul server.
Nomad v0.11.0 or greater

If you do not have an existing Nomad cluster, you can learn how to deploy on using the Install Nomad guide. Similarly, if you do not have an existing Consul datacenter, you can learn how to deploy Consul with the Install Consul guide.

Create the mock-app job file

This example uses a looping script, in the config stanza, to mock service payloads.

Create an HCL file named mock-app.nomad.hcl with the following content.

job "mock-app" {
  datacenters = ["dc1"]
  type        = "service"

  group "mock-app" {
    # disable deployments
    update {
      max_parallel = 0
    }

    task "await-mock-service" {
      driver = "docker"

      config {
        image        = "busybox:1.28"
        command      = "sh"
        args         = ["-c", "echo -n 'Waiting for service'; until nslookup mock-service.service.consul 2>&1 >/dev/null; do echo '.'; sleep 2; done"]
        network_mode = "host"
      }

      resources {
        cpu    = 200
        memory = 128
      }

      lifecycle {
        hook    = "prestart"
        sidecar = false
      }
    }

    task "mock-app-container" {
      driver = "docker"

      config {
        image   = "busybox"
        command = "sh"
        args    = ["-c", "echo The app is running! && sleep 3600"]
      }

      resources {
        cpu    = 200
        memory = 128
      }
    }
  }
}

The job contains two tasks—"await-mock-service" and "mock-app". The "await-mock-service" task is configured to busy-wait until the "mock-service" service is advertised in Consul. For this guide, this will not happen until you run the mock-service.nomad.hcl job. In a more real-world case, this could be any service dependency that advertises itself in Consul.

You can use this pattern to model more complicated chains of service dependency by including more await-style workloads.

Ensure that name resolution works properly

Since the "await-mock-service" task uses nslookup inside of a Docker container, you will need to ensure that your container can perform lookups against your Consul DNS API endpoint. This tutorial uses network_mode = host to allow the container to use the Nomad client nodes DNS resolution pathway.

The nslookup application will only perform queries on the standard DNS port (53). You might need to use an application to forward requests from port 53 to the Consul DNS API port—port 8600 by default. You can learn several ways to accomplish this forwarding in the Forward DNS Learn guide.

You could also add a dns_servers value to the config stanza of the "await-mock-service" task in the mock-app.nomad.hcl file to direct the query to a DNS server directly that meets the above criteria.

Run the mock-app job

Run nomad run mock-app.nomad.hcl.

$ nomad run mock-app.nomad.hcl

The job will launch and provide you an allocation ID in the output.

$ nomad run mock-app.nomad.hcl
==> Monitoring evaluation "01c73d5a"
    Evaluation triggered by job "mock-app"
    Allocation "3044dda0" created: node "f26809e6", group "mock-app"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "01c73d5a" finished with status "complete"

Verify mock-app-container is pending

Run the nomad alloc status command with the provided allocation ID.

$ nomad alloc status 3044dda0

The nomad alloc status command provides you with useful information about the resource. For this guide, focus on the status of each task. Each task's status is output in lines that look like Task "await-mock-service" is "running".

$ nomad alloc status 3044dda0
ID                  = 3044dda0-8dc1-1bac-86ea-66a3557c67d3
Eval ID             = 01c73d5a
Name                = mock-app.mock-app[0]
Node ID             = f26809e6
Node Name           = nomad-client-2.node.consul
Job ID              = mock-app
Job Version         = 0
Client Status       = running
Client Description  = Tasks are running
Desired Status      = run
Desired Description = <none>
Created             = 43s ago
Modified            = 42s ago

Task "await-mock-service" (prestart) is "running"
Task Resources
CPU        Memory          Disk     Addresses
3/200 MHz  80 KiB/128 MiB  300 MiB

Task Events:
Started At     = 2020-03-18T17:07:26Z
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type        Description
2020-03-18T13:07:26-04:00  Started     Task started by client
2020-03-18T13:07:26-04:00  Task Setup  Building Task Directory
2020-03-18T13:07:26-04:00  Received    Task received by client

Task "mock-app-container" is "pending"
Task Resources
CPU      Memory   Disk     Addresses
200 MHz  128 MiB  300 MiB

Task Events:
Started At     = N/A
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type      Description
2020-03-18T13:07:26-04:00  Received  Task received by client

Notice that the await-mock-service task is running and that the "mock-app-container" task is pending. The "mock-app-container" task will remain in pending until the "await-mock-service" task completes successfully.

Create the mock-service job file

Create a file named mock-service.nomad.hcl with the following content.

job "mock-service" {
  datacenters = ["dc1"]
  type        = "service"

  group "mock-service" {
    task "mock-service" {
      driver = "docker"

      config {
        image   = "busybox"
        command = "sh"
        args    = ["-c", "echo The service is running! && while true; do sleep 2; done"]
      }

      resources {
        cpu    = 200
        memory = 128
      }

      service {
        name = "mock-service"
      }
    }
  }
}

This job advertises the "mock-service" service in Consul. When run, this will allow the await-mock-service task to complete successfully and let the "mock-app-container" task start up.

Start mock-service job

Run nomad run mock-service.nomad.hcl.

$ nomad run mock-service.nomad.hcl

Nomad will start the job and return information about the scheduling information.

$ nomad run mock-service.nomad
==> Monitoring evaluation "f31f8eb1"
    Evaluation triggered by job "mock-service"
    Allocation "d7767adf" created: node "f26809e6", group "mock-service"
    Evaluation within deployment: "3d86e09a"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "f31f8eb1" finished with status "complete"

Verify mock-app-container is running

Finally, check the output of the nomad alloc status command again to check the task statuses. Use the allocation ID from when you ran the "mock-app" job.

$ nomad alloc status 3044dda0

Again, focus on the task status lines for "await-mock-service" and "mock-app-container".

ID                  = 3044dda0-8dc1-1bac-86ea-66a3557c67d3
Eval ID             = 01c73d5a
Name                = mock-app.mock-app[0]
Node ID             = f26809e6
Node Name           = nomad-client-2.node.consul
Job ID              = mock-app
Job Version         = 0
Client Status       = running
Client Description  = Tasks are running
Desired Status      = run
Desired Description = <none>
Created             = 21m38s ago
Modified            = 7m27s ago

Task "await-mock-service" (prestart) is "dead"
Task Resources
CPU        Memory          Disk     Addresses
0/200 MHz  80 KiB/128 MiB  300 MiB

Task Events:
Started At     = 2020-03-18T17:07:26Z
Finished At    = 2020-03-18T17:21:35Z
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type        Description
2020-03-18T13:21:35-04:00  Terminated  Exit Code: 0
2020-03-18T13:07:26-04:00  Started     Task started by client
2020-03-18T13:07:26-04:00  Task Setup  Building Task Directory
2020-03-18T13:07:26-04:00  Received    Task received by client

Task "mock-app-container" is "running"
Task Resources
CPU        Memory          Disk     Addresses
0/200 MHz  32 KiB/128 MiB  300 MiB

Task Events:
Started At     = 2020-03-18T17:21:37Z
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type        Description
2020-03-18T13:21:37-04:00  Started     Task started by client
2020-03-18T13:21:35-04:00  Driver      Downloading image
2020-03-18T13:21:35-04:00  Task Setup  Building Task Directory
2020-03-18T13:07:26-04:00  Received    Task received by client

Notice, the "await-mock-service" task is dead and based on the "Recent Events" table terminated with "Exit Code: 0". This indicates that it completed successfully. The "mock-app-container" task has now transitioned to the "running" status and the container is running.

Next steps

Now that you have completed this guide, you have experimented with using Nomad task dependencies to model inter-job dependencies.