Parameterized jobs on Nomad
Nomad parameterized jobs enable dynamic runtime behaviors from a single job specification. These specialized batch-type jobs act less like regular Nomad jobs and more like functions. They are reusable, can accept parameters, and run only when invoked.
Parameterized jobs provide three key benefits to end users.
Creating instances of the job without resubmitting the job specification.
Providing dynamic run-time values as parameters to the job during dispatch.
Proving, when the job allows, an opaque payload that Nomad will write to one or more of the job's tasks.
Nomad registers, but does not run, a parameterized job when you submit it to
the cluster. To run a parameterized job, end-users invoke—or dispatch—it
with the nomad job dispatch command or Nomad Job
Dispatch API.
During dispatch, the user provides the parameter values and payload. Nomad combines these values with the registered job, creates a dispatched instance, and returns the scheduling information to the end-user. Once dispatched, the created instances are handled like regular Nomad batch jobs.
Challenge
In this tutorial, you will learn the fundamentals of Nomad parameterized jobs. You will take a Nomad batch job that renders a template, and
- Convert it to a parameterized job
- Enhance it with optional and required parameters
- Define default values for optional parameters
- Handle dispatch payloads
These fundamental practices can be used to create more complex parameterized workloads in your own environment.
Prerequisites
- Nomad v1.0 or greater
- Nomad dev agent or Nomad cluster
Configure your learning environment
Fetch the tutorial content
This tutorial uses content provided in the hashicorp/learn-nomad-jobspec
repository on GitHub. You can download a ZIP archive directly or use git to
clone the repository.
Unarchive the downloaded release.
The unzipping process creates a directory named learn-nomad-jobspec-release.
Change into it and into the tutorial's folder.
Configure tutorial environment
This tutorial uses HCL2 variables to reduce the amount of editing that you need
to do to the job specification based on your environment. The defaults are
optimized for a Linux/macOS boxes targeting a local Nomad dev agent using the
default datacenter value of dc1. These values can be customized using
environment variables before running the job file.
Override default task driver (optional)
This tutorial uses either the raw_exec or exec task driver (raw_exec by default) to render a text file and output it to standard output.
Because Windows and macOS do not support the exec driver and the raw_exec driver is not enabled by default on non-dev agents, this job uses an HCL2 variable to let you set your preferred driver without editing the job.
The driver variable defaults to raw_exec, suitable for all learners using a
dev agent. If you need to use the exec driver instead, set the
NOMAD_VAR_driver environment variable to exec.
Override default datacenter (optional)
The job also allows you to specify your datacenter value if it differs from the
default value of dc1. If you are using a non-default datacenter value,
set the NOMAD_VAR_datacenter environment variable.
If you are unsure of a valid datacenter value for your target cluster or dev
agent, run the nomad node status command to list all the datacenters
available.
In this sample output, all the clients are in a datacenter named mydc,
based on this output you would run:
Run the starting job
To get you started, the tutorial repository includes a Nomad batch job
specification that you will use as a base to build on. Use the nomad job run
command to run the start.nomad job file.
Make note of the allocation ID shown in your output. In the provided sample
output, the allocation ID is cb1d80a1.
Fetch the output of the job
Since this job writes its output to standard output, you can retrieve it with
the nomad alloc logs command. Run the nomad alloc logs command, supplying
the allocation ID you noted in the previous step.
Pro Tip - You only need to provide enough of the allocation ID to The
nomad alloc logs command to point to a distinct allocation. This works with
many identifiers in the Nomad command-line tool.
Purge the job
Nomad will not allow you to convert a batch job to a parameterized job without
purging the original version as a safety feature. When run with the -purge
flag, the nomad job stop command deletes the job and information about any
former runs of it from the Nomad server state.
Note
The start.nomad file defines a Nomad job named template. The
filename and job name are intentionally different allowing you to keep copies
of the different states of the job as you progress through the tutorial.
Make the job parameterized
Adding a parameterized stanza converts a batch-type job to a parameterized one.
An empty parameterized stanza creates a parameterized job that isn't
customizable at dispatch time. This is still a useful feature, since it enables
an operator or application to run an instance of the job without having to have
the job specification itself.
Add parameterized stanza
Copy the start.nomad file to a new file named parameterized.nomad.
Open parameterized.nomad in a text editor. Add an empty parameterized stanza
inside of the job stanza and save the file.
Register the job
Run the parameterized version of the job.
Notice that the output doesn't show any scheduling activity—no evaluation or allocation information. Parameterized jobs themselves do not run.
If you get an error
If you receive the following error, it indicates that you missed purging the
non-parameterized version of the template job. Run nomad job stop -purge template and re-run nomad job run parameterized.nomad to resolve it.
Run the nomad job status command to verify your parameterized job is available
for dispatch.
Dispatch the job
Run the nomad job dispatch command to dispatch an instance of the
parameterized job. The nomad job dispatch command's final argument is the
registered job ID to dispatch, unlike the nomad job run command which
uses the filename of the job specification.
Examine the output of the command. There are some key differences from the
typical output of the nomad job run command. Notice that Nomad generates a
Dispatched Job ID. This generated job ID refers to this specific instance of
the parameterized job, and it enables you to manage discrete instances of a
parameterized job in the same ways you manage your other Nomad batch jobs.
The output also provides scheduling information. Collect the allocation ID from
your output. In the preceding output it's 4e317cb8. Like earlier, run the
nomad alloc logs command for your allocation ID.
Parameterized jobs without variables can be used to provide a means for running a batch job without having to supply the job specification.
Use dispatch variables
Parameterized jobs also provide the ability to submit run-specific variables while dispatching the job. The job specification defines the variable names. They can be optional or required.
Nomad provides the dispatched variable values as environment variables. The
environment variable names that Nomad creates for these variables are prepended
with NOMAD_META_, for example NOMAD_META_customer_email.
Nomad will not create environment variables for unset optional variables; your workload should handle this case. You may also define defaults for optional variables as part of the job specification.
Update the job to use email.tmpl
Copy the parameterized.nomad file to a new file named variables.nomad.
Open the variables.nomad file in a text editor and change the HCL2 file
function to read in the email.tmpl file. This template generates an email-like
output.
Change
to
Define job variables
Required and optional variables are defined in the meta_required and
meta_optional attributes of the parameterized stanza, respectively. These
attributes take a list of quoted variable names the value.
The email.tmpl template needs more information to render properly:
- required -
customer_email,customer_name - optional -
rep_name,rep_email,rep_title,product_name
Define the variables by adding these attributes to the variables.nomad file
inside of the parameterized stanza.
Provide defaults for optional values
You can define default values for your optional variables using a job-level
meta stanza. Use the variable name as the attribute name and set the value to
the default.
For this example, create a default for the rep_name and the rep_email values
to provide generic values when the rep_name and rep_email are not provided.
In the variables.nomad file, add this meta stanza inside of the job
stanza.
Deploy and dispatch the job
Save the variables.nomad file and submit it using the nomad job run command.
Dispatch the job with variables
Now, dispatch a copy with the nomad job dispatch command. Use the -meta flag
to set the customer_email to alovelace@programmers.net and customer_name
to Ada Lovelace.
The command will output scheduling information.
Fetch the template output using the nomad alloc logs command for the
allocation ID output when you dispatched the job.
Test the required variables
Nomad validates that the dispatch request contains all required variables. The operator or application dispatching the job must provide all required variables when dispatching the job. If not, Nomad will return an error.
Observe this by dispatching the job with no variables set.
Use the optional variables
Now, use the optional variables to personalize the email. Dispatch the job again with the following command.
The command outputs the scheduling information.
Fetch the output from the generated allocation using the nomad alloc logs
command.
Consume a dispatched payload
When defining a Nomad parameterized job, you have the option to use a
payload. A payload is a means to supply up to 16Kib of opaque data to
instances of your parameterized job during dispatch.
Note
These payloads are included in the dispatched version's job definition data and will consume a corresponding amount of server memory until the dispatched instance of the job is garbage collected.
The parameterized stanza's payload value determines if a payload is
required, optional, or forbidden when the job is dispatched.
Nomad writes the dispatch payload into any task containing a dispatch_payload
stanza. The path attribute specifies where to write the data relative to the
task's local directory.
Enable a mandatory payload
Copy the variables.nomad file to a new file named payload.nomad.
Open payload.nomad in your text editor. Add a payload = "required attribute inside the parameterized stanza.
Write the payload into the task
Also, add a dispatch_payload stanza inside the output task stanza.
Submit the updated job
Save the payload.nomad file and submit the job to Nomad.
Dispatch the job with the payload
There are two ways to provide a payload to the nomad job dispatch command
- from a file
- from standard input
Get payload from a file
The tutorial repository contains a sample text file for you to use as the payload—engine.txt
Use the nomad job dispatch command to submit the payload directly
The command outputs the scheduling information.
Note the allocation ID; retrieve the output of the allocation using the nomad alloc logs command.
Get payload from standard input
Using standard input allows you submit dynamic payloads to your parameterized
jobs. Use - as the filename in the nomad job dispatch command to collect the
payload from standard input. This example uses the date command to generate
a dynamic payload.
Fetch the output of the dispatched allocation using the nomad alloc logs
command.
Clean up
Clean up the target environment
The cleanup instructions vary depending on your target infrastructure. Select the one that applies.
Nomad dev agent
Stop the Nomad dev agent with Ctrl-C. Nomad dev agents use ephemeral storage
and require
no further cleanup.
Nomad cluster
Since Nomad clusters maintain persistent state, you will need to perform extra steps to remove all the tutorial artifacts from the cluster.
Remove the parameterized job
Stop the template job. You can optionally pass the -purge flag to tell Nomad
to immediately remove it from the server state. Since no scheduling activity is
required to remove the job, there is no output generated by the command.
Remove dispatched instances (optional)
Each time you dispatch a parameterized job, Nomad creates a distinct but related
job. These jobs will naturally be removed in time by Nomad's garbage collection
process. You can see a list of these instances by using the nomad job status
command referencing an ambiguous prefix, like template\. These IDs will be
specific to your environment.
Pro Tip - The Nomad command-line tool displays a list of matching IDs whenever an incomplete ID matches more than one item.
These jobs will naturally be removed in time by Nomad's garbage collection process.
You can also use the nomad job stop command with the -purge flag set to
remove individual dispatched instances immediately. For example:
Remove the tutorial's working directory
Ensure that you have closed any file editors that might have files in the working directory open.
Change to your home directory and recursively remove the learning environment you created in the first step.
Remove the unarchived learning environment.
Delete the downloaded archive.
Review what you learned
You create a parameterized job by adding the parameterized stanza to a
batch-type Nomad job specification.
You dispatch an instance of a parameterized job by running thennomad job dispatch
command or using the Job Dispatch API.
Nomad dispatch variables are supplied to the workload as environment variables.
The environment variable's names are generated by prepending NOMAD_META_ to
the variable's name, like NOMAD_META_customer_email.
Dispatch payloads can be up to 16KiB. Payloads are written directly into the dispatched job's definition, so they use that same amount of server RAM until the job is complete and garbage collected.
Next steps
Nomad parameterized jobs are a powerful tool, encouraging reuse of a job specification by allowing users to provide runtime specific values when dispatching the job. Parameterized jobs enable job creators to make consistent and reusable experiences by encapsulating the specifics of the job and only requiring users to provide the run-time specific information. This ultimately simplifies the user experience for both end-users and API consumers.
In this tutorial, you performed these job specification development actions:
- Converted a batch job to a parameterized job.
- Added required and optional variables to the job specification.
- Defined default values for optional variables
- Configured the job to use a dispatch payload.
You performed these end-user operations:
- Dispatched instances of a parameterized job
- Provided variables during dispatch
- Submitted a static and dynamic payload during dispatch
You performed these cluster-operator tasks:
- Retrieved logs for an allocation using the
nomad alloc logscommand
Learn more about the template language used in the .tmpl files in the Learn
Go Template Syntax and in the consul-template
documentation. You can also learn more about parameterized jobs in the
Nomad documentation.