Express job placement preferences with affinities
The affinity stanza allows operators to express placement preferences for their jobs on particular types of nodes. Note that there is a key difference between the constraint stanza and the affinity stanza. The constraint stanza strictly filters where jobs are run based on attributes and client metadata. If no nodes are found to match, the placement does not succeed. The affinity stanza acts like a "soft constraint." Nomad will attempt to match the desired affinity, but placement will succeed even if no nodes match the desired criteria. This is done in conjunction with scoring based on the Nomad scheduler's bin packing algorithm which you can read more about here.
In this guide, you will encounter a sample application. Your application can run
in datacenters dc1
and dc2
; however, you have a strong preference to run it in
dc2. You will learn how to configure the job to inform the scheduler of your
preference, while still allowing it to place your workload in dc1
if the
desired resources aren't available in dc2.
By specify an affinity with the proper weight, the Nomad scheduler can find the best nodes on which to place your job. The affinity weight will be included when scoring nodes for placement along with other factors like the bin-packing algorithm.
Prerequisites
To perform the tasks described in this guide, you need to have a Nomad environment with Consul installed. You can use this repository to provision a sandbox environment. This tutorial will assume a cluster with one server node and three client nodes.
Tip
This tutorial is for demo purposes and is only using a single server node. In a production cluster, 3 or 5 server nodes are recommended.
Place one of the client nodes in a different datacenter
You are going express your job placement preference based on the datacenter your
nodes are located in. Choose one of your client nodes and edit
/etc/nomad.d/nomad.hcl
to change its location to dc2
. A snippet of an
example configuration file is show below with the required change is shown
below.
After making the change on your chosen client node, restart the Nomad service
If everything worked correctly, one of your nodes will now show datacenter dc2
when you run the nomad node status
command.
Create a job with an affinity
Create a file with the name redis.nomad.hcl
and place the following content in it:
Observe that the job uses the affinity
stanza and specifies dc2
as the value
for the ${node.datacenter}
attribute. It also uses the value
100
for the weight which will cause the Nomad scheduler to rank
nodes in datacenter dc2
with a higher score. Keep in mind that weights can
range from -100 to 100, inclusive. Negative weights serve as anti-affinities
which cause Nomad to avoid placing allocations on nodes that match the criteria.
Register the redis Nomad job
Run the Nomad job with the following command:
Note that two of the allocations in this example have been placed on node
6b6e9518
. This is the node configured to be in datacenter dc2
. The Nomad
scheduler selected this node because of the affinity specified. All of the
allocations have not been placed on this node because the Nomad scheduler
considers other factors in the scoring such as bin-packing. This helps avoid
placing too many instances of the same job on a node and prevents reduced
capacity during a node level failure. You will take a detailed look at the
scoring in the next few steps.
Check the status of the job
At this point, Check the status of the job and verify where the allocations have been placed. Run the following command:
There should be four instances of the job running in the Summary
section of
the output as shown below:
You can cross-check this output with the results of the nomad node status
command to verify that the majority of your workload has been placed on the node
in dc2
. In the case of the above output, that node is 6b6e9518
.
Obtain detailed scoring information on job placement
The Nomad scheduler will not always place all of your workload on nodes you have
specified in the affinity
stanza even if the resources are available. This is
because affinity scoring is combined with other metrics as well before making a
scheduling decision. In this step, you will take a look at some of those other
factors.
Using the output from the previous step, find an allocation that has been placed
on a node in dc2
and use the nomad alloc status
command with
the -verbose
option to obtain detailed scoring information on it.
In this example, the allocation ID to be inspected is 0dfcf0ba
(your
allocation IDs will be different).
The resulting output will show the Placement Metrics
section at the bottom.
Note that the results from the binpack
, job-anti-affinity
,
node-reschedule-penalty
, and node-affinity
columns are combined to produce
the numbers listed in the final score
column for each node. The Nomad
scheduler uses the final score for each node in deciding where to make
placements.
Next steps
Experiment with the weight provided in the affinity
stanza (the value can be
from -100 through 100) and observe how the final score given to each node
changes (use the nomad alloc status
command as shown in the previous step).
Reference material
- The affinity stanza documentation
- Scheduling with Nomad