If you are not familiar with the oc command, refer to OpenShift - Getting Started with the oc command.

There are a few different ways to run a pod on a specific node or to prevent a pod from running on certain nodes.

• Using nodeSelector
  • Nodes are labeled with a key=value (e.g. region=east)
  • If namespace node-selector is an exact match of a nodes key=value label, then the pod can be scheduled run on the node
  • If a pods "nodeSelector" is an exact match of a nodes key=value label, then the pod can be scheduled run on the node
• Using nodeAffinity
  • Nodes are labeled with a key=value (e.g. region=east)
  • If a pods "nodeAffinity" regular expression matches a nodes key=value label, the pod can be scheduled to run on the node
• Using podAffinity
  • Pods are labeled with a key=value (e.g. region=east)
  • If a pods "podAffinity" regular expression matches another pods key=value label, the pod can be scheduled to run on the same node that the other pods is running on
• Using Taint and Toleration (this article) to prevent pods from running on certain nodes
  • Taint is used to prevent a pod from being scheduled on certain nodes unless the pod is configured with a toleration to "tolerate the taint"
  • Nodes are labeled with a key=value:taint (e.g. region=east:NoSchedule)
  • If a pods tolerations uses "Exists" and the pods tolerations key matches the nodes toleration key, the pod is allowed to run on the node
  • If a pods tolerations uses "Equal" and the pods tolerations key and value is an exact match of the nodes tolerations key and value, the pod is allowed to run on the node

The scheduler is responsible for determining which node a resource should get created on. For example, when deploying a new application to OpenShift, the scheduler determines which worker node the pod should be created on, typically the worker node with the most available memory and CPU. Check out my article on the default scheduler.

The oc adm taint command can be used to apply a taint to a node. The taint takes three fields: key=value:effect.

Following are the effects.

• NoSchedule
  • If the tolerations operator is Equal and the tolerations key=value:effect is NOT an exact match of the nodes key=value:effect taint, the pod cannot be scheduled on the node
  • If the tolerations operator is Exists and the tolerations key and effect is NOT an exact match of the nodes key and effect taint, the pod cannot be scheduled on the node
• NoExecute
  • ​If the tolerations operator is Equal and the tolerations key=value:effect is NOT an exact match of the nodes nodes key=value:effect taint and the pod is running on the tainted node, the pod will be evicted from the node
  • If the tolerations operator is Exists and the tolerations key and effect is NOT an exact match of the nodes key and effect taint and the pod is running on the tainted node, the pod will be evicted from the node
• PreferNoSchedule
  • If the tolerations operator is Equal and the tolerations key=value:effect is NOT an exact match of the nodes key=value:effect taint, the pod will prefer to not be scheduled on the node
  • If the tolerations operator is Exists and the tolerations key and effect is NOT an exact match of the nodes key and effect taint, the pod will prefer to not be scheduled on the node

In this example, two worker nodes are tainted.

~]$ oc adm taint node my-node-worker-5n4fj region=east:NoSchedule
node/my-node-worker-5n4fj tainted

~]$ oc adm taint node my-node-worker-v8r9r region=west:NoSchedule
node/my-node-worker-v8r9r tainted

The oc describe node command can be used to see the taint that have been applied to the worker nodes.

~]$ oc describe node my-node-worker-5n4fj
Taints:             region=east:NoSchedule

~]$ oc describe node my-node-worker-v8r9r
Taints:             region=west:NoSchedule

namespace toleration

namespaces can be configured with a default toleration so that deployments / pods / replica sets / stateful sets in the namespace will "tolerate the taint" thus the pod will be allowed to be scheduled on the node. It is noteworthy that daemon sets are excluded from this control. The oc patch command can be used to configure my-project with a default toleration.

The operator can be Equal or Exists.

If Equal is used, the key (region in this example) and value (east in this example) must be an exact match of the node taint key=value for the pod to "tolerate the taint". In this example, pods in the namespace can be scheduled on nodes with taint region=east.

oc patch namespace my-project --patch '{"metadata":{"annotations":{"scheduler.alpha.kubernetes.io/defaultTolerations": "[{\"Key\": \"region\", \"Operator\": \"Equal\", \"Value\": \"east\"}]"}}}'

If Exists is used, the key (region in this example) must be an exact match of the node taint key for the pod to "tolerate the taint". In this example, pods in the namespace can be scheduled on nodes with taint region.

oc patch namespace my-project --patch '{"metadata":{"annotations":{"scheduler.alpha.kubernetes.io/defaultTolerations": "[{\"Key\": \"region\", \"Operator\": \"Exists\"}]"}}}'

Optionally, an Effect can also be used.

• NoSchedule
  • If the tolerations operator is Equal and the tolerations key=value:effect is NOT an exact match of the nodes key=value:effect taint, the pod cannot be scheduled on the node
  • If the tolerations operator is Exists and the tolerations key and effect is NOT an exact match of the nodes key and effect taint, the pod cannot be scheduled on the node
• NoExecute
  • ​If the tolerations operator is Equal and the tolerations key=value:effect is NOT an exact match of the nodes nodes key=value:effect taint and the pod is running on the tainted node, the pod will be evicted from the node
  • If the tolerations operator is Exists and the tolerations key and effect is NOT an exact match of the nodes key and effect taint and the pod is running on the tainted node, the pod will be evicted from the node
• PreferNoSchedule
  • If the tolerations operator is Equal and the tolerations key=value:effect is NOT an exact match of the nodes key=value:effect taint, the pod will prefer to not be scheduled on the node
  • If the tolerations operator is Exists and the tolerations key and effect is NOT an exact match of the nodes key and effect taint, the pod will prefer to not be scheduled on the node

For example, to patch a namespace with a default Equal toleration include an Effect. In this example, pods in the namespace can be scheduled on nodes with taint region=east:NoSchedule.

oc patch namespace my-project --patch '{"metadata":{"annotations":{"scheduler.alpha.kubernetes.io/defaultTolerations": "[{\"Key\": \"region\", \"Operator\": \"Equal\", \"Value\": \"east\", \"Effect\": \"NoSchedule\"}]"}}}'

For example, to patch a namespace with a default Exsts toleration include an Effect. In this example, pods in the namespace can be scheduled on nodes with taint key region and NoSchedule.

oc patch namespace my-project --patch '{"metadata":{"annotations":{"scheduler.alpha.kubernetes.io/defaultTolerations": "[{\"Key\": \"region\", \"Operator\": \"Exists\", \"Effect\": \"NoSchedule\"}]"}}}'

The oc get namespace command can be used to display the namespace annotations.

~]$ oc get namespace my-project --output yaml
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/defaultTolerations: '[{Key: region, Operator: Equal,
      Value: east, Effect: NoSchedule}]'

deployment toleration

The operator can be Equal or Exists.

If Equal is used, the key (region in this example) and value (east in this example) must be an exact match of the node taint key=value for the pod to "tolerate the taint". In this example, pods in the namespace can be scheduled on nodes with taint region=east.

spec:
  template:
    spec:
      tolerations:
      - key: region
        value: east
        operator: Equal

If the node taint includes an Effect (NoSchedule, NoExecute, PreferNoSchedule) then the deployment tolerations key=value:effect must be an exact match of the nodes key=value:effect taint for the pod to be allowed to be scheduled on the node. In this example, pods in the namespace can be scheduled on nodes with taint region=east:NoSchedule.

spec:
  template:
    spec:
      tolerations:
      - key: region
        value: east
        operator: Equal
        effect: NoSchedule

If Exists is used, the deployment key (region in this example) must be an exact match of the node taint key for the pod to "tolerate the taint". In this example, pods in the namespace can be scheduled on nodes with taint region.

spec:
  template:
    spec:
      tolerations:
      - key: region
        operator: Exists

If the node taint includes an Effect (NoSchedule, NoExecute, PreferNoSchedule) then the deployment tolerations key:effect must be an exact match of the nodes key:effect taint for the pod to be allowed to be scheduled on the node. In this example, pods in the namespace can be scheduled on nodes with taint region:NoSchedule.

spec:
  template:
    spec:
      tolerations:
      - key: region
        operator: Exists
        effect: NoSchedule

Here is an example of how to update a deployment YAML using the oc patch command.

oc patch deployment my-deployment --patch '{"spec":{"template":{"spec":{"tolerations":[{"key":"region","value":"east","operator":"Equal","effect":"NoSchedule"}]}}}}'

Remove taint from a node

Here is how you would remove a taint from a node, replacing "region" with whatever key the node has been tainted with.

~]$ oc adm taint node my-node-worker-5n4fj region-
node/my-node-worker-5n4fj untainted

~]$ oc adm taint node my-node-worker-v8r9r region-
node/my-node-worker-v8r9r untainted