
Let's say you are using the oc adm drain command to evict pods from a node.
oc adm drain worker-hsjrp
And something like this is being returned.
error when evicting pods/"my-pod-gqfs2" -n "my-namespace" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
This means a Pod Disruption Budget is preventing the pod from being evicted from the node. Notice in this example that the pod is in my-namespace and min available is 1 meaning there must always be 1 running pod. The oc get PodDisruptionBudgets command can be used to the the Pod Disruption Budgets in the namespace the pod is in.
~]$ oc get PodDisruptionBudgets --namespace my-namespace
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
my-pod-disruption-budget 1 N/A 0 26s
This often occurs when the deployment / replica set / pod is configured with 1 replica and the Pod Disruption Budget is configured with 1 minimum, meaning there must always be 1 running pod.
~]# oc get deployment my-deployment --output yaml
spec:
replicas: 1
Probably the easiest solution is to update the deployment to have 1 more replica than the Pod Disruption Budget minimum available. For example, if the Pod Disruption Budget is configured with 1 minimum, then configure the deployment to have 2 (or more) replicas.
~]# oc get deployment my-deployment --output yaml
spec:
replicas: 2
Which should spawn a second pod running on a different node.
~]$ oc get pods --output wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
my-pod-7vj6h 1/1 Running 0 21s 10.131.1.131 worker-2xqdt <none> <none>
my-pod-gqfs2 1/1 Running 0 24h 10.128.2.67 worker-4nd2v <none> <none>
Now when you drain the node the pod is gracefully evicted from the pod. Cool.
~]$ oc adm drain worker-4nd2v --pod-selector app=my-deployment
node/worker-4nd2v cordoned
evicting pod my-namespace/my-pod-gqfs2
pod/my-pod-gqfs2 evicted
node/worker-4nd2v drained
And a new pod was spawned. Perfect.
~]$ oc get pods --output wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
my-pod-42zq4 1/1 Running 0 32s 10.131.1.132 lab001-worker-2xqdt <none> <none>
my-pod-7vj6h 1/1 Running 0 4m 10.131.1.131 lab001-worker-2xqdt <none> <none>
Did you find this article helpful?
If so, consider buying me a coffee over at