OpenShift - Resolve "Disk High Watermark" (pod health status yellow or red)

If you are not familiar with the oc command, refer to OpenShift - Getting Started with the oc command.

The oc get command can be used to determine if the health of a pod is green (healthy), yellow (warning) or red (error). In this example, the health of pod001 is red.

~]# oc get pod001 --all-namespaces
NAMESPACE    NAME     MANAGEMENT STATE   HEALTH   NODES   DATA NODES   SHARD ALLOCATION   INDEX MANAGEMENT
project001   pod001   Managed            red      3       3            all

 

When the health of a pod is yellow or red, use the oc describe pod to view the pod events. One of the more common conditions is Disk High Watermark.

~]# oc describe pod/pod001 -n project001
status:
  cluster:
    activePrimaryShards: 197
    activeShards: 364
    initializingShards: 0
    numDataNodes: 3
    numNodes: 3
    pendingTasks: 0
    relocatingShards: 0
    status: red
    unassignedShards: 30
  nodes:
  - conditions:
    - lastTransitionTime: "2021-03-03T11:26:42Z"
      message: Disk storage usage for node is 166.07Gb (91.07821465128325%). Shards will be relocated from this node.
      reason: Disk Watermark High
      status: "True"
      type: NodeStorage
    deploymentName: pod001
    upgradeStatus:
      upgradePhase: controllerUpdated

 

Use the oc exec and df command to determine if any of the volumes are running low on available disk space. In this example, the /elasticsearch/persistent volume has exceeded 90% capacity.

~]# oc exec pod001 df
Filesystem                           1K-blocks     Used Available Use% Mounted on
/dev/sda1                               372607   170989    177862  50% /boot
/dev/sdb                                129774     6994    122780  91% /elasticsearch/persistent

 

In this scenario, you will want to list the "infra" indices.

~]# oc exec pod001 es_util --query=_cat/indices?v | grep infra
green  open  infra-000194
green  open  infra-000195
green  open  infra-000196

 

Delete the oldest indices.

~]# oc exec pod001 es_util --query=infra-000194 -XDELETE | grep infra

 




Did you find this article helpful?

If so, consider buying me a coffee over at Buy Me A Coffee

Add a Comment




We will never share your name or email with anyone. Enter your email if you would like to be notified when we respond to your comment.





Please enter 9ab67 in the box below so that we can be sure you are a human.




Comments

Web design by yours truely - me, myself, and I   |   jeremy.canfield@freekb.net   |