This diagram illustrates the systems that are typically used to parse log data on OpenShift. Fluentd collects the log data from the containers and passes the log data onto Elastic Search. Optionally, Kibana can be used as a tool that may make it easier to visualize the logs.

This is similar to the ELK stack (Elastic Search, Logstash, Kibana), but would actually be the EFK stack (Elastic Search, Fluentd, Kibana).

If you are not familiar with the oc command, refer to OpenShift - Getting Started with the oc command.

The es_util command or cURL can be used to do something in an Elastic Search pod

The oc get pods command can be used to list the Elastic Search pods in the openshift-logging project.

~]$ oc get pods --namespace openshift-logging
NAME                                            READY   STATUS      RESTARTS   AGE
elasticsearch-cdm-11c6ubcb-1-6888c6fcb9-frlxz   2/2     Running     0          5d5h
elasticsearch-cdm-11c6ubcb-2-6bddf77998-9m28k   2/2     Running     0          5d4h
elasticsearch-cdm-11c6ubcb-3-5955f88f5-mm9c2    2/2     Running     0          5d4h
elasticsearch-im-app-28342275-dkg7m             0/1     Completed   0          4m38s
elasticsearch-im-audit-28342275-nldkt           0/1     Completed   0          4m38s
elasticsearch-im-infra-28342275-b624d           0/1     Completed   0          4m38s

The oc exec and es_util commands can be used to list the shards.

oc exec elasticsearch-cdm-11c6ubcb-1-6888c6fcb9-frlxz --container elasticsearch --namespace openshift-logging -- es_util --query=_cat/shards

Or like this, using curl.

oc exec elasticsearch-cdm-mrpf7eom-3-566bd5f5cb-lkdz9 --container elasticsearch --namespace openshift-logging -- curl -tls1.2 --silent --insecure --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key --request GET --url https://localhost:9200/_cat/shards

Something like this should be returned.

.kibana_-904978102_johndoe_1     0 r STARTED       2  20.9kb 10.128.7.132 elasticsearch-cdm-11c6ubcb-3
.kibana_-904978102_janedoe_1     0 p STARTED       2  20.9kb 10.130.4.248 elasticsearch-cdm-11c6ubcb-2

audit-002813                     2 p STARTED          0    261b 10.130.4.248 elasticsearch-cdm-11c6ubcb-2
audit-002813                     2 r STARTED          0    261b 10.129.6.157 elasticsearch-cdm-11c6ubcb-1
audit-002813                     1 p STARTED          0    261b 10.128.7.132 elasticsearch-cdm-11c6ubcb-3
audit-002813                     1 r STARTED          0    261b 10.129.6.157 elasticsearch-cdm-11c6ubcb-1
audit-002813                     0 r STARTED          0    261b 10.128.7.132 elasticsearch-cdm-11c6ubcb-3
audit-002813                     0 p STARTED          0    261b 10.130.4.248 elasticsearch-cdm-11c6ubcb-2

infra-019981                     2 p STARTED     852275     2gb 10.130.4.248 elasticsearch-cdm-11c6ubcb-2
infra-019981                     2 r STARTED     852275     2gb 10.129.6.157 elasticsearch-cdm-11c6ubcb-1
infra-019981                     1 p STARTED     851826     2gb 10.128.7.132 elasticsearch-cdm-11c6ubcb-3
infra-019981                     1 r STARTED     851826     2gb 10.129.6.157 elasticsearch-cdm-11c6ubcb-1
infra-019981                     0 r STARTED     851471     2gb 10.128.7.132 elasticsearch-cdm-11c6ubcb-3
infra-019981                     0 p STARTED     851471     2gb 10.130.4.248 elasticsearch-cdm-11c6ubcb-2
infra-019990                     2 p STARTED     823888   1.9gb 10.128.7.132 elasticsearch-cdm-11c6ubcb-3
infra-019990                     2 r STARTED     823888   1.9gb 10.130.4.248 elasticsearch-cdm-11c6ubcb-2
infra-019990                     1 r STARTED     825464   1.9gb 10.128.7.132 elasticsearch-cdm-11c6ubcb-3
infra-019990                     1 p STARTED     825464   1.9gb 10.129.6.157 elasticsearch-cdm-11c6ubcb-1
infra-019990                     0 r STARTED     824400   1.9gb 10.130.4.248 elasticsearch-cdm-11c6ubcb-2
infra-019990                     0 p STARTED     824400   1.9gb 10.129.6.157 elasticsearch-cdm-11c6ubcb-3

app-019993                       2 r STARTED    4794499   7.5gb 10.128.7.132 elasticsearch-cdm-11c6ubcb-3
app-019993                       2 p STARTED    4794499   7.5gb 10.130.4.248 elasticsearch-cdm-11c6ubcb-2
app-019993                       1 p STARTED    4794299   7.5gb 10.128.7.132 elasticsearch-cdm-11c6ubcb-3
app-019993                       1 r STARTED    4794299   7.5gb 10.129.6.157 elasticsearch-cdm-11c6ubcb-1
app-019993                       0 r RELOCATING 4794026   7.5gb 10.128.7.132 elasticsearch-cdm-11c6ubcb-3 -> 10.129.6.157 Yg84vLgMTQy5uZlxuXJDuQ elasticsearch-cdm-11c6ubcb-1
app-019993                       0 p STARTED    4794026   7.5gb 10.130.4.248 elasticsearch-cdm-11c6ubcb-2