Logging in OpenShift is separated into different systems and services that serve a specific purpose.

• Collecting log data - typically done with filebeat, fluentd, logstash or vector
• Storing log data, for example, in an Amazon Web Services (AWS) S3 Bucket - typically done with Loki or Elastic Search
• Visualizing and query log data - typically done in the OpenShift console or in Kibana

There are abbreviations used as shorthand for the collection of systems and services to collect, store and visualize log data.

• EFK (Elastic Search, Fluentd, Kibana)
• ELK (Elastic Search, Logstash, Kibana)
• EVK (Elastic Search, Vector, Kibana)
• LFK (Loki, Fluentd, Kibana)
• LLK (Loki, Logstash, Kibana)
• LVK (Loki, Vector, Kibana)

The first step in configuring OpenShift to collect log data from various different types of resources in your OpenShift cluster, such as nodes, pods, and so on is to install an Operator that will collect the log data. This is often done by installing the Cluster Logging Operator. Check out my article OpenShift - Getting Started with Cluster Logging.

Then you can install the Elastic Search Operator. This is typically done in the OpenShift console at Operators > Installed Operators. After you have installed the Elastic Search Operator, the oc get pods command can be used list the Elastic Search pods in the openshift-logging project.

oc get pods -n openshift-logging

Something like this should be returned.

NAME                                            READY   STATUS      RESTARTS   AGE
cluster-logging-operator-7f65964859-gtlvv       1/1     Running     0          50d
curator-1622518200-qpnfc                        0/1     Error       0          9d
curator-1623295800-h658p                        0/1     Completed   0          22h
elasticsearch-cdm-dm8dl5ki-1-5d4d54988f-qzz4x   2/2     Running     0          50d
elasticsearch-cdm-dm8dl5ki-2-674f9db4c6-k4r2x   2/2     Running     0          37d
elasticsearch-cdm-dm8dl5ki-3-7d55fbfbff-8ssnk   2/2     Running     0          50d
elasticsearch-im-app-1623377700-2xnxv           0/1     Completed   0          7m57s
elasticsearch-im-audit-1623377700-bsrcl         0/1     Completed   0          7m57s
elasticsearch-im-infra-1623377700-5ltdm         0/1     Completed   0          7m57s
fluentd-22pbq                                   1/1     Running     0          50d
fluentd-44v9v                                   1/1     Running     0          50d
fluentd-6lpwh                                   1/1     Running     0          50d
fluentd-89xsl                                   1/1     Running     0          50d
fluentd-995zv                                   1/1     Running     0          50d
fluentd-b5vj7                                   1/1     Running     0          50d
fluentd-bc4zg                                   1/1     Running     0          50d
fluentd-br7ft                                   1/1     Running     0          50d
fluentd-cmnqr                                   1/1     Running     0          50d
fluentd-gc6zv                                   1/1     Running     0          50d
fluentd-gl68p                                   1/1     Running     0          50d
fluentd-gplgt                                   1/1     Running     0          50d
fluentd-kbvx8                                   1/1     Running     0          50d
fluentd-kgzvm                                   1/1     Running     0          50d
fluentd-kzpjk                                   1/1     Running     0          50d
fluentd-nbm9v                                   1/1     Running     0          50d
fluentd-pd287                                   1/1     Running     0          50d
fluentd-rml9r                                   1/1     Running     0          50d
fluentd-vj7mw                                   1/1     Running     0          50d
fluentd-vp5jq                                   1/1     Running     0          50d
fluentd-x5j5g                                   1/1     Running     1          50d
fluentd-xl257                                   1/1     Running     0          50d
fluentd-xpw7s                                   1/1     Running     0          50d
fluentd-xttg7                                   1/1     Running     0          50d
fluentd-zdn6j                                   1/1     Running     0          50d
fluentd-zh2vc                                   1/1     Running     0          50d
kibana-7b676c4bf8-d9t6w                         2/2     Running     0          50d

The oc logs command can be used to view the logs in the Elastic Search pods.

oc logs elasticsearch-cdm-dm8dl5ki-1-5d4d54988f-qzz4x --container elasticsearch -n openshift-logging

Elastic Search organized the log data from Fluentd into datastores called indices. The following command can be used to list the indices in each Elastic Search pod.

oc exec elasticsearch-cdm-dm8dl5ki-1-5d4d54988f-qzz4x --container elasticsearch -- es_util --query=_cat/indices?v 

Something like this should be returned.

health status index                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_-911233311_c007137_7139  0CTseHnsQFKw6WxzFsKOug   1   1          2            0     28.4kb         14.2kb
green  open   .kibana_-905684497_c067937_67938 554X5_58QfSRSd3YtxufHQ   1   1          1            0      7.4kb          3.7kb
green  open   infra-001509                     Nv1IDefPSImOYnaZxuN45A   3   1   13511117            0     16.6gb          8.3gb
green  open   audit-000195                     V4_7fuCHSZ6Nzh3Do6Ol9w   3   1          0            0      1.5kb           783b
green  open   app-001529                       VGu-dSpJT2mF23ELVg05WQ   3   1     916929            0        1gb        524.8mb

Or the only list the infra indices.

oc exec elasticsearch-cdm-dm8dl5ki-1-5d4d54988f-qzz4x --container elasticsearch -- es_util --query=_cat/indices/infra