This diagram illustrates the systems that are typically used to parse log data on OpenShift. Fluentd collects the log data from the containers and passes the log data onto Elastic Search. Optionally, Kibana can be used as a tool that may make it easier to visualize the logs.
This is similar to the ELK stack (Elastic Search, Logstash, Kibana), but would actually be the EFK stack (Elastic Search, Fluentd, Kibana).
The oc get pods command can be used list the Elastic Search pods in the openshift-logging project.
oc get pods -n openshift-logging
Something like this should be returned.
NAME READY STATUS RESTARTS AGE
cluster-logging-operator-7f65964859-gtlvv 1/1 Running 0 50d
curator-1622518200-qpnfc 0/1 Error 0 9d
curator-1623295800-h658p 0/1 Completed 0 22h
elasticsearch-cdm-dm8dl5ki-1-5d4d54988f-qzz4x 2/2 Running 0 50d
elasticsearch-cdm-dm8dl5ki-2-674f9db4c6-k4r2x 2/2 Running 0 37d
elasticsearch-cdm-dm8dl5ki-3-7d55fbfbff-8ssnk 2/2 Running 0 50d
elasticsearch-im-app-1623377700-2xnxv 0/1 Completed 0 7m57s
elasticsearch-im-audit-1623377700-bsrcl 0/1 Completed 0 7m57s
elasticsearch-im-infra-1623377700-5ltdm 0/1 Completed 0 7m57s
fluentd-22pbq 1/1 Running 0 50d
fluentd-44v9v 1/1 Running 0 50d
fluentd-6lpwh 1/1 Running 0 50d
fluentd-89xsl 1/1 Running 0 50d
fluentd-995zv 1/1 Running 0 50d
fluentd-b5vj7 1/1 Running 0 50d
fluentd-bc4zg 1/1 Running 0 50d
fluentd-br7ft 1/1 Running 0 50d
fluentd-cmnqr 1/1 Running 0 50d
fluentd-gc6zv 1/1 Running 0 50d
fluentd-gl68p 1/1 Running 0 50d
fluentd-gplgt 1/1 Running 0 50d
fluentd-kbvx8 1/1 Running 0 50d
fluentd-kgzvm 1/1 Running 0 50d
fluentd-kzpjk 1/1 Running 0 50d
fluentd-nbm9v 1/1 Running 0 50d
fluentd-pd287 1/1 Running 0 50d
fluentd-rml9r 1/1 Running 0 50d
fluentd-vj7mw 1/1 Running 0 50d
fluentd-vp5jq 1/1 Running 0 50d
fluentd-x5j5g 1/1 Running 1 50d
fluentd-xl257 1/1 Running 0 50d
fluentd-xpw7s 1/1 Running 0 50d
fluentd-xttg7 1/1 Running 0 50d
fluentd-zdn6j 1/1 Running 0 50d
fluentd-zh2vc 1/1 Running 0 50d
kibana-7b676c4bf8-d9t6w 2/2 Running 0 50d
The oc logs command can be used to view the logs in the Elastic Search pods.
oc logs elasticsearch-cdm-dm8dl5ki-1-5d4d54988f-qzz4x --container elasticsearch -n openshift-logging
Elastic Search organized the log data from Fluentd into datastores called indices. The following command can be used to list the indices in each Elastic Search pod.
oc exec elasticsearch-cdm-dm8dl5ki-1-5d4d54988f-qzz4x --container elasticsearch -- es_util --query=_cat/indices?v
Something like this should be returned.
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana_-911233311_c007137_7139 0CTseHnsQFKw6WxzFsKOug 1 1 2 0 28.4kb 14.2kb
green open .kibana_-905684497_c067937_67938 554X5_58QfSRSd3YtxufHQ 1 1 1 0 7.4kb 3.7kb
green open infra-001509 Nv1IDefPSImOYnaZxuN45A 3 1 13511117 0 16.6gb 8.3gb
green open audit-000195 V4_7fuCHSZ6Nzh3Do6Ol9w 3 1 0 0 1.5kb 783b
green open app-001529 VGu-dSpJT2mF23ELVg05WQ 3 1 916929 0 1gb 524.8mb
Or the only list the infra indices.
oc exec elasticsearch-cdm-dm8dl5ki-1-5d4d54988f-qzz4x --container elasticsearch -- es_util --query=_cat/indices/infra
Did you find this article helpful?
If so, consider buying me a coffee over at