This diagram illustrates the systems that are typically used to parse log data on OpenShift. Fluentd collects the log data from the containers and passes the log data onto Elastic Search. Optionally, Kibana can be used as a tool that may make it easier to visualize the logs.

This is similar to the ELK stack (Elastic Search, Logstash, Kibana), but would actually be the EFK stack (Elastic Search, Fluentd, Kibana).

Fluentd and Elastic Search are being superceded by Vector and Loki.

You configure logging by first installing the Loki Operator or OpenShift Elasticsearch Operator to manage your log storage followed by the OpenShift Logging Operator to manage the components of logging. The oc get operators command can be used to list the installed Operators. In this example, the Loki and Elasticsearch and OpenShift Cluster Logging Operators have already been installed.

~]$ oc get operators
NAME                                                              AGE
cluster-logging.openshift-logging                                 604d
elasticsearch-operator.openshift-operators-redhat                 604d
loki-operator.openshift-operators-redhat                          181d

The oc get pods command can be used list the Fluentd pods in the openshift-logging project.

~]# oc get pods --namespace openshift-logging
NAME                                            READY   STATUS      RESTARTS   AGE
cluster-logging-operator-7f65964859-gtlvv       1/1     Running     0          50d
curator-1622518200-qpnfc                        0/1     Error       0          9d
curator-1623295800-h658p                        0/1     Completed   0          22h
elasticsearch-cdm-dm8dl5ki-1-5d4d54988f-qzz4x   2/2     Running     0          50d
elasticsearch-cdm-dm8dl5ki-2-674f9db4c6-k4r2x   2/2     Running     0          37d
elasticsearch-cdm-dm8dl5ki-3-7d55fbfbff-8ssnk   2/2     Running     0          50d
elasticsearch-im-app-1623377700-2xnxv           0/1     Completed   0          7m57s
elasticsearch-im-audit-1623377700-bsrcl         0/1     Completed   0          7m57s
elasticsearch-im-infra-1623377700-5ltdm         0/1     Completed   0          7m57s
fluentd-22pbq                                   1/1     Running     0          50d
fluentd-44v9v                                   1/1     Running     0          50d
fluentd-6lpwh                                   1/1     Running     0          50d
fluentd-89xsl                                   1/1     Running     0          50d
fluentd-995zv                                   1/1     Running     0          50d
fluentd-b5vj7                                   1/1     Running     0          50d
fluentd-bc4zg                                   1/1     Running     0          50d
fluentd-br7ft                                   1/1     Running     0          50d
fluentd-cmnqr                                   1/1     Running     0          50d
fluentd-gc6zv                                   1/1     Running     0          50d
fluentd-gl68p                                   1/1     Running     0          50d
fluentd-gplgt                                   1/1     Running     0          50d
fluentd-kbvx8                                   1/1     Running     0          50d
fluentd-kgzvm                                   1/1     Running     0          50d
fluentd-kzpjk                                   1/1     Running     0          50d
fluentd-nbm9v                                   1/1     Running     0          50d
fluentd-pd287                                   1/1     Running     0          50d
fluentd-rml9r                                   1/1     Running     0          50d
fluentd-vj7mw                                   1/1     Running     0          50d
fluentd-vp5jq                                   1/1     Running     0          50d
fluentd-x5j5g                                   1/1     Running     1          50d
fluentd-xl257                                   1/1     Running     0          50d
fluentd-xpw7s                                   1/1     Running     0          50d
fluentd-xttg7                                   1/1     Running     0          50d
fluentd-zdn6j                                   1/1     Running     0          50d
fluentd-zh2vc                                   1/1     Running     0          50d
kibana-7b676c4bf8-d9t6w                         2/2     Running     0          50d

The oc logs command can be used to view the logs in the fluentd pods. These are not the logs that fluentd collects from your OpenShfit pods.

oc logs fluentd-22pbq -n openshift-logging

Fluentd uses the Linux journal daemon (journald) to collect log data. The following command can be used to list the journal log files. Of course, replace b03724ec5eda4cc3b6f845a1a1d7cd41 with whatever string is displayed.

oc exec fluentd-22pbq -- tail /var/log/journal/b03724ec5eda4cc3b6f845a1a1d7cd41

Something like this should be returned. The cat command could then be used to read each .journal file.

-rw-r-----+ 1 root systemd-journal  58720256 Jun 11 11:58 system.journal
-rw-r-----+ 1 root systemd-journal 134217728 Jun 11 04:09 system@c622def313224131b5b2a9733c20a557-000000001e9f33cd-0005c47575fb8abb.journal
-rw-r-----+ 1 root systemd-journal 134217728 Jun 11 04:24 system@c622def313224131b5b2a9733c20a557-000000001ea1136a-0005c475ac6da6b6.journal
-rw-r-----+ 1 root systemd-journal 134217728 Jun 11 04:38 system@c622def313224131b5b2a9733c20a557-000000001ea2f2f5-0005c475e4028a83.journal