Logging in OpenShift is separated into different systems and services that serve a specific purpose.
- Collecting log data - typically done with filebeat, fluentd, logstash or vector
- Storing log data, for example, in an Amazon Web Services (AWS) S3 Bucket - typically done with Loki or Elastic Search
- Visualizing and query log data - typically done in the OpenShift console or in Kibana
There are abbreviations used as shorthand for the collection of systems and services to collect, store and visualize log data.
- EFK (Elastic Search, Fluentd, Kibana)
- ELK (Elastic Search, Logstash, Kibana)
- EVK (Elastic Search, Vector, Kibana)
- LFK (Loki, Fluentd, Kibana)
- LLK (Loki, Logstash, Kibana)
- LVK (Loki, Vector, Kibana)
The first step in configuring OpenShift to collect log data from various different types of resources in your OpenShift cluster, such as nodes, pods, and so on is to install an Operator that will collect the log data. This is often done by installing the Cluster Logging Operator. Check out my article OpenShift - Getting Started with Cluster Logging. The oc get operators command can be used to display the Operators you have installed.
~]$ oc get operators
NAME AGE
cluster-logging.openshift-logging 2y13d
Next you will probably install an Operator that will be used to store the log data, for example, in object storage such as an Amazon Web Services (AWS) S3 Bucket.
The oc get operators command can be used to display the Operators you have installed.
~]$ oc get operators
NAME AGE
cluster-logging.openshift-logging 2y13d
elasticsearch-operator.openshift-operators-redhat 2y13d
loki-operator.openshift-operators-redhat 320d
The Cluster Logging Custom Resource (CR) is used to define the system that will collect log data (fluentd in this example), where the log data will be stored (Loki in this example), and where the log data can be visualized and queried (the OpenShift console in this example).
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
name: instance
namespace: openshift-logging
spec:
collection:
type: fluentd
logStore:
lokistack:
name: logging-loki
type: lokistack
visualization:
type: ocp-console
ocpConsole:
logsLimit: 15
managementState: Managed
The oc get pods command can be used list the Fluentd pods in the openshift-logging project.
~]# oc get pods --namespace openshift-logging
NAME READY STATUS RESTARTS AGE
cluster-logging-operator-7f65964859-gtlvv 1/1 Running 0 50d
curator-1622518200-qpnfc 0/1 Error 0 9d
curator-1623295800-h658p 0/1 Completed 0 22h
elasticsearch-cdm-dm8dl5ki-1-5d4d54988f-qzz4x 2/2 Running 0 50d
elasticsearch-cdm-dm8dl5ki-2-674f9db4c6-k4r2x 2/2 Running 0 37d
elasticsearch-cdm-dm8dl5ki-3-7d55fbfbff-8ssnk 2/2 Running 0 50d
elasticsearch-im-app-1623377700-2xnxv 0/1 Completed 0 7m57s
elasticsearch-im-audit-1623377700-bsrcl 0/1 Completed 0 7m57s
elasticsearch-im-infra-1623377700-5ltdm 0/1 Completed 0 7m57s
fluentd-22pbq 1/1 Running 0 50d
fluentd-44v9v 1/1 Running 0 50d
fluentd-6lpwh 1/1 Running 0 50d
fluentd-89xsl 1/1 Running 0 50d
fluentd-995zv 1/1 Running 0 50d
fluentd-b5vj7 1/1 Running 0 50d
fluentd-bc4zg 1/1 Running 0 50d
fluentd-br7ft 1/1 Running 0 50d
fluentd-cmnqr 1/1 Running 0 50d
fluentd-gc6zv 1/1 Running 0 50d
fluentd-gl68p 1/1 Running 0 50d
fluentd-gplgt 1/1 Running 0 50d
fluentd-kbvx8 1/1 Running 0 50d
fluentd-kgzvm 1/1 Running 0 50d
fluentd-kzpjk 1/1 Running 0 50d
fluentd-nbm9v 1/1 Running 0 50d
fluentd-pd287 1/1 Running 0 50d
fluentd-rml9r 1/1 Running 0 50d
fluentd-vj7mw 1/1 Running 0 50d
fluentd-vp5jq 1/1 Running 0 50d
fluentd-x5j5g 1/1 Running 1 50d
fluentd-xl257 1/1 Running 0 50d
fluentd-xpw7s 1/1 Running 0 50d
fluentd-xttg7 1/1 Running 0 50d
fluentd-zdn6j 1/1 Running 0 50d
fluentd-zh2vc 1/1 Running 0 50d
kibana-7b676c4bf8-d9t6w 2/2 Running 0 50d
The oc logs command can be used to view the logs in the fluentd pods. These are not the logs that fluentd collects from your OpenShfit pods.
oc logs fluentd-22pbq -n openshift-logging
Fluentd uses the Linux journal daemon (journald) to collect log data. The following command can be used to list the journal log files. Of course, replace b03724ec5eda4cc3b6f845a1a1d7cd41 with whatever string is displayed.
oc exec fluentd-22pbq -- tail /var/log/journal/b03724ec5eda4cc3b6f845a1a1d7cd41
Something like this should be returned. The cat command could then be used to read each .journal file.
-rw-r-----+ 1 root systemd-journal 58720256 Jun 11 11:58 system.journal
-rw-r-----+ 1 root systemd-journal 134217728 Jun 11 04:09 system@c622def313224131b5b2a9733c20a557-000000001e9f33cd-0005c47575fb8abb.journal
-rw-r-----+ 1 root systemd-journal 134217728 Jun 11 04:24 system@c622def313224131b5b2a9733c20a557-000000001ea1136a-0005c475ac6da6b6.journal
-rw-r-----+ 1 root systemd-journal 134217728 Jun 11 04:38 system@c622def313224131b5b2a9733c20a557-000000001ea2f2f5-0005c475e4028a83.journal
Did you find this article helpful?
If so, consider buying me a coffee over at 