Prometheus and ELK (Elastic Search, Logstash, Kibana) are similar tools used to gather, display, and analyze data. This is sometimes referred to a "scraping metrics". For example, both Prometheus and ELK can be used to display data about servers, virtual machines (VMs), databases, containers (e.g. Docker, OpenShift), messaging (e.g. IBM MQ, RabbitMQ), and the list goes on.
Often, Prometheus or ELK send alerts to some other system, such as Alert Manager and then Alert Manager would route alerts to certain targets, such as an SMTP email server or OpsGenie.
The oc get pods command can be used to list the Prometheus pods, often in the openshift-monitoring namespace.
~]$ oc get pods --namespace openshift-monitoring
NAME READY STATUS RESTARTS AGE
prometheus-adapter-6b98c646c7-m4g76 1/1 Running 0 8d
prometheus-adapter-6b98c646c7-tczr2 1/1 Running 0 8d
prometheus-k8s-0 6/6 Running 0 11d
prometheus-k8s-1 6/6 Running 0 11d
prometheus-operator-6766f68555-mkfv9 2/2 Running 0 11d
prometheus-operator-admission-webhook-8589888cbc-mq2jx 1/1 Running 0 11d
prometheus-operator-admission-webhook-8589888cbc-t62mt 1/1 Running 0 11d
The oc get pods command can be used to list the ELK collector pods in the openshift-logging namespace. The --output wide option is used to include the pods IP address.
~]$ oc get pods --namespace openshift-logging --output wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
collector-26tk2 2/2 Running 0 33h 10.129.4.2 stg001-datadog-worker-95xbq <none> <none>
collector-2mjhm 2/2 Running 0 6d4h 10.129.17.50 stg001-worker-dwkfh <none> <none>
collector-2qs52 2/2 Running 0 6d4h 10.129.27.177 stg001-worker-4b84w <none> <none>
collector-45gzj 2/2 Running 0 6d4h 10.130.0.168 stg001-master-0 <none> <none>
collector-52z9t 2/2 Running 0 6d4h 10.130.7.38 stg001-edge-tgsrw <none> <none>
collector-62zrn 2/2 Running 0 6d4h 10.131.17.209 stg001-worker-xp76w <none> <none>
collector-6snss 2/2 Running 0 6d4h 10.128.22.144 stg001-worker-d4rdw <none> <none>
collector-74t62 2/2 Running 0 6d4h 10.128.19.117 stg001-worker-zkrbn <none> <none>
collector-7z5cd 2/2 Running 0 6d4h 10.130.18.86 stg001-worker-jnhl2 <none> <none>
collector-dbdt2 2/2 Running 0 6d4h 10.128.16.196 stg001-worker-bkfph <none> <none>
collector-g27xg 2/2 Running 0 6d4h 10.128.0.194 stg001-master-2 <none> <none>
collector-gs9s4 2/2 Running 0 6d4h 10.128.2.50 stg001-worker-gt4wr <none> <none>
collector-gzqst 2/2 Running 0 6d4h 10.129.21.10 stg001-worker-xxxr2 <none> <none>
collector-h65tv 2/2 Running 0 6d4h 10.130.23.105 stg001-worker-chwbx <none> <none>
collector-h7lwt 2/2 Running 0 6d4h 10.130.20.227 stg001-worker-678fb <none> <none>
collector-hbpgp 2/2 Running 0 6d4h 10.131.24.233 stg001-worker-kf9td <none> <none>
collector-j4svl 2/2 Running 0 6d4h 10.128.27.5 stg001-worker-lbstf <none> <none>
collector-jptz9 2/2 Running 0 6d4h 10.131.20.106 stg001-worker-kgkz7 <none> <none>
collector-lpgx7 2/2 Running 0 6d4h 10.130.27.107 stg001-worker-ntq65 <none> <none>
collector-mprj4 2/2 Running 0 6d4h 10.130.17.176 stg001-worker-rxgwv <none> <none>
collector-nhkvs 2/2 Running 0 6d4h 10.128.6.199 stg001-infra-5h56w <none> <none>
collector-ns9m8 2/2 Running 0 6d4h 10.129.19.108 stg001-worker-msmjk <none> <none>
collector-nw8q4 2/2 Running 0 6d4h 10.129.7.38 stg001-edge-d7mtx <none> <none>
collector-qfpd4 2/2 Running 0 6d4h 10.128.5.210 stg001-infra-wzwzn <none> <none>
collector-r8lpn 2/2 Running 0 6d4h 10.129.0.187 stg001-master-1 <none> <none>
collector-rbs5n 2/2 Running 0 33h 10.130.4.4 stg001-datadog-worker-bsnc9 <none> <none>
collector-rxmpq 2/2 Running 0 33h 10.130.2.2 stg001-datadog-worker-zs7ht <none> <none>
collector-rzndh 2/2 Running 0 6d4h 10.131.22.92 stg001-worker-bv2f5 <none> <none>
collector-tq5l2 2/2 Running 0 6d4h 10.129.23.157 stg001-worker-dlc6w <none> <none>
collector-tqxpc 2/2 Running 0 6d4h 10.131.3.193 stg001-infra-7jb2f <none> <none>
collector-v4sn2 2/2 Running 0 6d4h 10.131.18.118 stg001-worker-qfm8f <none> <none>
collector-vbxxv 2/2 Running 0 6d4h 10.128.21.125 stg001-worker-rznsf <none> <none>
collector-wrf8r 2/2 Running 0 6d4h 10.131.14.170 stg001-worker-d5c4g <none> <none>
collector-xgw8s 2/2 Running 0 6d4h 10.130.25.161 stg001-worker-xprqc <none> <none>
collector-xnj6x 2/2 Running 0 6d4h 10.131.1.226 stg001-worker-qfbbn <none> <none>
collector-z7kzk 2/2 Running 0 33h 10.131.4.2 stg001-datadog-worker-8qhp4 <none> <none>
collector-z8jcn 2/2 Running 0 6d4h 10.128.24.82 stg001-worker-nbrdd <none> <none>
collector-zsg47 2/2 Running 0 6d4h 10.129.24.115 stg001-worker-xr66b <none> <none>
The oc exec and curl commands can be used to issue a GET request inside of your Prometheus pod to the /metrics endpoint of one of the collector pods running on the datadog worker node. In this example, IP address 10.129.4.2 is the IP of the collector-26tk2 pod.
This will often output a slew of data, too much to parse as stdout, so almost always the output will be redirected to an out file.
oc exec pod/prometheus-k8s-0 --container prometheus --namespace openshift-monitoring -- curl --insecure --request GET --url 'http://10.129.4.2:24231/metrics' | tee --append port_24231_metrics.out
Similarlly, you may also want to get the metrics on port 2112.
oc exec pod/prometheus-k8s-0 --container prometheus --namespace openshift-monitoring -- curl --insecure --request GET --url 'http://10.129.4.2:2112/metrics' | tee --append port_2112_metrics.outDid you find this article helpful?
If so, consider buying me a coffee over at 