Bootstrap FreeKB - OpenShift - List Alert Manager Alerts using amtool
OpenShift - List Alert Manager Alerts using amtool

Updated:   |  OpenShift articles

If you are not familiar with the oc command, refer to OpenShift - Getting Started with the oc command.

In the openshift-monitoring namespace, there should be one or more alertmanager pods.

~]$ oc get pods --namespace openshift-monitoring
NAME                       READY   STATUS    RESTARTS        AGE
alertmanager-main-0        6/6     Running   0               6d10h
alertmanager-main-1        6/6     Running   1 (6d10h ago)   6d10h

 

The oc exec command can be used to run the amtool CLI in one of the alertmanager pods. The amtool alert command can be used to list the active alerts.

~]$ oc exec pod/alertmanager-main-0 --namespace openshift-monitoring -- amtool --alertmanager.url="http://localhost:9093" alert
Alertname                           Starts At                Summary                                                                                                                         State   
Watchdog                            2024-01-12 23:12:17 UTC  An alert that should always be firing to certify that Alertmanager is working properly.                                         active  
UpdateAvailable                     2024-01-12 23:12:23 UTC  Your upstream update recommendation service recommends you update your cluster.                                                 active  
KubeJobFailed                       2024-01-12 23:13:36 UTC  Job failed to complete.                                                                                                         active  
KubeJobFailed                       2024-01-12 23:13:36 UTC  Job failed to complete.                                                                                                         active  
KubeJobFailed                       2024-01-12 23:13:36 UTC  Job failed to complete.                                                                                                         active  
KubeJobFailed                       2024-01-12 23:13:36 UTC  Job failed to complete.                                                                                                         active  
ClusterNotUpgradeable               2024-01-12 23:13:42 UTC  One or more cluster operators have been blocking minor version cluster upgrades for at least an hour.                           active  
SimpleContentAccessNotAvailable     2024-01-12 23:17:41 UTC  Simple content access certificates are not available.                                                                           active  
PodStartupStorageOperationsFailing  2024-01-12 23:17:41 UTC  Pods can't start because volume_mount of volume plugin kubernetes.io/secret is permanently failing on node stg001-infra-wzwzn.  active  
InsightsRecommendationActive        2024-01-12 23:19:11 UTC  An Insights recommendation is active for this cluster.                                                                          active  
InsightsRecommendationActive        2024-01-12 23:19:11 UTC  An Insights recommendation is active for this cluster.                                                                          active  
InsightsRecommendationActive        2024-01-12 23:19:11 UTC  An Insights recommendation is active for this cluster.                                                                          active                  
MachineWithNoRunningPhase           2024-01-16 17:51:01 UTC  machine stg001-datadog-worker-wwlbv is in phase:                            active  
MachineWithoutValidNode             2024-01-16 17:51:17 UTC  machine stg001-datadog-worker-wwlbv does not have valid node reference      active  
APIRemovedInNextReleaseInUse        2024-01-17 14:32:06 UTC  Deprecated API that will be removed in the next version is being used.      active  
APIRemovedInNextEUSReleaseInUse     2024-01-17 14:32:06 UTC  Deprecated API that will be removed in the next EUS version is being used.  active

 

query can be used to only display alerts matching an exact alertname.

alertmanager.url="http://localhost:9093" alert query alertname="watchdog"
Alertname                           Starts At                Summary                                                                                                                         State   
Watchdog                            2024-01-12 23:12:17 UTC  An alert that should always be firing to certify that Alertmanager is working properly.                                         active

 

The -o or --output extended option can be used to display additional output.

]$ oc exec pod/alertmanager-main-0 --namespace openshift-monitoring -- amtool --alertmanager.url="http://localhost:9093" alert query alertname="FluentdNodeDown" --output extended
Labels                                                                                                                                                                                                                                                                                                                                Annotations                                                                                                     Starts At                Ends At                  Generator URL                                                                                                                                                                                                                                             State   
alertname="FluentdNodeDown" container="collector" endpoint="metrics" instance="10.11.12.13:24231" job="collector" namespace="openshift-logging" openshiftCluster="foo.openshift.example.com" openshift_io_alert_source="platform" pod="collector-5zw9c" prometheus="openshift-monitoring/k8s" service="collector" severity="critical"  message="Prometheus could not scrape fluentd collector for more than 10m." summary="Fluentd cannot be scraped"  2024-01-18 22:29:09 UTC  2024-01-23 05:24:09 UTC  https://console-openshift-console.apps.foo.openshift.example.com/monitoring/graph?g0.expr=up%7Bcontainer%3D%22collector%22%2Cjob%3D%22collector%22%7D+%3D%3D+0+or+absent%28up%7Bcontainer%3D%22collector%22%2Cjob%3D%22collector%22%7D%29+%3D%3D+1&g0.tab=1  active

 

The amtool alert command with the --silenced flag can be used to list silenced alerts.

~]$ oc exec pod/alertmanager-main-0 --namespace openshift-monitoring -- amtool --alertmanager.url="http://localhost:9093" alert --silenced
ID                                    Matchers                     Ends At                  Created By  Comment                                     
d86e8aa8-91f3-463d-a34a-daf530682f38  alertname="FluentdNodeDown"  2024-01-20 09:29:29 UTC  john.doe    temporarily silencing this alert for 1 day

 




Did you find this article helpful?

If so, consider buying me a coffee over at Buy Me A Coffee



Comments


Add a Comment


Please enter 5adc53 in the box below so that we can be sure you are a human.