OpenShift - List Alert Manager Alerts using amtool
by
Jeremy Canfield |
Updated: January 22 2024
| OpenShift articles
If you are not familiar with the oc command, refer to OpenShift - Getting Started with the oc command.
In the openshift-monitoring namespace, there should be one or more alertmanager pods.
~]$ oc get pods --namespace openshift-monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 6/6 Running 0 6d10h
alertmanager-main-1 6/6 Running 1 (6d10h ago) 6d10h
The oc exec command can be used to run the amtool CLI in one of the alertmanager pods. The amtool alert command can be used to list the active alerts.
~]$ oc exec pod/alertmanager-main-0 --namespace openshift-monitoring -- amtool --alertmanager.url="http://localhost:9093" alert
Alertname Starts At Summary State
Watchdog 2024-01-12 23:12:17 UTC An alert that should always be firing to certify that Alertmanager is working properly. active
UpdateAvailable 2024-01-12 23:12:23 UTC Your upstream update recommendation service recommends you update your cluster. active
KubeJobFailed 2024-01-12 23:13:36 UTC Job failed to complete. active
KubeJobFailed 2024-01-12 23:13:36 UTC Job failed to complete. active
KubeJobFailed 2024-01-12 23:13:36 UTC Job failed to complete. active
KubeJobFailed 2024-01-12 23:13:36 UTC Job failed to complete. active
ClusterNotUpgradeable 2024-01-12 23:13:42 UTC One or more cluster operators have been blocking minor version cluster upgrades for at least an hour. active
SimpleContentAccessNotAvailable 2024-01-12 23:17:41 UTC Simple content access certificates are not available. active
PodStartupStorageOperationsFailing 2024-01-12 23:17:41 UTC Pods can't start because volume_mount of volume plugin kubernetes.io/secret is permanently failing on node stg001-infra-wzwzn. active
InsightsRecommendationActive 2024-01-12 23:19:11 UTC An Insights recommendation is active for this cluster. active
InsightsRecommendationActive 2024-01-12 23:19:11 UTC An Insights recommendation is active for this cluster. active
InsightsRecommendationActive 2024-01-12 23:19:11 UTC An Insights recommendation is active for this cluster. active
MachineWithNoRunningPhase 2024-01-16 17:51:01 UTC machine stg001-datadog-worker-wwlbv is in phase: active
MachineWithoutValidNode 2024-01-16 17:51:17 UTC machine stg001-datadog-worker-wwlbv does not have valid node reference active
APIRemovedInNextReleaseInUse 2024-01-17 14:32:06 UTC Deprecated API that will be removed in the next version is being used. active
APIRemovedInNextEUSReleaseInUse 2024-01-17 14:32:06 UTC Deprecated API that will be removed in the next EUS version is being used. active
query can be used to only display alerts matching an exact alertname.
alertmanager.url="http://localhost:9093" alert query alertname="watchdog"
Alertname Starts At Summary State
Watchdog 2024-01-12 23:12:17 UTC An alert that should always be firing to certify that Alertmanager is working properly. active
The -o or --output extended option can be used to display additional output.
]$ oc exec pod/alertmanager-main-0 --namespace openshift-monitoring -- amtool --alertmanager.url="http://localhost:9093" alert query alertname="FluentdNodeDown" --output extended
Labels Annotations Starts At Ends At Generator URL State
alertname="FluentdNodeDown" container="collector" endpoint="metrics" instance="10.11.12.13:24231" job="collector" namespace="openshift-logging" openshiftCluster="foo.openshift.example.com" openshift_io_alert_source="platform" pod="collector-5zw9c" prometheus="openshift-monitoring/k8s" service="collector" severity="critical" message="Prometheus could not scrape fluentd collector for more than 10m." summary="Fluentd cannot be scraped" 2024-01-18 22:29:09 UTC 2024-01-23 05:24:09 UTC https://console-openshift-console.apps.foo.openshift.example.com/monitoring/graph?g0.expr=up%7Bcontainer%3D%22collector%22%2Cjob%3D%22collector%22%7D+%3D%3D+0+or+absent%28up%7Bcontainer%3D%22collector%22%2Cjob%3D%22collector%22%7D%29+%3D%3D+1&g0.tab=1 active
The amtool alert command with the --silenced flag can be used to list silenced alerts.
~]$ oc exec pod/alertmanager-main-0 --namespace openshift-monitoring -- amtool --alertmanager.url="http://localhost:9093" alert --silenced
ID Matchers Ends At Created By Comment
d86e8aa8-91f3-463d-a34a-daf530682f38 alertname="FluentdNodeDown" 2024-01-20 09:29:29 UTC john.doe temporarily silencing this alert for 1 day
Did you find this article helpful?
If so, consider buying me a coffee over at