Let's say something like this is being returned.

KubeletDown
Kubelet has disappeared from Prometheus target discovery
Target disappeared from Prometheus target discovery

In the openshift-monitoring namespace, there should be a prometheus rulesfile configmap.

~]$ oc get configmaps --namespace openshift-monitoring
NAME                                             DATA   AGE
prometheus-k8s-rulefiles-0                       46     96d

And I was able to see this alert configuration is in a Prometheus rules file configmap.

~]$ oc describe configmap prometheus-custom-rulefiles-0 --namespace openshift-monitoring
  - alert: KubeletDown
    annotations:
      description: Kubelet has disappeared from Prometheus target discovery.
      runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/KubeletDown.md
      summary: Target disappeared from Prometheus target discovery.
    expr: |
      absent(up{job="kubelet", metrics_path="/metrics"} == 1)
    for: 15m
    labels:
      namespace: kube-system
      severity: critical

I've saw this twice. Here is the cause for each occurrence of this I saw.

• Rotated Certificate was not picked up by certain resources (Prometheus-custom and alertmanager-custom)
• Persistent Volumes for the clusters Prometheus instances reached 100% capacity thus Prometheus was not able to pull in metrics from the cluster nodes to evaluate

Persistent Volumes for the clusters Prometheus instances reached 100% capacity

The oc get persistentvolumes command can be used to list the Persistent Volumes. Notice there are two Prometheus Persistent Volumes and two Alert Manager Persistent Volumes.

~]$ oc get persistentvolumes
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                          STORAGECLASS   REASON   AGE
pvc-53b2f9a5-1cc3-4945-a619-00d82490055a   300Gi      RWO            Delete           Bound    openshift-monitoring/prometheusk8s-db-prometheus-k8s-0         gp3-csi                 254d
pvc-9f311f27-5b80-4afb-8e39-be05b88bc309   10Gi       RWO            Delete           Bound    openshift-monitoring/alertmanagermain-db-alertmanager-main-0   gp3-csi                 254d
pvc-cf6a37aa-c619-4e1b-95e7-1449928401a6   10Gi       RWO            Delete           Bound    openshift-monitoring/alertmanagermain-db-alertmanager-main-1   gp3-csi                 254d
pvc-d2c305c6-847c-4b76-bdf1-e0015ac40b84   300Gi      RWO            Delete           Bound    openshift-monitoring/prometheusk8s-db-prometheus-k8s-1         gp3-csi                 254d

The oc get persistentvolumeclaims command can be used to list the Persistent Volume Claims associated with the Persistent Volumes.

~]$ oc get pvc --all-namespaces
NAMESPACE                     NAME                                      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
openshift-monitoring          alertmanagermain-db-alertmanager-main-0   Bound     pvc-9f311f27-5b80-4afb-8e39-be05b88bc309   10Gi       RWO            gp3-csi        254d
openshift-monitoring          alertmanagermain-db-alertmanager-main-1   Bound     pvc-cf6a37aa-c619-4e1b-95e7-1449928401a6   10Gi       RWO            gp3-csi        254d
openshift-monitoring          prometheusk8s-db-prometheus-k8s-0         Bound     pvc-53b2f9a5-1cc3-4945-a619-00d82490055a   300Gi      RWO            gp3-csi        254d
openshift-monitoring          prometheusk8s-db-prometheus-k8s-1         Bound     pvc-d2c305c6-847c-4b76-bdf1-e0015ac40b84   300Gi      RWO            gp3-csi        254d

Notice the Prometheus and Alert Manager Persistent Volume Claims are in the openshift-monitoring namespace. The oc get events command can be used to see if there are any interesting events in the openshift-monitoring namespace.

oc get events --namespace openshift-monitoring --sort-by='.metadata.creationTimestamp'

Likewise, you can describe the persistent volume cliams to see if there are any interesting events with the Persistent Volume Claim.

oc describe persistentvolumeclaim prometheusk8s-db-prometheus-k8s-0 --namespace openshift-monitoring

Rotated Certificate was not picked up by certain resources

In the openshift-monitoring namespace there should be statefulsets.

~]$ oc get statefulsets --namespace openshift-monitoring
NAME                READY   AGE
alertmanager-main   2/2     2y105d
prometheus-k8s      2/2     2y105d

And I was able to see a stateful set was configured to use the configmap.

~]$ oc get statefulset prometheus-k8s --output yaml --namespace openshift-monitoring
spec:
  template:
    spec:
      containers:
        name: prometheus
        volumeMounts:
        - mountPath: /etc/prometheus/rules/prometheus-custom-rulefiles-0
          name: prometheus-custom-rulefiles-0

And the stateful set was managed by the cluster-monitoring-operator.

metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: custom
    app.kubernetes.io/managed-by: cluster-monitoring-operator
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: custom-monitoring
    app.kubernetes.io/version: 2.46.0
    operator.prometheus.io/name: custom
    operator.prometheus.io/shard: "0"

The first time I saw this alert I saw a bunch of "certificate has expired" events in the log, like this.

And these events started near the same time I started getting the KubeletDown alerts.

Apr 19 03:37:50.211885 ip-10-29-86-25 kubenswrapper[1507]: E0419 03:37:50.211852    1507 server.go:299] "Unable to authenticate the request due to an error" err="verifying certificate SN=12345678901212345678901263ABCD073300112, SKID=, AKID=67:3A:A2:57:2C:18:52:21:04:BC:36:3B:3A:F4:3A:C8:1B:B3:01:7B failed: x509: certificate has expired or is not yet valid: current time 2024-04-19T03:37:50Z is after 2024-04-19T01:12:29Z"

And I noticed new "cert" configmaps were created near the same time the alerts started. In this example, the 564 config map is the config map before the issue started and the 565 config map was created at nearly the same exact time the issue started.

~]$ oc get configmaps --namespace openshift-kube-apiserver | grep cert
kube-apiserver-cert-syncer-kubeconfig-564   1      7d11h
kube-apiserver-cert-syncer-kubeconfig-565   1      4d

I used the oc get configmap command with the --output jsonpath option to view the content of the ca-bunder.crt file in the 564 config map (the config map before the issue started). I took the content of the ca-bunder.crt file and placed it in a file named ca-bunder.crt and then used the openssl command to show the certficate details. The certificate in the 564 config map was indeed the certificate that expired.

~]$ oc get configmap kubelet-serving-ca-564 --namespace openshift-kube-apiserver --output jsonpath="{.data.ca-bundle\.crt}" | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4867864711386769100 (0x438e2129f8de26cc)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=openshift-kube-controller-manager-operator_csr-signer-signer@1709471471
        Validity
            Not Before: Mar 20 01:12:28 2024 GMT
            Not After : Apr 19 01:12:29 2024 GMT <- expired when the issue started
        Subject: CN=kube-csr-signer_@1710897148
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                67:3A:A2:57:2C:18:52:21:04:BC:36:3B:3A:F4:3A:C8:1B:B3:01:7B <- this matches the AKID in the log

I created a Must Gather and opened a case with Red Hat.

oc adm must-gather

Likewise, the oc get service command can be used to see the kubelet service is in the kube-system namespace.

~]$ oc get service --namespace kube-system
NAME      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                        AGE
kubelet   ClusterIP   None         <none>        10250/TCP,10255/TCP,4194/TCP   2y72d

The oc describe service command can be used to see that the kubelet service forwards reqests on the nodes via HTTP, HTTPS and on port 4194 for the cadvisor (Certificate Authority Advisor).

~]$ oc describe service kubelet --namespace kube-system
Name:              kubelet
Namespace:         kube-system
Labels:            app.kubernetes.io/managed-by=prometheus-operator
                   app.kubernetes.io/name=kubelet
                   k8s-app=kubelet
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  RequireDualStack
IP Families:       IPv4,IPv6
IP:                None
IPs:               None
Port:              https-metrics  10250/TCP
TargetPort:        10250/TCP
Endpoints:         10.11.12.13:10250,10.11.12.13:10250,10.29.149.96:10250 + 6 more...
Port:              http-metrics  10255/TCP
TargetPort:        10255/TCP
Endpoints:         10.29.146.118:10255,10.29.148.190:10255,10.29.149.96:10255 + 6 more...
Port:              cadvisor  4194/TCP
TargetPort:        4194/TCP
Endpoints:         10.29.146.118:4194,10.29.148.190:4194,10.29.149.96:4194 + 6 more...
Session Affinity:  None
Events:            <none>

You can start one of the nodes that the kubelet service forwards requests onto in debug mode. 

~]# oc debug node/my-node-5n4fj
Starting pod/my-node-5n4fj-debug ...
sh-4.4#

Typically you will first issue the chroot /host command is used to set /host as the root directory because the root file system is mounted to /host in the debug pod.

sh-4.4# chroot /host

systemctl can be used to determine if the kubelet service is running.

sh-5.1# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: disabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─01-kubens.conf, 10-mco-default-madv.conf, 20-aws-node-name.conf, 20-aws-providerid.conf, 20-logging.conf
     Active: active (running) since Tue 2024-04-23 04:14:54 UTC; 1 day 1h ago

The output of the systemctl command may contain events like this.

Jul 27 12:04:58 ip-10-29-108-13 kubenswrapper[2779]: E0727 12:04:58.215777    2779 server.go:336] "Unable to authenticate the request due to an error" err="verifying certificate SN=1870853530484269975151209

Here is a one liner that I use to loop through each node and to return the status of the kubelet service.

for node in $(oc get nodes | grep -v ^NAME | awk '{print $1}'); do echo $node; oc debug node/$node -- chroot /host /usr/bin/systemctl status kubelet | grep Active:; done;

Which should return something like this. In this sceanrio, I can see that the kubelet has been running since before the certificate expired, meaning the kubelet service was not restarted after the certificate renewed.

infra-node-1
     Active: active (running) since Thu 2024-04-04 16:31:39 UTC; 2 weeks 5 days ago
infra-node-2
     Active: active (running) since Thu 2024-04-04 16:18:48 UTC; 2 weeks 5 days ago
infra-node-3
     Active: active (running) since Thu 2024-04-04 16:26:03 UTC; 2 weeks 5 days ago

master-node-1
     Active: active (running) since Thu 2024-04-04 16:34:01 UTC; 2 weeks 5 days ago
master-node-2
     Active: active (running) since Thu 2024-04-04 16:21:05 UTC; 2 weeks 5 days ago
master-node-3
     Active: active (running) since Thu 2024-04-04 16:27:32 UTC; 2 weeks 5 days ago

worker-node-1
     Active: active (running) since Tue 2024-04-04 16:34:01 UTC; 1 day 1h ago
worker-node-2
     Active: active (running) since Tue 2024-04-04 16:21:05 UTC; 1 day 1h ago
worker-node-3
     Active: active (running) since Tue 2024-04-04 16:27:32 UTC; 1 day 2h ago

This one liner can be used to restart the kubelet service in each node, if you want to try restarting the kubelet service to see if this resolves the issue.

for node in $(oc get nodes | grep -v ^NAME | awk '{print $1}'); do echo $node; oc debug node/$node -- chroot /host /usr/bin/systemctl restart kubelet; done;

The default directory where for kubelet SSL is /var/lib/kubelet/pki.

sh-5.1# ls -l /var/lib/kubelet/pki
total 32
-rw-------. 1 root root 1183 Feb 29 05:33 kubelet-client-2024-02-29-05-33-41.pem
-rw-------. 1 root root 1183 Mar 15 00:19 kubelet-client-2024-03-15-00-19-13.pem
-rw-------. 1 root root 1183 Mar 30 14:40 kubelet-client-2024-03-30-14-40-24.pem
-rw-------. 1 root root 1183 Apr 13 11:20 kubelet-client-2024-04-13-11-20-57.pem
lrwxrwxrwx. 1 root root   59 Apr 13 11:20 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2024-04-13-11-20-57.pem
-rw-------. 1 root root 1269 Feb 29 05:33 kubelet-server-2024-02-29-05-33-42.pem
-rw-------. 1 root root 1269 Mar 14 12:49 kubelet-server-2024-03-14-12-49-47.pem
-rw-------. 1 root root 1269 Mar 28 21:48 kubelet-server-2024-03-28-21-48-50.pem
-rw-------. 1 root root 1269 Apr 12 22:10 kubelet-server-2024-04-12-22-10-56.pem
lrwxrwxrwx. 1 root root   59 Apr 12 22:10 kubelet-server-current.pem -> /var/lib/kubelet/pki/kubelet-server-2024-04-12-22-10-56.pem

The openssl command can be used to display the details of each PEM. I noticed the prior PEM also had the expired certificate.

sh-5.1# openssl x509 -in /var/lib/kubelet/pki/kubelet-client-2024-03-30-14-40-24.pem -text -noout
Certificate:
    Data:
        Issuer: CN = kube-csr-signer_@1712193161
        Validity
            Not Before: Mar 30 14:35:24 2024 GMT
            Not After : Apr 19 01:12:29 2024 GMT <- expired when the issue started
        Subject: O = system:nodes, CN = system:node:ip-10-29-148-190.us-east-2.compute.internal
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                67:3A:A2:57:2C:18:52:21:04:BC:36:3B:3A:F4:3A:C8:1B:B3:01:7B <- this matches the AKID in the log

The which command should return the absolute path to the kubelet CLI.

sh-5.1# which kubelet
/usr/bin/kubelet

The kubelet --version command can be used to ensure the kubelet CLI is working as expected and returning stdout.

sh-5.1# kubelet --version
Kubernetes v1.26.13+8f85140

The oc adm node-logs command can be used to return the node logs pertaining to kubelet. The logs will be extensive, which is why I redirect the output to a file.

oc adm node-logs --role=master --unit=kubelet >> kubelet_node.log

Certificate signed by unknown authority / machine-config degraded

Let’s start a debug node.

oc debug node/my-node

And chroot /host.

sh-4.4# chroot /host

Let’s use systemctl to look for interesting events with the kubelet systemd service.

systemctl status kubelet | less

We I saw this, where the certificate with serial number 123456789123456789123456789123456789123 and AKID 12:34:56:78:90:AB:CD:EF:GH:IJ:KL:MN:OP:QR:ST:UV:WX:YZ:1A:2B failed validation because it was signed by an unknown authority.

Jul 27 13:10:03 ip-10-29-108-13 kubenswrapper[2779]: E0727 13:10:03.686514    2779 server.go:336] "Unable to authenticate the request due to an error" err="verifying certificate SN=123456789123456789123456789123456789123, SKID=, AKID=12:34:56:78:90:AB:CD:EF:GH:IJ:KL:MN:OP:QR:ST:UV:WX:YZ:1A:2B failed: x509: certificate signed by unknown authority"

The events in the openshift-monitoring namespace also showed that OpenShiftMonitoringMetricsServerClientCertRequester and OpenShiftMonitoringTelemeterClientCertRequester were rotated.

~]$ oc get events --namespace openshift-monitoring --sort-by='.metadata.creationTimestamp'
LAST SEEN   TYPE      REASON                       OBJECT                                   MESSAGE
78m         Normal    ClientCertificateCreated     deployment/cluster-monitoring-operator   A new client certificate for OpenShiftMonitoringTelemeterClientCertRequester is available
78m         Normal    CSRCreated                   deployment/cluster-monitoring-operator   A csr "system:openshift:openshift-monitoring-8kgvc" is created for OpenShiftMonitoringTelemeterClientCertRequester
78m         Normal    ClientCertificateCreated     deployment/cluster-monitoring-operator   A new client certificate for OpenShiftMonitoringMetricsServerClientCertRequester is available
78m         Normal    CertificateRotationStarted   deployment/cluster-monitoring-operator   The current client certificate for OpenShiftMonitoringMetricsServerClientCertRequester expires in 119h22m27s. Start certificate rotation
78m         Normal    CSRCreated                   deployment/cluster-monitoring-operator   A csr "system:openshift:openshift-monitoring-sgd5k" is created for OpenShiftMonitoringMetricsServerClientCertRequester
78m         Normal    CertificateRotationStarted   deployment/cluster-monitoring-operator   The current client certificate for OpenShiftMonitoringTelemeterClientCertRequester expires in 119h22m27s. Start certificate rotation

And the machine-config ClusterOperator was degraded.

~]$ oc get ClusterOperators
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE    MESSAGE
machine-config                             4.20.14   True        False         True       2y44d    Failed to resync 4.20.14 because: bootimage update failed: 3 Degraded MAPI MachineSets | 0 Degraded CAPI MachineSets | 0 CAPI MachineDeployments | Error(s): [error syncing MAPI MachineSet node1: refusing to reconcile machineset node2, hot loop detected. Please opt-out of boot image updates, adjust your machine provisioning workflow to prevent hot loops and opt back in to resume boot image updates, error syncing MAPI MachineSet node3: refusing to reconcile machineset node3, hot loop detected. Please opt-out of boot image updates, adjust your machine provisioning workflow to prevent hot loops and opt back in to resume boot image updates, error syncing MAPI MachineSet node1: refusing to reconcile machineset node1, hot loop detected. Please opt-out of boot image updates, adjust your machine provisioning workflow to prevent hot loops and opt back in to resume boot image updates]

In this scenario, the metrics-server-client-certs secret in the openshift-monitoring namespace contains the rotated secret with key ID 12:34:56:78:90:AB:CD:EF:GH:IJ:KL:MN:OP:QR:ST:UV:WX:YZ:1A:2B. Notice the certificate was issued by kube-csr-signer_@1234567890.

~]$ oc get secret metrics-server-client-certs --namespace openshift-monitoring --ooutput jsonpath="{.data.tls\.crt}" | base64 --decode | openssl x509 -text -noout
Certificate:
    Data:
        Issuer: CN = kube-csr-signer_@1234567890
        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:12:34:56:78:90:AB:CD:EF:GH:IJ:KL:MN:OP:QR:ST:UV:WX:YZ:1A:2B

The kube-apiserver-client-ca configmap should contain the issue (kube-csr-signer_@1234567890 in this example).

~]$ oc get configmap kube-apiserver-client-ca --namespace openshift-config-managed --output jsonpath='{.data.ca-bundle\.crt}' | openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -noout -text | grep -E "Subject:|Subject Key Identifier" | paste - -
        Subject: CN=kube-csr-signer_@1234567890

Ultimately, this was resolved by updating the MachineConfiguration cluster resource to have the following.

spec:
  managedBootImages:
    machineManagers:
    - apiGroup: machine.openshift.io
      resource: machinesets
      selection:
        mode: None

And then applying this change to trigger a reconciliation.

oc label machineset machine1 --namespace openshift-machine-api workaround=applied --overwrite 
oc label machineset machine2 --namespace openshift-machine-api workaround=applied --overwrite
oc label machineset machine3 --namespace openshift-machine-api workaround=applied --overwrite

Once this was done, the machine-config was no longer degraded, and the KubeletDown alerts stopped.

It is noteworthy that the certificate rotation itself did not cause the Machine Config Operator to become degraded. Instead, two independent events occurred at the same time.

The cluster automatically rotated the monitoring certificates as part of its normal certificate lifecycle. This is expected behavior, and the Authentication and Monitoring components began using the new certificates successfully. At the same time, the Machine Config Operator entered a degraded state due to the Boot Image Management issue. While the MCO was degraded, it was unable to propagate configuration changes to the nodes, including the updated CA bundle required by the kubelet.

As a result, the monitoring components presented certificates signed by the new CA, while the kubelets on the nodes were still using the older CA bundle. Since the updated CA had not yet been propagated, the kubelets rejected the connections with the certificate signed by unknown authority error, which ultimately resulted in the KubeletDown alerts.

Once the Boot Image Management issue was addressed by configuring managedBootImages with mode: None, the MCO recovered and resumed reconciling the pending configuration changes. The updated CA bundle was then propagated to the nodes, allowing the kubelets to trust the new certificates and clearing the alerts.