If you are not familiar with the oc command, refer to OpenShift - Getting Started with the oc command.

The oc get clusteroperator command can be used to list the cluster operators. Something like this should be returned. Notice in this example that operator-lifecycle-manager-packageserver is not available and has message "serving cert not active".

~]$ oc get clusteroperator operator-lifecycle-manager-packageserver
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
operator-lifecycle-manager-packageserver   4.14.23   False       True          False      0s      ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: APIServiceResourceIssue, message: found the serving cert not active

The following secrets in the openshift-operator-lifecycle-manager namespace contain the certificates used by operator-lifecycle-manager-packageserver.

• catalog-operator-serving-cert
• olm-operator-serving-cert
• packageserver-service-cert

The oc get secrets command can be used to list the secrets in the openshift-operator-lifecycle-manager namespace.

~]$ oc get secrets --namespace openshift-operator-lifecycle-manager
NAME                                          TYPE                                  DATA   AGE
builder-dockercfg-4rc6t                       kubernetes.io/dockercfg               1      4y2d
builder-token-kqc74                           kubernetes.io/service-account-token   4      4y2d
builder-token-mhdwj                           kubernetes.io/service-account-token   4      4y2d
catalog-operator-serving-cert                 kubernetes.io/tls                     2      4y2d
collect-profiles-dockercfg-8fslz              kubernetes.io/dockercfg               1      729d
collect-profiles-token-bms54                  kubernetes.io/service-account-token   4      729d
collect-profiles-token-xrhzd                  kubernetes.io/service-account-token   4      729d
default-dockercfg-4rn7w                       kubernetes.io/dockercfg               1      4y2d
default-token-fltjm                           kubernetes.io/service-account-token   4      4y2d
default-token-wrbtl                           kubernetes.io/service-account-token   4      4y2d
deployer-dockercfg-hn2b4                      kubernetes.io/dockercfg               1      4y2d
deployer-token-jwmcd                          kubernetes.io/service-account-token   4      4y2d
deployer-token-zbtgq                          kubernetes.io/service-account-token   4      4y2d
olm-operator-serviceaccount-dockercfg-z575w   kubernetes.io/dockercfg               1      4y2d
olm-operator-serviceaccount-token-75tx4       kubernetes.io/service-account-token   4      4y2d
olm-operator-serviceaccount-token-f2lwj       kubernetes.io/service-account-token   4      4y2d
olm-operator-serving-cert                     kubernetes.io/tls                     2      4y2d
packageserver-service-cert                    kubernetes.io/tls                     3      4y2d
pprof-cert                                    kubernetes.io/tls                     2      729d

This one-liner can be used to list the date that the certificate in the catalog-operator-serving-cert was issued (notBefore) and the date the certificate expires (notAfter).

~]$ oc get secret catalog-operator-serving-cert --namespace openshift-operator-lifecycle-manager --output jsonpath="{.data.tls\.crt}" | base64 --decode | openssl x509 -noout -dates
notBefore=Oct  4 09:53:50 2022 GMT
notAfter=Oct  2 09:53:50 2024 GMT

Similarly, this one-liner can be used to list the date that the certificate in the olm-operator-serving-cert was issued (notBefore) and the date the certificate expires (notAfter).

~]$ oc get secret olm-operator-serving-cert --namespace openshift-operator-lifecycle-manager --output jsonpath="{.data.tls\.crt}" | base64 --decode | openssl x509 -noout -dates
notBefore=Oct  4 09:53:50 2022 GMT
notAfter=Oct  2 09:53:50 2024 GMT

And this one-liner can be used to list the date that the certificate in the packageserver-service-cert was issued (notBefore) and the date the certificate expires (notAfter).

~]$ oc get secret packageserver-service-cert --namespace openshift-operator-lifecycle-manager --output jsonpath="{.data.tls\.crt}" | base64 --decode | openssl x509 -noout -dates
notBefore=Oct  4 09:53:50 2022 GMT
notAfter=Oct  2 09:53:50 2024 GMT

Typically, at least one of these secrets will contain an expired certificate. Or sometimes all three of the secrets will contain an expired certificate. Let's say all three secrets contain an expired certificate. First and foremost, let's create backup copies of the YAML for each secret.

oc get secret catalog-operator-serving-cert --namespace openshift-operator-lifecycle-manager --output yaml > /tmp/catalog-operator-serving-cert-secret.backup.yaml

oc get secret olm-operator-serving-cert --namespace openshift-operator-lifecycle-manager --output yaml > /tmp/olm-operator-serving-cert-secret.backup.yaml

oc get secret packageserver-service-cert --namespace openshift-operator-lifecycle-manager --output yaml > /tmp/packageserver-service-cert-secret.backup.yaml

And then let's use the oc delete secret command to delete each secret.

oc delete secret catalog-operator-serving-cert --namespace openshift-operator-lifecycle-manager

oc delete secret olm-operator-serving-cert --namespace openshift-operator-lifecycle-manager

oc delete secret packageserver-service-cert --namespace openshift-operator-lifecycle-manager

New secrets should immediate be re-created. Notice that the AGE of the catalog-operator-serving-cert and olm-operator-serving-cert and packageserver-service-cert secrets are now only a few seconds old.

~]$ oc get secrets --namespace openshift-operator-lifecycle-manager
NAME                                          TYPE                                  DATA   AGE
builder-dockercfg-4rc6t                       kubernetes.io/dockercfg               1      4y2d
builder-token-kqc74                           kubernetes.io/service-account-token   4      4y2d
builder-token-mhdwj                           kubernetes.io/service-account-token   4      4y2d
catalog-operator-serving-cert                 kubernetes.io/tls                     2      15s
collect-profiles-dockercfg-8fslz              kubernetes.io/dockercfg               1      729d
collect-profiles-token-bms54                  kubernetes.io/service-account-token   4      729d
collect-profiles-token-xrhzd                  kubernetes.io/service-account-token   4      729d
default-dockercfg-4rn7w                       kubernetes.io/dockercfg               1      4y2d
default-token-fltjm                           kubernetes.io/service-account-token   4      4y2d
default-token-wrbtl                           kubernetes.io/service-account-token   4      4y2d
deployer-dockercfg-hn2b4                      kubernetes.io/dockercfg               1      4y2d
deployer-token-jwmcd                          kubernetes.io/service-account-token   4      4y2d
deployer-token-zbtgq                          kubernetes.io/service-account-token   4      4y2d
olm-operator-serviceaccount-dockercfg-z575w   kubernetes.io/dockercfg               1      4y2d
olm-operator-serviceaccount-token-75tx4       kubernetes.io/service-account-token   4      4y2d
olm-operator-serviceaccount-token-f2lwj       kubernetes.io/service-account-token   4      4y2d
olm-operator-serving-cert                     kubernetes.io/tls                     2      43s
packageserver-service-cert                    kubernetes.io/tls                     3      55s
pprof-cert                                    kubernetes.io/tls                     2      729d

This one-liner can be used again and now the date that the certificate in the catalog-operator-serving-cert was issued (notBefore) should be the current date.

~]$ oc get secret catalog-operator-serving-cert --namespace openshift-operator-lifecycle-manager --output jsonpath="{.data.tls\.crt}" | base64 --decode | openssl x509 -noout -dates
notBefore=Oct  2 10:34:53 2024 GMT
notAfter=Oct  2 10:34:53 2026 GMT

Similarly, this one-liner can be used again to see that the date that the certificate in the olm-operator-serving-cert was issued (notBefore) should be the current date.

~]$ oc get secret olm-operator-serving-cert --namespace openshift-operator-lifecycle-manager --output jsonpath="{.data.tls\.crt}" | base64 --decode | openssl x509 -noout -dates
notBefore=Oct  2 10:34:53 2024 GMT
notAfter=Oct  2 10:34:53 2026 GMT

And the same for the packageserver-service-cert secret.

~]$ oc get secret packageserver-service-cert --namespace openshift-operator-lifecycle-manager --output jsonpath="{.data.tls\.crt}" | base64 --decode | openssl x509 -noout -dates
notBefore=Oct  2 10:34:53 2024 GMT
notAfter=Oct  2 10:34:53 2026 GMT

And typically, this will resolve the issue, and operator-lifecycle-manager-packageserver should again be available and not have an error message.

~]$ oc get clusteroperator
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE    MESSAGE
operator-lifecycle-manager-packageserver   4.14.23   True        False         False      89s

According to this Red Hat article, the defect is that the certificate in the catalog-operator-serving-cert and olm-operator-serving-cert and packageserver-service-cert secrets are not automatically rotated, and the fix for this defect is in the following version of OpenShift.

• 4.14.35
• 4.15.23
• 4.16.2

The oc get clusterversion command can be used to list the version of your OpenShift cluster.

~]$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.23   True        False         132d    Cluster version is 4.14.23