Bootstrap FreeKB - OpenShift - Resolve Project stuck in a Terminating state
OpenShift - Resolve Project stuck in a Terminating state

Updated:   |  OpenShift articles

Let's say you use the oc delete project command to delete a project.

oc delete project my-project

 

But the project status remains Terminating indefinately.

~]$ oc get project my-project
NAME         DISPLAY NAME   STATUS
my-project                  Terminating

 

Let's take a look at the project YAML. Notice in this example the status has NamespaceFinalizersRemaining. In this scenario, you don't want to try to force delete the project. Instead, you want to determine why the project has remaining finalizers and then take appropriate action with the finalizers.

~]$ oc get project my-project --output yaml
apiVersion: project.openshift.io/v1
kind: Project
metadata:
  annotations:
    openshift.io/description: ""
    openshift.io/display-name: ""
    openshift.io/sa.scc.mcs: s0:c58,c17
    openshift.io/sa.scc.supplemental-groups: 1003340000/10000
    openshift.io/sa.scc.uid-range: 1003340000/10000
  creationTimestamp: "2021-04-15T20:03:59Z"
  deletionTimestamp: "2022-09-16T15:27:51Z"
  labels:
    kubernetes.io/metadata.name: my-project
    maistra.io/member-of: my-project
    kiali.io/member-of: my-project
  name: my-project
  resourceVersion: "1273963823"
  uid: f112402c-cbdf-4805-a66e-66228d245de3
spec:
  finalizers:
  - kubernetes
status:
  conditions:
  - lastTransitionTime: "2022-09-16T15:27:58Z"
    message: 'Some content in the namespace has finalizers remaining: finalizer.kiali
      in 1 resource instances, maistra.io/my-project in 2 resource instances'
    reason: SomeFinalizersRemain
    status: "True"
    type: NamespaceFinalizersRemaining

 

It is noteworthy that I had a situation where the istio-system project was stuck in a Terminating state, and I had to open a case with Red Hat. I first got the name of the master pods in the openshift-etcd project.

~]$ oc get pods --namespace openshift-etcd
NAME                          READY   STATUS      RESTARTS   AGE
etcd-master-0                 5/5     Running     0          27d
etcd-master-1                 5/5     Running     0          27d
etcd-master-2                 5/5     Running     0          27d

 

I then issued the etcdclt del command in one of the master pods to delete the maistra Service Mesh control planes and finally the istio-system namespace.

oc exec --tty true --stdin true etcd-master-0 --namespace openshift-etcd -- etcdctl del /kubernetes.io/maistra.io/servicemeshcontrolplanes/istio-system/basic

oc exec --tty true --stdin true etcd-master-0 --namespace openshift-etcd -- etcdctl del /kubernetes.io/maistra.io/servicemeshmemberrolls/istio-system/default

oc exec --tty true --stdin true etcd-master-0 --namespace openshift-etcd -- etcdctl del /kubernetes.io/namespaces/istio-system

 

Remove the Custom Resource Definitions

I then had to use the oc get customresourcedefinitions command to return the istio, Jaegar, Kiali, Maistra, Service Mesh Custom Resource Definitions.

~]# oc get customresourcedefinitions | egrep -i 'istio|jaegar|kiali|maistra|servicemesh'
kialis.kiali.io                                                   2021-04-15T20:01:30Z
monitoringdashboards.monitoring.kiali.io                          2021-04-15T20:01:30Z
servicemeshextensions.maistra.io                                  2021-04-15T20:09:13Z
servicemeshmemberrolls.maistra.io                                 2021-04-15T20:02:49Z
servicemeshmembers.maistra.io                                     2021-04-15T20:02:49Z
servicemeshpolicies.authentication.maistra.io                     2021-04-15T20:09:13Z
servicemeshrbacconfigs.rbac.maistra.io                            2021-04-15T20:09:13Z

 

And then use the oc delete customresourcedefinition command to delete the Custom Resource Definitions.

oc delete crd kialis.kiali.io
oc delete crd monitoringdashboards.monitoring.kiali.io
oc delete crd servicemeshextensions.maistra.io
oc delete crd servicemeshmemberrolls.maistra.io
oc delete crd servicemeshmembers.maistra.io
oc delete crd servicemeshpolicies.authentication.maistra.io
oc delete crd servicemeshrbacconfigs.rbac.maistra.io

 

And then re-issue the oc get customresourcedefinitions command to ensure the Custom Resource Definitions were deleted. No results should be returned.

oc get customresourcedefinitions | egrep -i 'istio|kiali|maistra|servicemesh'

 

If the Custom Resource Definition remains, you may need to remove finalizers from the Custom Resource Definition. For example, to delete the kialis.kiali.io Custom Resource Definition, I had to use the oc edit command and remove the customresourcecleanup.apiextensions.k8s.io finalizer. Once the finalizer was removed, the Custom Resource Definition was no longer listed.

~]$ oc get customresourcedefinition kialis.kiali.io --output yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "2021-04-15T20:01:30Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2022-09-16T15:42:35Z"
  finalizers:
  - customresourcecleanup.apiextensions.k8s.io

 

Remove the Operators

I then had to use the oc get operators command to return the istio, Jaegar, Kiali, Maistra, Service Mesh Operators.

~]$ oc get operators
NAME                                                AGE
cluster-logging.openshift-logging                   2y151d
elasticsearch-operator.openshift-operators-redhat   2y151d
jaeger-product.openshift-operators                  2y72d
kiali-ossm.openshift-operators                      3d18h
servicemeshoperator.openshift-operators             3d18h

 

And then I used the oc delete operator command to uninstall the Jaegar, Kiali, and Service Mesh Operators.

oc delete operator jaeger-product.openshift-operators
oc delete operator kiali-ossm.openshift-operators
oc delete operator servicemeshoperator.openshift-operators

 

And then I re-issued the oc get operators command to ensure the Jaegar, Kiali, and Service Mesh Operators were uninstalled.

~]$ oc get operators
NAME                                                AGE
cluster-logging.openshift-logging                   2y151d
elasticsearch-operator.openshift-operators-redhat   2y151d

 

 

 




Did you find this article helpful?

If so, consider buying me a coffee over at Buy Me A Coffee



Comments


Add a Comment


Please enter f3badd in the box below so that we can be sure you are a human.