Let's you are have an app in ArgoCD that is stuck refreshing. In the ArgoCD console the Refresh button spins endlessly and perhaps eventually returns Error.

And the kubectl (Kubernetes) or oc (OpenShift) get applications.argoproj.io command has no sync status and no health status for the application.

~]$ oc get applications.argoproj.io --namespace openshift-gitops
NAME           SYNC STATUS   HEALTH STATUS
hello-world    Unknown       Healthy
hello-world2
hello-world3   Unknown       Healthy

If Error in red is displayed, click on the red Error text and see if there are details about the error, perhaps something like this.

Likewise, the argocd app list command may also return something interesting such as ComparisonError.

~]$ argocd app list
NAME                           CLUSTER          NAMESPACE     PROJECT     STATUS   HEALTH   SYNCPOLICY  CONDITIONS       REPO                               PATH    TARGET
openshift-gitops/hello-world   dev-cluster      my-cluster    my-project  Unknown  Healthy  Manual      ComparisonError  https://github.com/foo/bar.git     my-files main

And the argocd app get command should display the same Error as displayed in the console.

~]$ argocd app get openshift-gitops/hello-world1
Name:               openshift-gitops/hello-world
Project:            my-project
Server:             dev-cluster
Namespace:          my-namespace
URL:                https://openshift-gitops-server-openshift-gitops.apps.openshift.example.com/applications/hello-world
Source:
- Repo:             https://github.com/foo/bar.git
  Target:           main
  Path:             my-files
SyncWindow:         Sync Allowed
Sync Policy:        Manual
Sync Status:        Unknown
Health Status:      Healthy

CONDITION        MESSAGE                                                                                                                                                                                                                                                                                  LAST TRANSITION
ComparisonError  Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed timeout after 1m30s  2025-02-03 04:31:39 -0600 CST

And let's say you used the following YAML file to create the application.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello-world2
  namespace: openshift-gitops
spec:
  destination:
    namespace: my-project
    server: dev-cluster
  project: default
  source:
    path: python
    repoURL: https://github.com/foo/bar.git

Has the Cluster been added to ArgoCD?

The argocd cluster list command can be used to list the Kubernetes or OpenShift clusters that have been added to ArgoCD. In this example since we are trying to create the application in the dev-cluster we need to ensure a cluster named dev-cluster has been added to ArgoCD.

If the cluster has not been added to ArgoCD check out my article FreeKB - ArgoCD - Add Cluster using the CLI.

~]$ argocd cluster list
SERVER                                         NAME                VERSION  STATUS      MESSAGE       PROJECT
https://api.dev.openshift.example.com:6443     dev-cluster         1.27     Successful
https://api.stage.openshift.example.com:6443   stage-cluster       1.27     Successful
https://api.prod.openshift.example.com:6443    prod-cluster        1.27     Successful
https://kubernetes.default.svc                 in-cluster          1.27     Successful

Additionally let's ensure that when you run the argocd cluster list command in the ArgoCD server pod you get the same output.

~]$ SERVER_POD=$(oc get pod --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-server --output custom-columns=POD:.metadata.name --no-headers)
~]$ ADMIN_PASSWD=$(oc get secret openshift-gitops-cluster --namespace openshift-gitops --output jsonpath='{.data.admin\.password}' | base64 --decode)
~]$ URL=$(oc get routes --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-server --output jsonpath="{.items[*].spec.host}")
~]$ oc exec $SERVER_POD --namespace openshift-gitops -- /bin/bash -c "argocd login --username admin --password $ADMIN_PASSWD $URL:443 --insecure --grpc-web --config /home/argocd/.config; argocd cluster list --server $URL:443 --insecure --grpc-web --config /home/argocd/.config"
'admin:login' logged in successfully
SERVER                                         NAME                VERSION  STATUS      MESSAGE       PROJECT
https://api.dev.openshift.example.com:6443     dev-cluster         1.27     Successful
https://api.stage.openshift.example.com:6443   stage-cluster       1.27     Successful
https://api.prod.openshift.example.com:6443    prod-cluster        1.27     Successful
https://kubernetes.default.svc                 in-cluster          1.27     Successful

Can the application be created in the target namespace?

Let's use the argocd proj list command to ensure the project exists (the "default" project in this example). 

• Ensure DESTINATIONS allows the application to be created in the target namespace. In this example *,* means applications can be created in any target namespace
• Ensure sourceRepos allows ArgoCD to talk to the source repository. In this example * means ArgoCD can reach any repo.

~]$ argocd proj list
NAME        DESCRIPTION  DESTINATIONS  SOURCES  CLUSTER-RESOURCE-WHITELIST  NAMESPACE-RESOURCE-BLACKLIST  SIGNATURE-KEYS  ORPHANED-RESOURCES  DESTINATION-SERVICE-ACCOUNTS
default                  *,*           *        */*                         <none>                        <none>          disabled            <none>

Additionally let's ensure that when you run the argocd proj list command in the ArgoCD server pod you get the same output.

~]$ SERVER_POD=$(oc get pod --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-server --output custom-columns=POD:.metadata.name --no-headers)
~]$ ADMIN_PASSWD=$(oc get secret openshift-gitops-cluster --namespace openshift-gitops --output jsonpath='{.data.admin\.password}' | base64 --decode)
~]$ URL=$(oc get routes --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-server --output jsonpath="{.items[*].spec.host}")
~]$ oc exec $SERVER_POD --namespace openshift-gitops -- /bin/bash -c "argocd login --username admin --password $ADMIN_PASSWD $URL:443 --insecure --grpc-web --config /home/argocd/.config; argocd proj list --server $URL:443 --insecure --grpc-web --config /home/argocd/.config"
'admin:login' logged in successfully
NAME        DESCRIPTION  DESTINATIONS  SOURCES  CLUSTER-RESOURCE-WHITELIST  NAMESPACE-RESOURCE-BLACKLIST  SIGNATURE-KEYS  ORPHANED-RESOURCES  DESTINATION-SERVICE-ACCOUNTS
default                  *,*           *        */*                         <none>                        <none>          disabled            <none>

Likewise, the kubectl (Kubernetes) or oc (OpenShift) get appproject.argoproj.io command on the Kubernetes or OpenShift cluster that ArgoCD is on can be used.

• Ensure destinations namespace allows the application to be created in the target namespace. In this example '*' means applications can be created in any target namespace
• Ensure destinations server allows the application to be created in the target Kubernetes or OpenShift server. In this example '*' means applications can be created in any Kubernetes or OpenShift cluster.
• Ensure sourceRepos allows ArgoCD to talk to the source repository. In this example '*' means ArgoCD can reach any repo.

~]$ oc get appproject.argoproj.io default --namespace openshift-gitops --output yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  creationTimestamp: "2024-11-26T02:28:06Z"
  generation: 1
  name: default
  namespace: openshift-gitops
  resourceVersion: "374751652"
  uid: 34e881f3-0ec6-44e6-a4d2-9971361004b6
spec:
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'
  destinations:
  - namespace: '*'
    server: '*'
  sourceRepos:
  - '*'
status: {}

Does the namespace exist on the target cluster?

On the Kubernetes or OpenShift cluster that you want the application to be installed in, let's use the kubectl (Kubernetes) or oc (OpenShift) get projects command to determine if the namespace exists.

~]$ oc get projects
NAME                                                        DISPLAY NAME   STATUS
my-project                                                                 Active

If the namespace does not exist, one option is to use the kubectl (Kubernetes) or oc (OpenShift) new-project command to create the namespace.

oc new-project my-project

Or if creating an ArgoCD application using a YAML file you can include the CreateNamespace=true option.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  labels:
    app.kubernetes.io/part-of: argocd
    namespace: my-namespace
  name: my-app
  namespace: openshift-gitops
spec:
  destination:
    name: onprem-dev
    namespace: my-namespace
  project: my-project
  source:
    path: my-files
    repoURL: https://github.com/foo/bar.git
    targetRevision: main
  syncPolicy:
    managedNamespaceMetadata:
      labels:
        argocd.argoproj.io/managed-by: openshift-gitops
    syncOptions:
    - CreateNamespace=true

Does the namespace have the argocd.argoproj.io/managed-by label?

The kubectl (Kubernetes) or oc (OpenShift) describe namespace command can be used to determine if the namespaces on your target Kubernetes or OpenShift cluster has been configured to be managed by ArgoCD. Supposedly this is needed to allow ArgoCD to manage resources such as deployments, pods, services, et cetera in namespaces outside of the namespace ArgoCD is in.

In this example, the my-project namespace does not have the argocd.argoproj.io/managed-by=<the namespace ArgoCD is in> label meaning my-project is not configured to be managed by ArgoCD.

~]$ oc describe namespace my-project
Name:         my-project
Labels:       kubernetes.io/metadata.name=my-project
              pod-security.kubernetes.io/audit=baseline
              pod-security.kubernetes.io/audit-version=v1.24
              pod-security.kubernetes.io/warn=baseline
              pod-security.kubernetes.io/warn-version=v1.24

The kubectl (Kubernetes) or oc (OpenShift) label command can be used to label the namespace. In this example openshift-gitops is used because ArgoCD is in the openshift-gitops namespace.

~]$ oc label namespace my-project argocd.argoproj.io/managed-by=openshift-gitops
namespace/my-project labeled

Or if creating an ArgoCD application using a YAML file you can include the argocd.argoproj.io/managed-by option.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  labels:
    app.kubernetes.io/part-of: argocd
    namespace: my-namespace
  name: my-app
  namespace: openshift-gitops
spec:
  destination:
    name: onprem-dev
    namespace: my-namespace
  project: my-project
  source:
    path: my-files
    repoURL: https://github.com/foo/bar.git
    targetRevision: main
  syncPolicy:
    managedNamespaceMetadata:
      labels:
        argocd.argoproj.io/managed-by: openshift-gitops
    syncOptions:
    - CreateNamespace=true

And now the namespace has the argocd.argoproj.io/managed-by=openshift-gitops label.

~]$ oc describe namespace my-project
Name:         my-project
Labels:       argocd.argoproj.io/managed-by=openshift-gitops
              kubernetes.io/metadata.name=my-project
              pod-security.kubernetes.io/audit=baseline
              pod-security.kubernetes.io/audit-version=v1.24
              pod-security.kubernetes.io/warn=baseline
              pod-security.kubernetes.io/warn-version=v1.24

Has the repo been added to ArgoCD?

The argocd repo list command can be used to list the repo's you have added to ArgoCD.

~]$ argocd repo list
TYPE  NAME                 REPO                                             INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE  PROJECT
git   argocd-example-apps  https://github.com/foo/bar.git  false     false  false  false  Successful           default

Additionally let's ensure that when you run the argocd repo list command in the ArgoCD server pod you get the same output.

~]$ SERVER_POD=$(oc get pod --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-server --output custom-columns=POD:.metadata.name --no-headers)
~]$ ADMIN_PASSWD=$(oc get secret openshift-gitops-cluster --namespace openshift-gitops --output jsonpath='{.data.admin\.password}' | base64 --decode)
~]$ URL=$(oc get routes --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-server --output jsonpath="{.items[*].spec.host}")
~]$ oc exec $SERVER_POD --namespace openshift-gitops -- /bin/bash -c "argocd login --username admin --password $ADMIN_PASSWD $URL:443 --insecure --grpc-web --config /home/argocd/.config; argocd repo list --server $URL:443 --insecure --grpc-web --config /home/argocd/.config"
'admin:login' logged in successfully
TYPE  NAME                 REPO                                             INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE  PROJECT
git   argocd-example-apps  https://github.com/foo/bar.git  false     false  false  false  Successful           default

If the repo that the application is using is not listed, the argocd repo add command can be used to add the repo to ArgoCD.

argocd repo add https://github.com/foo/bar.git

Is there a certificate for the repo?

The argocd cert list command can be used to list the certificates that ArgoCD has been configured to trust for the target version control system, github.com in this example. These are the SSH keys that the target version control system presents that would be appended to known_hosts on the connecting system, which is ArgoCD in this scenario.

~]$ argocd cert list
HOSTNAME                    TYPE   SUBTYPE              INFO
[ssh.github.com]:443        ssh    ecdsa-sha2-nistp256  SHA256:p2QAMXNICABCDFzIOttrVc98/R1BUFWu3/LiyKgUfQM
[ssh.github.com]:443        ssh    ssh-ed25519          SHA256:+DiY3wvvV6CABCDFzZisF/zLDA0zPMSvHdkr4UvCOqU
[ssh.github.com]:443        ssh    ssh-rsa              SHA256:uNiVztksCABCDFz0u9e8BujQXVUpKZIDTMczCvj3tDs
github.com                  ssh    ssh-rsa              SHA256:uNiVztksCCABCDFz0u9e8BujQXVUpKZIDTMczCvjD2s
github.com                  ssh    ssh-ed25519          SHA256:+DiY3wvvV6TCABCDFzisF/zLDA0zPMSvHdkr4UvCOqU
github.com                  ssh    ecdsa-sha2-nistp256  SHA256:p2QAMXNICCABCDFzOttrVc98/R1BUFWu3/LiyKgUfQM

Additionally let's ensure that when you run the argocd cert list command in the ArgoCD server pod you get the same output.

~]$ SERVER_POD=$(oc get pod --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-server --output custom-columns=POD:.metadata.name --no-headers)
~]$ ADMIN_PASSWD=$(oc get secret openshift-gitops-cluster --namespace openshift-gitops --output jsonpath='{.data.admin\.password}' | base64 --decode)
~]$ URL=$(oc get routes --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-server --output jsonpath="{.items[*].spec.host}")
~]$ oc exec $SERVER_POD --namespace openshift-gitops -- /bin/bash -c "argocd login --username admin --password $ADMIN_PASSWD $URL:443 --insecure --grpc-web --config /home/argocd/.config; argocd cert list --server $URL:443 --insecure --grpc-web --config /home/argocd/.config"
'admin:login' logged in successfully
HOSTNAME                    TYPE   SUBTYPE              INFO
[ssh.github.com]:443        ssh    ecdsa-sha2-nistp256  SHA256:p2QAMXNICABCDFzIOttrVc98/R1BUFWu3/LiyKgUfQM
[ssh.github.com]:443        ssh    ssh-ed25519          SHA256:+DiY3wvvV6CABCDFzZisF/zLDA0zPMSvHdkr4UvCOqU
[ssh.github.com]:443        ssh    ssh-rsa              SHA256:uNiVztksCABCDFz0u9e8BujQXVUpKZIDTMczCvj3tDs
github.com                  ssh    ssh-rsa              SHA256:uNiVztksCCABCDFz0u9e8BujQXVUpKZIDTMczCvjD2s
github.com                  ssh    ssh-ed25519          SHA256:+DiY3wvvV6TCABCDFzisF/zLDA0zPMSvHdkr4UvCOqU
github.com                  ssh    ecdsa-sha2-nistp256  SHA256:p2QAMXNICCABCDFzOttrVc98/R1BUFWu3/LiyKgUfQM

Ingress and Egress Network Policies

The kubectl (Kubernetes) or oc (OpenShift) get networkpolicies command can then be see if there are any ingress network policies denying incoming requests in the namespace you want to deploy the application to.

~]$ oc get networkpolicies
No resources found in my_project namespace.

The kubectl (Kubernetes) or oc (OpenShift) get egressnetworkpolicies command can be used to see if there are any egress network policies denying outgoing requests in the namespace ArgoCD is in.

~]$ oc get egressnetworkpolicies --namespace openshift-gitops
No resources found in openshift-gitops namespace.

NetNamespaces

The kubectl (Kubernetes) or oc (OpenShift) get netnamespaces can be used to see if the namespace that you want to deploy the application to has a dedicated egress IP address.

~]$ oc get netnamespaces my_project
NAME               NETID     EGRESS IPS
my_project         9740194

If not, you may need to patch the namespace to have a dedicated egress IP address.

~]$ oc patch netnamespace my_project --type merge --patch '{ "egressIPs": [ "10.84.189.2" ] }'
netnamespace.network.openshift.io/my_project  patched

Are the ArgoCD pods up and running?

The kubectl (Kubernetes) or oc (OpenShift) get pods command can be used to check if the ArgoCD pods in the cluster ArgoCD is running on are up and running.

By the way, notice in this example that the application controller pod has 13 restarts whereas the other pods have 0 restarts. This is because when an application would get stuck refreshing my application controller pod would run out of memory and start crash looping (more on this in a moment).

~]$ oc get pods
NAME                                                          READY   STATUS    RESTARTS         AGE
cluster-f4c765c6f-tznxm                                       1/1     Running   0                3d15h
gitops-plugin-76c8d56cbc-xbdl7                                1/1     Running   0                3d15h
kam-6fd58b8d87-ggrlr                                          1/1     Running   0                3d15h
openshift-gitops-application-controller-0                     1/1     Running   13 (2d20h ago)   3d15h
openshift-gitops-applicationset-controller-6858947bc8-7nfhv   1/1     Running   0                3d15h
openshift-gitops-dex-server-5f8549dd48-24x72                  1/1     Running   0                3d15h
openshift-gitops-redis-7bd479c78f-zb8xg                       1/1     Running   0                3d15h
openshift-gitops-repo-server-694ddc84cc-66r2c                 1/1     Running   0                3d15h
openshift-gitops-server-7dddf67589-g5nzq                      1/1     Running   0                3d15h

Restart the pods

Of course this isn't a permanent solution but restarting the pods can be a good throubeshooting option. If you restart the pods and then the application successfully refreshes then you know something with one or more of the pods is not quite right. However, if you restart the pods and the application remains stuck there you know that it's not like the pods are in a bad way. Assuming you have a replica set or stateful set configured with 1 or more replicas then deleting the pod should cause a new pod to immediately spawn.

oc delete pod openshift-gitops-application-controller-0

Check pod logs for interesting events

The kubectl (Kubernetes) or oc (OpenShift) logs command can be used to view the logs in each the ArgoCD pod.

oc logs openshift-gitops-application-controller-0

Configure ArgoCD to log events at log level DEBUG

If there are no interesting events in the ArgoCD pod logs, you may want to update ArgoCD to log events at log level DEBUG. For example, here is how you could check to see if the ArgoCD repo server has ben configured with a logLevel. If there is no logLevel then this simply means that no log level has been set thus the resource (repo in this example) would log events at the default log level, which is INFO.

~]$ oc get argocd openshift-gitops --namespace openshift-gitops --output jsonpath="{.spec.repo}" | jq
{
  "logLevel": "debug",
  "resources": {
    "limits": {
      "cpu": "1",
      "memory": "1Gi"
    },
    "requests": {
      "cpu": "250m",
      "memory": "256Mi"
    }
  }
}

For example, to update "repo" to log level debug. This should get the repo-server pod to log events at log level debug.

oc patch argocd openshift-gitops --namespace openshift-gitops --patch '{"spec":{"repo":{"logLevel":"debug"}}}' --type=merge

Are the resources getting created on the target cluster?

Let's say you are attempting to create an application in the my-project namespace in your development cluster. In the development cluster, check to see if there are any resources in the target namespace. If not, then you know that ArgoCD is unable to reach or create resources in the target namespace in the development cluster.

~]$ oc get all --namespace my-project
No resources found in my-project namespace.

Ingress and Egress Network Policies

The kubectl (Kubernetes) or oc (OpenShift) get networkpolicies command can then be see if there are any ingress network policies denying incoming requests in the namespace you want to deploy the application to.

~]$ oc get networkpolicies
No resources found in my_project namespace.

The kubectl (Kubernetes) or oc (OpenShift) get egressnetworkpolicies command can be used to see if there are any egress network policies denying outgoing requests in the namespace ArgoCD is in.

~]$ oc get egressnetworkpolicies --namespace openshift-gitops
No resources found in openshift-gitops namespace.

NetNamespaces

The kubectl (Kubernetes) or oc (OpenShift) get netnamespaces can be used to see if the namespace that you want to deploy the application to has a dedicated egress IP address.

~]$ oc get netnamespaces my_project
NAME               NETID     EGRESS IPS
my_project         9740194

If not, you may need to patch the namespace to have a dedicated egress IP address.

~]$ oc patch netnamespace my_project --type merge --patch '{ "egressIPs": [ "10.84.189.2" ] }'
netnamespace.network.openshift.io/my_project  patched