Let's say something like this is being returned.
ComparisonError Failed to load target state: failed to generate manifest for source 1 of 1: rpc error:
code = Unknown
desc = failed to initialize repository resources: rpc error:
code = Internal
desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed timeout after 1m30s
2025-02-03 04:31:39 -0600 CST
I saw this when using the argocd app get command. This suggests a timeout between ArgoCD and the target repo.
~]$ argocd app get openshift-gitops/hello-world1
Name: openshift-gitops/hello-world
Project: my-project
Server: dev-cluster
Namespace: my-namespace
URL: https://openshift-gitops-server-openshift-gitops.apps.openshift.example.com/applications/hello-world
Source:
- Repo: https://github.com/foo/bar.git
Target: main
Path: my-files
SyncWindow: Sync Allowed
Sync Policy: Manual
Sync Status: Unknown
Health Status: Healthy
CONDITION MESSAGE LAST TRANSITION
ComparisonError Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed timeout after 1m30s 2025-02-03 04:31:39 -0600 CST
Has the repo been added to ArgoCD?
The argocd repo list command can be used to list the repo's you have added to ArgoCD.
~]$ argocd repo list
TYPE NAME REPO INSECURE OCI LFS CREDS STATUS MESSAGE PROJECT
git argocd-example-apps https://github.com/foo/bar.git false false false false Successful default
Additionally let's ensure that when you run the argocd repo list command in the ArgoCD server pod you get the same output.
~]$ SERVER_POD=$(oc get pod --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-server --output custom-columns=POD:.metadata.name --no-headers)
~]$ ADMIN_PASSWD=$(oc get secret openshift-gitops-cluster --namespace openshift-gitops --output jsonpath='{.data.admin\.password}' | base64 --decode)
~]$ URL=$(oc get routes --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-server --output jsonpath="{.items[*].spec.host}")
~]$ oc exec $SERVER_POD --namespace openshift-gitops -- /bin/bash -c "argocd login --username admin --password $ADMIN_PASSWD $URL:443 --insecure --grpc-web --config /home/argocd/.config; argocd repo list --server $URL:443 --insecure --grpc-web --config /home/argocd/.config"
'admin:login' logged in successfully
TYPE NAME REPO INSECURE OCI LFS CREDS STATUS MESSAGE PROJECT
git argocd-example-apps https://github.com/foo/bar.git false false false false Successful default
If the repo that the application is using is not listed, the argocd repo add command can be used to add the repo to ArgoCD.
argocd repo add https://github.com/foo/bar.git
Is there a certificate for the repo?
The argocd cert list command can be used to list the certificates that ArgoCD has been configured to trust for the target version control system, github.com in this example. These are the SSH keys that the target version control system presents that would be appended to known_hosts on the connecting system, which is ArgoCD in this scenario.
~]$ argocd cert list
HOSTNAME TYPE SUBTYPE INFO
[ssh.github.com]:443 ssh ecdsa-sha2-nistp256 SHA256:p2QAMXNICABCDFzIOttrVc98/R1BUFWu3/LiyKgUfQM
[ssh.github.com]:443 ssh ssh-ed25519 SHA256:+DiY3wvvV6CABCDFzZisF/zLDA0zPMSvHdkr4UvCOqU
[ssh.github.com]:443 ssh ssh-rsa SHA256:uNiVztksCABCDFz0u9e8BujQXVUpKZIDTMczCvj3tDs
github.com ssh ssh-rsa SHA256:uNiVztksCCABCDFz0u9e8BujQXVUpKZIDTMczCvjD2s
github.com ssh ssh-ed25519 SHA256:+DiY3wvvV6TCABCDFzisF/zLDA0zPMSvHdkr4UvCOqU
github.com ssh ecdsa-sha2-nistp256 SHA256:p2QAMXNICCABCDFzOttrVc98/R1BUFWu3/LiyKgUfQM
Additionally let's ensure that when you run the argocd cert list command in the ArgoCD server pod you get the same output.
~]$ SERVER_POD=$(oc get pod --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-server --output custom-columns=POD:.metadata.name --no-headers)
~]$ ADMIN_PASSWD=$(oc get secret openshift-gitops-cluster --namespace openshift-gitops --output jsonpath='{.data.admin\.password}' | base64 --decode)
~]$ URL=$(oc get routes --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-server --output jsonpath="{.items[*].spec.host}")
~]$ oc exec $SERVER_POD --namespace openshift-gitops -- /bin/bash -c "argocd login --username admin --password $ADMIN_PASSWD $URL:443 --insecure --grpc-web --config /home/argocd/.config; argocd cert list --server $URL:443 --insecure --grpc-web --config /home/argocd/.config"
'admin:login' logged in successfully
HOSTNAME TYPE SUBTYPE INFO
[ssh.github.com]:443 ssh ecdsa-sha2-nistp256 SHA256:p2QAMXNICABCDFzIOttrVc98/R1BUFWu3/LiyKgUfQM
[ssh.github.com]:443 ssh ssh-ed25519 SHA256:+DiY3wvvV6CABCDFzZisF/zLDA0zPMSvHdkr4UvCOqU
[ssh.github.com]:443 ssh ssh-rsa SHA256:uNiVztksCABCDFz0u9e8BujQXVUpKZIDTMczCvj3tDs
github.com ssh ssh-rsa SHA256:uNiVztksCCABCDFz0u9e8BujQXVUpKZIDTMczCvjD2s
github.com ssh ssh-ed25519 SHA256:+DiY3wvvV6TCABCDFzisF/zLDA0zPMSvHdkr4UvCOqU
github.com ssh ecdsa-sha2-nistp256 SHA256:p2QAMXNICCABCDFzOttrVc98/R1BUFWu3/LiyKgUfQM
Ingress and Egress Network Policies
The kubectl (Kubernetes) or oc (OpenShift) get networkpolicies command can then be see if there are any ingress network policies denying incoming requests in the namespace you want to deploy the application to.
~]$ oc get networkpolicies
No resources found in openshift-gitops namespace.
The kubectl (Kubernetes) or oc (OpenShift) get egressnetworkpolicies command can be used to see if there are any egress network policies denying outgoing requests in the namespace ArgoCD is in.
~]$ oc get egressnetworkpolicies --namespace openshift-gitops
No resources found in openshift-gitops namespace.
NetNamespaces
The kubectl (Kubernetes) or oc (OpenShift) get netnamespaces can be used to see if the namespace that you want to deploy the application to has a dedicated egress IP address.
~]$ oc get netnamespaces openshift-gitops
NAME NETID EGRESS IPS
openshift-gitops 9740194
If not, you may need to patch the namespace to have a dedicated egress IP address.
~]$ oc patch netnamespace openshift-gitops --type merge --patch '{ "egressIPs": [ "10.84.189.2" ] }'
netnamespace.network.openshift.io/openshift-gitops patched
Are the ArgoCD pods up and running?
The kubectl (Kubernetes) or oc (OpenShift) get pods command can be used to check if the ArgoCD pods in the cluster ArgoCD is running on are up and running.
By the way, notice in this example that the application controller pod has 13 restarts whereas the other pods have 0 restarts. This is because when an application would get stuck refreshing my application controller pod would run out of memory and start crash looping (more on this in a moment).
~]$ oc get pods
NAME READY STATUS RESTARTS AGE
cluster-f4c765c6f-tznxm 1/1 Running 0 3d15h
gitops-plugin-76c8d56cbc-xbdl7 1/1 Running 0 3d15h
kam-6fd58b8d87-ggrlr 1/1 Running 0 3d15h
openshift-gitops-application-controller-0 1/1 Running 13 (2d20h ago) 3d15h
openshift-gitops-applicationset-controller-6858947bc8-7nfhv 1/1 Running 0 3d15h
openshift-gitops-dex-server-5f8549dd48-24x72 1/1 Running 0 3d15h
openshift-gitops-redis-7bd479c78f-zb8xg 1/1 Running 0 3d15h
openshift-gitops-repo-server-694ddc84cc-66r2c 1/1 Running 0 3d15h
openshift-gitops-server-7dddf67589-g5nzq 1/1 Running 0 3d15h
Configure ArgoCD to log events at log level DEBUG
If there are no interesting events in the ArgoCD pod logs, you may want to update ArgoCD to log events at log level DEBUG. For example, here is how you could check to see if the ArgoCD repo server has ben configured with a logLevel. If there is no logLevel then this simply means that no log level has been set thus the resource (repo in this example) would log events at the default log level, which is INFO.
~]$ oc get argocd openshift-gitops --namespace openshift-gitops --output jsonpath="{.spec.repo}" | jq
{
"logLevel": "debug",
"resources": {
"limits": {
"cpu": "1",
"memory": "1Gi"
},
"requests": {
"cpu": "250m",
"memory": "256Mi"
}
}
}
For example, to update "repo" to log level debug. This should get the repo-server pod to log events at log level debug.
oc patch argocd openshift-gitops --namespace openshift-gitops --patch '{"spec":{"repo":{"logLevel":"debug"}}}' --type=merge
Test git commands in the ArgoCD Repo pod
Let's first define the REPO_POD variable.
~]# REPO_POD=$(oc get pod --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-repo-server --output custom-columns=POD:.metadata.name --no-headers)
The repo pod log should have something like this.
~]# oc logs $REPO_POD --namespace openshift-gitops
time="2025-02-27T02:11:22Z" level=error msg="`git fetch origin --tags --force --prune` failed timeout after 1m30s" execID=e3cd7
time="2025-02-27T02:11:22Z" level=info msg=Trace args="[git fetch origin --tags --force --prune]" dir=/tmp/_argocd-repo/b3b0c48d-bc6e-42a5-9aa1-01bea37758e4 operation_name="exec git" time_ms=129097.675457
time="2025-02-27T02:11:22Z" level=error msg="finished unary call with code Unknown" error="failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed timeout after 1m30s" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2025-02-27T02:09:13Z" grpc.time_ms=129101.086 span.kind=server system=grpc
The kubectl (Kubernetes) or oc (OpenShift) exec command can be used to run commands in the repo pod. By default the /tmp/_argocd-repo directory has d-wx--S--- permissions meaning the directory does not have the r (read) permission thus you will get permission denied if you attempt to list the contents of the /tmp/_argocd-repo directory.
~]$ oc exec $REPO_POD --container argocd-repo-server --namespace openshift-gitops -- /bin/bash -c "ls -l /tmp"
total 0
d-wx--S---. 3 1000850000 1000850000 50 Feb 27 17:24 _argocd-rep
Let's update the /tmp/_argocd-repo directory to have the r (read) permisison, list the contents of the /tmp/_argocd-repo directory, and then remove the r (read) permission. Notice in this example that the UUID b3b0c48d-bc6e-42a5-9aa1-01bea37758e4 is the same UUID listed in the repo pod log. /tmp/_argocd-rep/<uuid> is where the cached repos are stored.
~]$ oc exec $REPO_POD --container argocd-repo-server --namespace openshift-gitops -- /bin/bash -c "chmod a+r /tmp/_argocd-repo; ls -l /tmp/_argocd-repo; chmod a-r /tmp/_argocd-repo"
total 0
drwx------. 3 1000850000 1000850000 18 Feb 27 17:24 b3b0c48d-bc6e-42a5-9aa1-01bea37758e4
Let's issue the git fetch command in the repo pod. Notice in this example that I get more details on what the actual cause of the issue is, a failure to connect to proxy.example.com on socks5 port 1080 (more details on this below).
~]# REPO_POD=$(oc get pod --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-repo-server --output custom-columns=POD:.metadata.name --no-headers)
~]# oc exec $REPO_POD --container argocd-repo-server --namespace openshift-gitops -- /bin/bash -c "cd /tmp/_argocd-repo/b3b0c48d-bc6e-42a5-9aa1-01bea37758e4; git fetch origin --tags --force --prune"
fatal: unable to access 'https://github.com/foo/bar/': Failed to connect to proxy.example.com port 1080: Connection timed out
command terminated with exit code 128
Using dir from the repo pod log let's ensure the git remote command returns the expected repo URLs.
~]$ oc exec $REPO_POD --container argocd-repo-server --namespace openshift-gitops -- /bin/bash -c "cd /tmp/_argocd-repo/b3b0c48d-bc6e-42a5-9aa1-01bea37758e4/.git; git remote -v"
origin https://github.com/foo/bar (fetch)
origin https://github.com/foo/bar (push)
You should also be able to confirm that the config file contains the expected repo. This is the "origin" in the "git fetch origin" command.
~]$ oc exec $REPO_POD --container argocd-repo-server --namespace openshift-gitops -- cat /tmp/_argocd-repo/b3b0c48d-bc6e-42a5-9aa1-01bea37758e4/.git/config
[core]
bare = false
[remote "origin"]
url = https://github.com/foo/bar
fetch = +refs/heads/*:refs/remotes/origin/*
Likewise, the git config command should return the same "origin" as the .git/config file.
~]$ oc exec $REPO_POD --container argocd-repo-server --namespace openshift-gitops -- /bin/bash -c 'cd /tmp/_argocd-repo/b3b0c48d-bc6e-42a5-9aa1-01bea37758e4; git config --list'
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
core.bare=false
remote.origin.url=https://github.com/foo/bar
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
Let's say HEAD is refs/heads/main. This should mean that the "git fetch origin" command is attempting to fetch the "main" reference (e,g. git fetch origin main).
~]$ oc exec $REPO_POD --container argocd-repo-server --namespace openshift-gitops -- cat /tmp/_argocd-repo/b3b0c48d-bc6e-42a5-9aa1-01bea37758e4/.git/HEAD
ref: refs/heads/main
This should be an exact match of HEAD if you were to git clone the repo outside of the pod and cat the HEAD.
~]$ git clone git@github.com:foo/bar.git
~]$ cat bar/.git/HEAD
ref: refs/heads/main
I once had an issue where HEAD in the repo pod had refs/heads/master but when I cloned the repo on any random Linux server HEAD had refs/heads/main. I removed the repo from ArgoCD and re-added the repo using the --upsert flag to ensure any prior references of the repo in ArgoCD were updated.
argocd repo rm https://github.com/foo/bar.git
argocd repo add https://github.com/foo/bar.git --upsert
But HEAD in the repo pod still had refs/heads/master and the log (at logLevel debug) had this.
time="2025-02-14T02:46:56Z" level=debug msg="Revision 'HEAD' is not a valid semver constraint, skipping semver resolution."
time="2025-02-14T02:46:56Z" level=debug msg="symbolic reference 'HEAD' (refs/heads/main) resolved to '70da4f4e1b9011c1ae50c7ce2ab4ea1af34c0e43'"
Is proxy connect timeout?
I once saw the following when issue the git clone command in the repo pod.
~]$ REPO_POD=$(oc get pod --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-repo-server --output custom-columns=POD:.metadata.name --no-headers)
~]$ ADMIN_PASSWD=$(oc get secret openshift-gitops-cluster --namespace openshift-gitops --output jsonpath='{.data.admin\.password}' | base64 --decode)
~]$ URL=$(oc get routes --namespace openshift-gitops --selector=app.kubernetes.io/name=openshift-gitops-server --output jsonpath="{.items[*].spec.host}")
~]$ oc exec $REPO_POD --container argocd-repo-server --namespace openshift-gitops -- git clone https://<personal_access_token>@github.com/foo/bar.git
Cloning into 'bar'...
fatal: unable to access 'https://github.com/foo/bar.git/': Failed to connect to proxy.example.com port 1080: Connection timed out
Notice in this example that the error has port 1080 (Failed to connect to proxy.example.com port 1080). This means the socks5 protocol is being used, like this.
socks5://proxy.example.com
Let's say your ArgoCD repo pod contain proxy environment variables, perhaps something like this.
~]$ oc get pod openshift-gitops-repo-server-7744cbb479-7w8xd --output yaml
spec:
containers:
env:
- name: HTTPS_PROXY
value: https://proxy.example.com
- name: HTTP_PROXY
value: http://proxy.example.com
- name: NO_PROXY
value: .cluster.local,.example.com,access.redhat.com,localhost,quay.io,registry.connect.redhat.com,registry.redhat.io
Likewise your ArgoCD deployment may also contain proxy environment variables.
~]$ oc get deployment openshift-gitops-server --output yaml
spec:
containers:
env:
- name: HTTPS_PROXY
value: https://proxy.example.com
- name: HTTP_PROXY
value: http://proxy.example.com
- name: NO_PROXY
value: .cluster.local,.example.com,access.redhat.com,localhost,quay.io,registry.connect.redhat.com,registry.redhat.io
Or you may have added the repo to ArgoCD including proxy.
argocd repo add git@github.com:foo/bar.git \
--ssh-private-key-path ~/.ssh/id_rsa \
--name my-repo \
--project default \
--proxy https://proxy.example.com
The env or printenv command in the ArgoCD pods should include HTTP_PROXY, HTTPS_PROXY and NO_PROXY.
~]$ oc exec $REPO_POD --container argocd-repo-server --namespace openshift-gitops -- env | grep -i proxy
HTTPS_PROXY=http://proxy.example.com
HTTP_PROXY=http://proxy.example.com
NO_PROXY=.cluster.local,.example.com,access.redhat.com,localhost,quay.io,registry.connect.redhat.com,registry.redhat.io
And if you try to remove the proxy environment variables from the deployment the proxy environment variable may immediately be re-added to the deployment. This probably means that your Kubernetes or OpenShift cluster has a proxy resource. The kubectl (Kubernetes) or oc (OpenShfit) get proxy command can be used to determine if the cluster has a proxy resource. In this example, there is a proxy resource named "cluster".
~]$ oc get proxy
NAME AGE
cluster 622d
And the proxy resource should contain the same proxy environment variables as listed in the ArgoCD pods and deployments.
~]$ oc get proxy cluster --output yaml
spec:
httpProxy: http://proxy.example.com
httpsProxy: https://proxy.example.com
noProxy: .cluster.local,.example.com,access.redhat.com,localhost,quay.io,registry.connect.redhat.com,registry.redhat.io
So one option here is to use the kubectl (Kubernetes) or (OpenShift) edit proxy cluster command to add the repo base URL such as github.com to the noProxy list in the proxy cluster resource. After doing this it may take a few moments for the change to complete since this is a cluster wide configuration change.
oc edit proxy cluster
You may need to delete the ArgoCD repo pod so that a new deployment gets spawned and the new deployment should contain the updated noProxy list.
oc delete deployment openshift-gitops-repo-server
Or when adding the repo to ArgoCD you can use the --no-proxy option so that requests to the repo's base URL such as github.com do not use the proxy.
argocd repo add https://github.com/foo/bar.git --no-proxy github.com
Did you find this article helpful?
If so, consider buying me a coffee over at 