Let's say you have deployed an application to OpenShift, and when requesting the application, Application is not available is being returned.

An OpenShift route or an Ingress route will provide a URL such as http://my-route-my-project.apps.openshift.example.com which is used to route a request onto a service, which is then routed onto a pod, and then to the container in the pod, and finally to the application running in the container.

Are you using the correct URL?

Before we get into a bunch of troubleshooting, let’s just make sure the URL being requested is correct. If you are using the command line, the oc get routes command can be used to list the routes in the project / namespace that contains the pod that is returning “Application is not available”. In this example, the URL would be https://my-route-my-project.apps.openshift.example.com.

~]# oc get routes --namespace my_project
NAME          HOST/PORT                                         PATH   SERVICES     PORT     TERMINATION          WILDCARD
my-route      my-route-my-project.apps.openshift.example.com           my-service   8080                          None

Deploy the hello-openshift pod

If find it makes the most sense to first try to submit a request using the socket (IP address and port) of the pod, because if the request isn't working using the IP address and port of the pod, then likewise the request using the Service or Route will probably also not work either. 

Be aware that many pods use /bin/busybox and /bin/busybox does not contain curl. The hello-openshift image contains curl. Check out my article Check out my article FreeKB - OpenShift - Deploy Hello Openshift.

This also will be useful to determine if communication can occur across different projects / namespaces.

~]# oc new-project hello-openshift

~]# oc import-image openshift4/ose-hello-openshift-rhel8:v4.7.0-202205312157.p0.g7706ed4.assembly.stream --from=registry.redhat.io/openshift4/ose-hello-openshift-rhel8:v4.7.0-202205312157.p0.g7706ed4.assembly.stream --confirm

~]# oc new-app registry.example.com/openshift4/ose-hello-openshift-rhel8

~]$ oc exec pod/ose-hello-openshift-rhel8-5959c4fb77-zss6g -- curl --silent localhost:8080
Hello OpenShift!

Or if you are using the OpenShift console, in the left panel select Home > Projects, and select the hello-openshift project / namespace. Then select the hello-openshift pod, select the Terminal tab, and issue command curl localhost:8080 and “Hello OpenShift!” should be returned.

Network Policies

The oc get networkpolicies command can then be see if there are any ingress network policies denying incoming requests in the project / namespace that contains the pod that is returning "Application is not available".

There is no need to use the oc get egressnetworkpolicies command since this issue is with ingress requests, not egress requests.

~]$ oc get networkpolicies
No resources found in my_project namespace.

NetNamespaces

The oc get netnamespaces can be used to see if the project / namespace that contains the pod that is returning "Application is not available" has a dedicated egress IP address.

~]$ oc get netnamespaces my_project
NAME               NETID     EGRESS IPS
my_project         9740194

If not, you may need to patch the project / namespace to have a dedicated egress IP address.

~]$ oc patch netnamespace my_project --type merge --patch '{ "egressIPs": [ "10.84.189.2" ] }'
netnamespace.network.openshift.io/my_project  patched

Curl from Pod to Pod

The oc get pods command can be used to list the pods in project / namespace that contains the pod that is returning "Application is not available".

~]# oc get pods
NAME            READY   STATUS     RESTARTS  AGE
my-pod-abcde    1/1     Running    0         8d

And the oc describe command can be used to return the IP address the pod is listening for connections on.

~]$ oc describe pod my-pod-abcde | grep ^IP:
    IP: 10.11.12.13

And the oc describe command can be used to return the ports the pod is listening for connections on.

~]$ oc describe pod my-pod-abcde | grep -i Ports
    Ports:          8080/TCP, 8443/TCP

Or if you are using the OpenShift console, in the left panel select Home > Projects, and select the project / namespace that contains the pod that is returning "Application is not available". Then select the pod that is returning "Application is not available", select the YAML tab, and the YAML output should contain the IP Address and ports the pod is listening for connections on.

Be aware that even if the pod is allowing connections on the expected ports (8080 and 8443 in this example) this does not mean that the app running in the container in the pod is listening for connections on the expected ports. This is a bit more trickly to isolate. Sometimes, the oc get logs command may help you determine if the app in the container in the pod is or is not listening for connections on the expected ports. For example, I once saw that Tomcat running in the pod started on ports 8080 and 8081 but not on 8443. To resolve this, Tomcat in the app was updated to listen on port 8443 and then redeployed to OpenShift.

~]$ oc logs pod/my-pod-abcde --container my-container | grep -i "tomcat start"
2024-08-19 22:27:37.029 INFO 7 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (https) with context path '/foo/bar'
2024-08-19 22:27:45.016 INFO 7 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8081 (http) with context path ''

Assuming there are no ingress network polices in the project / namespace that contains the pod that is returning "Application is not available" denying traffic, let's use the hello-openshift pod to submit a request to the pod that is returning "Application is not available" using curl in the hello-openshift pod and using the socket (IP address and port) of the pod that is returning "Application is not available". 

In this example I get a response to the GET request. It doesn't really matter what the response is. You just want to see if you get a response. Good - we know the application in the pod is able to respond to GET requests.

~]$ oc exec pod/ose-hello-openshift-rhel8-5959c4fb77-zss6g --namespace hello-openshift -- curl 10.11.12.13:8080 -v
> GET / HTTP/1.1
> Host: 10.11.12.13:8080
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 200
< Cache-Control: private, no-cache, no-store, must-revalidate, max-age=0
< Content-Type: application/json
< Date: Thu, 03 Oct 2024 01:08:04 GMT
<
Hello World

And here is an example of what this could look like when using the Terminal in the hello-openshift pod using the OpenShift console.

If you get something like "Connection refused", this may mean that the app in the pod is unable to respond to your GET request. At this point, you'll need to look into this to determine why the app in the pod is unable to respond to your GET request.

~]$ oc exec pod/ose-hello-openshift-rhel8-5959c4fb77-zss6g --namespace hello-openshift -- curl 10.11.12.13:12345 -v
* Failed to connect to 10.11.12.13 port 12345: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 10.11.12.13 port 12345: Connection refused
command terminated with exit code 7

Curl from Pod to Service

If you are able to get a response from the pod that is returning "Application is not available", I would then next see if I can get a response using the Service that forwards requests onto the pod.

The oc get services command can be used to list the services in the currently selected project / namespace. If you need to create the service, the oc expose pod or oc create service command can be used to create the service. 

~]$ oc get services
NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
my-service    ClusterIP   172.30.166.121   <none>        12345/TCP   3m24s

The oc describe service command can be used to show details about the service.

In this example, my-service is listening for connections on port 12345 and forwards request onto the pod that has label app=my-app which means that the service will only forward requests onto pods that have label app=my-app. Notice also that Endpoints contains an IP address and port, 10.11.12.13:8080. This means that there are one or more pods that have app=my-app thus the service will be able to forward requests onto the pod that has IP address 10.11.12.13 on port 8080.

~]$ oc describe service my-service
Name:              my-service
Namespace:         my-project
Labels:            <none>
Annotations:       <none>
Selector:          app=my-app
Type:              ClusterIP
IP:                172.30.40.50
IPs:               172.30.40.50
Port:              http 12345/TCP
TargetPort:        8080/TCP
Endpoints:         10.11.12.13:8080
Session Affinity:  None
Events:            <none>

Notice in this example that the pod has label app: my-app and also has IP address 10.11.12.13 and port 8080, thus this is the pod the service will forward requests onto.

~]$ oc get pod my-pod-abcde --output jsonpath="{.metadata.labels} {.status.podIPs} {.spec.containers[*].ports}" | jq
{
  "app": "my-app"

}
[
  {
    "ip": "10.128.18.55"
  }
]
[
  {
    "containerPort": 8080,
    "protocol": "TCP"
  },
]

Assuming there are no ingress network polices in the project / namespace that contains the pod that is returning "Application is not available" denying traffic, let's use the hello-openshift pod to submit a request to the Service that is forwarding requests onto the pod that is returning "Application is not available" using curl in the hello-openshift pod and using the socket (IP address and port) of the Service. 

In this example I get a response to the GET request. This means I am able to connect to the pod that is returning "Application is not available" using the Service. Good - we know the Service is able to forward requests onto the pod and the application in the pod is able to response to GET requests.

~]$ oc exec pod/ose-hello-openshift-rhel8-5959c4fb77-zss6g --namespace hello-openshift -- curl --silent 172.30.40.50:12345
> GET / HTTP/1.1
> Host: 10.11.12.13:8080
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 200
< Cache-Control: private, no-cache, no-store, must-revalidate, max-age=0
< Content-Type: application/json
< Date: Thu, 03 Oct 2024 01:08:04 GMT
<
Hello World

From Route to Service to Pod

If you are able to get a response from the pod that is returning "Application is not available" using the Service, I would then next see if I can get a response using the Route that forwards requests onto the pod.

The oc get routes command can be used to list the routes in the current selected project / namespace. If you need to create a route

• the oc expose service command can be used to create an insecured HTTP route
• the oc create route command can be used to create a secured HTTPS edge, passthrough or reencrypt route

~]# oc get routes
NAME          HOST/PORT                                         PATH   SERVICES     PORT     TERMINATION          WILDCARD
my-route      my-route-my-project.apps.openshift.example.com           my-service   8080                          None

The oc describe route command can be used to display more details about the route. By default, insecured routes will listen for HTTP requests. For example:

• http://my-route-my-project.<ingress domain name>

By default, secured routes will listen for HTTPS requests. For example:

• https://my-route-my-project.<ingress domain name>

The following command can be used to display your OpenShift ingress domain name.

~]$ oc get ingresses.config/cluster -o jsonpath={.spec.domain}
apps.openshift.example.com

Thus the HTTP and/or HTTPS URLs would be something like this in this example.

• http://my-route-my-project.apps.openshift.example.com
• https://my-route-my-project.apps.openshift.example.com

In this example, the route will forward requests onto my-service. 

• Notice that TLS Termination is <none> - This means this is an insecured HTTP route
• Notice also that the Endpoint Port is 8080. This is the port the pod is listening for connections on and also the TargetPort of the Service.

If the Route has "Endpoints: <none>" this means there is no endpoint that the Route can forward requests onto. On the other hand, if the Route has something like "Endpoints: 10.11.12.13:8080" this means the Route is properly configured to forward requests onto the pod with IP address 10.11.12.13 and listening for connections on port 8080. 

Additionally, you will want to ensure that the route is exposed on one of the OpenShift routers. In this example, the route is "exposed on router default"

~]$ oc describe route my-route
Name:                   my-route
Namespace:              my-project
Created:                17 minutes ago
Labels:                 <none>
Annotations:            openshift.io/host.generated=true
Requested Host:         my-route-my-project.apps.openshift.example.com
                           exposed on router default (host router-default.apps.openshift.example.com) 17 minutes ago
Path:                   <none>
TLS Termination:        <none>
Insecure Policy:        <none>
Endpoint Port:          8080

Service:        my-service
Weight:         100 (100%)
Endpoints:      10.11.12.13:8080

Likewise, this route is similar except that it is an HTTPS secured "edge" route.

~]$ oc describe route my-edge-route
Name:                   my-edge-route
Namespace:              my-project
Created:                17 minutes ago
Labels:                 <none>
Annotations:            openshift.io/host.generated=true
Requested Host:         my-edge-route-my-project.apps.openshift.example.com
                           exposed on router default (host router-default.apps.openshift.example.com) 17 minutes ago
Path:                   <none>
TLS Termination:        edge
Insecure Policy:        <none>
Endpoint Port:          8080

Service:        my-service
Weight:         100 (100%)
Endpoints:      10.11.12.13:8080

The OpenShift router pods are in the openshift-ingress project / namespace.

~]$ oc get pod --namespace openshift-ingress
NAME                             READY   STATUS    RESTARTS      AGE
router-default-f6d44996c-sljgl   1/1     Running   2 (58m ago)   56d

The router pods run an haproxy (high availability proxy) load balancer. The oc exec command can be used to determine if the pod/service have been exposed on haproxy.

~]$ oc exec pod/router-default-76c5c89559-dclkw --namespace openshift-ingress -- cat /var/lib/haproxy/conf/haproxy.config | grep my-route
  server pod:my-pod:my-service:8080-tcp:10.129.7.69:8080 10.129.7.69:8080 cookie 15b9071e86b87a2b4b6f976291de96cf weight 256 check inter 5000ms

Assuming there are no ingress network polices in the project / namespace that contains the pod that is returning "Application is not available" denying traffic, let's use a web browser such as Microsoft Edge to see if the request using the Route "makes it" to the Pod.