A JVM will have a single process, and the process identification number (PID) can be found using the ps command (Linux). A process will contain numerous threads. Some examples of threads are the JVMs garbage collector, a code optimizer thread, and a finalizer thread.
ps -ef | grep jvm_name
A WebSphere JVM can be configured to write hung thread events to the JVMs SystemOut.log.
- In the left panel of the WebSphere admin console, select Servers > Server Types > WebSphere application servers.
- Select an application server.
- Expand Administration and select Custom properties.
- Select New, and create the two properties in the table below. This two properties are required to write hung thread events to the JVMs SystemOut.log.
- Select Save.
- Restart the application server for this change to take effect.
|com.ibm.websphere.threadmonitor.interval||integer||Number of seconds between each interval to see if there are any hung threads, such as 30 (seconds).|
|com.ibm.websphere.threadmonitor.threshold||integer||Number of seconds that must elapse before a thread is considered hung, such as 60 (1 minute).|
Using the properties in the above table, a thread will be consider hung when the thread has been active for 10 minutes. As an example, a thread may be hung when an application issues a SQL request, and the database does not issue a response in a timely manner. Hung threads are problematic, because they use system resources (CPU, memory), and usually, the thread is unintentially hung, which means system resources are unnecessarily being used.
If you want the JVM to create a javacore dump when a hung thread is detected, add the following custom property. The javacore dump can be analyzed using IBMs Thread and Monitor Dump Analyzer tool, which is part of IBMs Support Assistant.
|com.ibm.websphere.threadmonitor.dump.java||1||Create a javacore dump when a hung thread is detected.|
Replicate hung thread
If you want to create a servlet in Eclipse that will produce a hung thread, refer to this article.
In the SystemOut.log, a hung thread will be identified by event code WSVR0605W, and when the thread is no longer hung, event WSVR0606W will be found in the log. When hung threads are detected, the number of hung threads will be identifed, along with how long the threads have been hung. In this example, there is 1 hung thread 655526 milliseconds, which is about 10 minutes.
WSVR0605W: Thread "WebContainer : 1" (000000a1) has been active for 655526 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
The SystemOut.log may list events that correlate to the hung thread. In this example, you would look for events near 10 minutes prior to the hung thread event. The list of possible causes of hung threads is far to vast to list. Each hung thread will need to be looked at. However, some common causes of hung threads are long database connections, some sort of system scan (such as anti-virus), or some sort of long running batch job. Not all hung threads are necessarily bad - for example, an anti-virus scan may be an OK reason for a long running thread.
Once the threads are no longer hung, the SystemOut.log will list the following event.
WSVR0606W: Thread "WebContainer : 1" (000000a1) was previously reported to be hung but has completed.
IBMs Support Assistant Thread and Monitor Dump Analyzer tool can also be used to spot hung threads.
Hung threads can cause an application to be totally unresponsive, which means that hung thread can be a significant issue, especially with enterprise production applications. For this reason, you will usually want to set up a system to alert your organizations WebSphere on-call person when there is a hung thread, so that hung threads can be addressed with priority. How to create the email alerts are outside of the scope of this article. However, one option would be to create a script (Python Perl Bash PowerShell) that would look for WSVR0605W in the SystemOut.log, and then to send an email if the event is found.