Enable verbose garbage collection
The native_stderr.log and native_stdout.log are WebSpheres verbose garbage collection logs. These logs are created when verbose garbage collection is enabled, by navigating to Servers > Server Types > your_JVM > Java and Process Management > Process Definition > Java Virtual Machines. After enabling verbose garbage collection, restart the JVM for this change to take effect.
Rotate the verbose garbage collection log
By default, the verbose garbage collection logs will not rotate, which will cause the log to become very large. A very large log is nearly impossible to work with. For example, in the screen shot below, there was an attempt to load a 124 MB log in IBMs Garbage Collection and Memory Visualizer tool. The tool started to hang at near 60 MB, and would take hours to load the log. The loading of the log should only take a few moments.
To configre the logs to be rotated, in the WebSphere admin console, select Servers > Server Types > your_JVM > Java and Process Management > Process Definition > Java Virtual Machines. In Generic JVM arguments, enter -Xverbosegclog:home/profiles/your_profile/logs/your_server/native_stderr.%Y%m%d.%H%M%S.%pid.txt,14,100000. This will configure your system to only retain a total of 14 native_stderr.logs. A new native_stderr.log will be created once every 100000 garbage collection cycles.
Restart the JVM for this change to take effect.
Another option (and probably the preferred option), is to use LogRotate (on Linux).
Analyze the verbose garbage collection log
To analyze a log, use IBMs Garbage Collection and Memory Visualizer, which is part of IBMs Support Assistant.
- In the IBM Support Assistant, select the Tools tab.
- Select Garbage Collection and Memory Visualizer Desktop.
- Select Launch.
Analyze the log. It is usually a good idea to compare at least a few of the log's, to see if the tuning recommendations in each log are similar or different.
- In the Garbage Collection and Memory Visualizer tool, select File > Load File.
- Select the log file you will like to analyze.
- At the bottom of the GUI, select the Report tab.
Tunng recommendations will be displayed. Following are the steps for known recommendations.
Recommendation: Your application appears to be leaking memory. This is an indication by the used heap increasing at a greater rate than the application workload (measured by the amount of data freed). To investigate further see IBM SDK User Guide.
What you should do: TBD
Recommendation: Excessive time (x.xx%) is being spent in GC. Consider increasing the size of the heap.
What you should do: The heap size determines how often and how long a JVM does garbage collection. When the maximum heap size is large, garbage collection will occur less frequently but will take longer. When the maximum heap size is small, garbage collection will occur more often but will take less time. Garbage collection negatively impacts a JVMs performance, so the goal is to find the heap size that cause the least performance hit. Typically, the proportion of time spent in garbage collection pauses should be less than 1%.
If the proportion of time spent in garbage collection pauses is greater than 1%, determine the steady state after the JVM is restarted. For this, you will need to get the native_stderr.log when the JVM was last restarted. In this example, the JVM was restarted around 3:00 am, and the steady state is near 250 MB, which means a minimum heap size of 256 MB would be appropriate.
Since a large maximum heap size will cause garbage collection to occur less often but will take longer, you don't want to set the maximum heap size to be unnecessarily large. What you want to do is to set the maximum heap to be larger than the minimum heap, and to then measure JVM performance.
Recommendation: At one point ### objects were queued for finalization. Using finalizers is not recommended as it can slow garbage collection and cause wasted space in the heap. Consider reviewing your application for occurrences of the finalize() method. You can use IBM Monitoring and Diagnostic Tools - Memory Analyzer to list objects that are only retained through finalizers.
What you should do: Determine if the applications running on the JVM are using the finalize() method. If they are, consider updating the apps to not use the finalize() method.
The number of objects queued for finalization can be visualized by selecting the Line plot tab, then select the Data selector tab, and then check Objects queued for finalization. In this example, the blue line near the top of the chart is the maximum heap size, the line with the saw tooth pattern is the heap usage, and the dark blue line at the bottom are the number of objects queued for finalization.
Recommendation: # global garbage collects took on average ###% longer than the average nursery collect. If you believe this is abnormally high and unacceptable, consider using the Balanced GC policy for applications deployed on a 64-bit platform with a heap size greater than 4GB.
What you should do: Since the recommendation here is to consider using the Balanced GC policy, first determine the current policy. This can be seen on the Report tab. In this example, the policy is gencon.
The GC policy can be changed in the WebSphere admin console by navigating to Servers > Server Types > your_JVM > Java and Process Management > Java Virtual Machines. Set the -Xgcpolicy to balanced, and then restart the JVM for this change to take effect.