Perm gen is short for permanent generation. Perm gen is used with version 7 of Java, and metaspace is used with version 8 of Java. Perm gen / metaspace is memory that is allocated for certain tasks, mostly notably, keeping a declaration of the classes being used by the Java application. The following JVM arguments are used to configure the max perm gen / metaspace size for the JVM (256 MB in this example). If you change the perm gen / metaspace size, you will need to restart the JVM for the change to take effect.

# Initial / Minimum Perm Gen Size
-XX:PermSize=256m

# Maximum Perm Gen Size
-XX:MaxPermSize=256m

# Maximum Metaspace Size
-XX:MaxMetaspaceSize=256m

What causes "Out of Memory"

If the JVM requires more perm gen / metaspace memory than has been allocated, an out of memory event will appear in the catalina.out log. A JVM will run out of perm gen / metaspace when the classes of the applications in the JVM being loaded into metaspace requires more memory than the max metaspace memory value. For example, if the JVM is configured with 256 MB of max perm gen / metaspace, and more than 256 MB of memory is required to load the classes in the apps in the JVM, the JVM will run out of metaspace memory.

java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: Metaspace

What causes a heap dump

A heap dump will occur after an JVM runs out of memory. There will be a heap dump event in the catalina.out log.

Dumping heap to /path/to/java_pid12345.hprof ...

First app deploy

If a JVM run out of perm gen / metaspace after the first deploy of the application to the JVM, this implies that the JVMs max perm gen / metaspace needs to be increased to be able to load the classes in the app into memory.

App re-deploys (memory leak)

When an app is redeployed to a JVM, the prior version of the app will be undeployed. If the undeploy fails to remove threads from perm gen / metaspace, when the app is redeployed, new threads for the same object will be created, which will cause the JVM to have to use more perm gen / metaspace memory than should be needed. You should be able to determine if this is happening in the JVMs catalina.log. After the undeploy, look for events that contain "This is very likely to create a memory leak".

22-Oct-2018 09:37:50.170 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [myApp] appears to have started a thread named [JMSCCThreadPoolMaster] but has failed to stop it. This is very likely to create a memory leak.

For example, let's say an app is using about 200 MB of perm gen / metaspace.

During the undeployment of the app, if some of the threads are not removed, they will continue to use perm gen / metaspace.

Then when the app is redeployed, all of the perm gen / metaspace is used.

This is particularly problematic if the app is being deployed to the JVM over and over again, thus it is usually a good idea to determine how many times the application was deployed since the JVM was last restarted. Thus it is a good idea to check catalina.out to determine when the JVM was last restarted.

mm dd, yyyy hh:mm:ss org.apache.catalina.startup.Catalina start
INFO: Server startup in 23615 ms

Check the catalina.log to determine how many times the application was deployed since the last JVM restart. It is important to recognize that when a JVM is restarted, the restart will include a deployment of the application.

mm dd, yyyy hh:mm:ss org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive /path/to/example.war

When does it make sense to increase the max perm gen / metaspace

• If the JVM runs out of perm gen or metaspace memory as part of a JVM restart, it makes sense to increase the max perm gen / metaspace.
• If the JVM runs out of perm gen or metaspace memory after a deploy of the app, and there are no "memory leak" events in catalina.out or catalina.log, it makes sense to increase the max perm gen / metaspace.
• If the JVM runs out of perm gen or metaspace memory after a deploy of the app and there are "memory leak" events in catalina.out or catalina.log, it probably does not make sense to increase the max perm / gen or metaspace. Instead, you will want to determine why there are "memory leak" events in catalina.out or catalina.log.