Analyze Memory Problems

Learn how to identify the cause of Java application memory issues. You can generate a heap dump automatically and manually, and a heap histogram, to identify the cause.

Description description

Environment

Experience Manager

Issue/Symptoms

The JAVA application runs slower and finally runs out of memory or you see an error in the logs or in the console output OutOfMemoryError: Java heap space or OutOfMemoryError: gc overhead limit exceeded.

Resolution resolution

Cause
Such problems can have many causes.

One possible cause is that the java application, in our case, CRX / CQ was started from the command line with the default heap memory settings of Java. This means that the jvm parameter -Xmx was not specified. CRX or CQ need at least 256 MB of heap allocated to run. If this is the problem, then starting from the command line,ensure that the heap memory settings are set. Example:

java -Xmx512m -jar *.jar

If this is not the case, then your application could be retaining too many objects without releasing them for Garbage Collection. This is called a memory leak, see here for more information. See the section below on how to analyze memory problems in java applications.
Create heap dump:

Generate a heap dump automatically

To automatically create a heap dump when running out of memory, you can add the jvm parameter -XX:+HeapDumpOnOutOfMemoryError to generate a heap dump automatically when the application throws an OutOfMemoryError. For example,

java -Xmx256m -XX:+HeapDumpOnOutOfMemoryError -jar *.jar

This generates a heap dump file (java_...hprof) in the process’s working directory whenever the java process runs out of memory. The process can continue to run after the heap dump was generated. Usually, one heap dump file is enough to analyze the problem.

Note:  if you are using crx-quickstart/server/start script to start your CRX instance then you can add -XX:+HeapDumpOnOutOfMemoryError to the CQ_JVM_OPTS variable (make sure that the variable is uncommented as well). For example:

CQ_JVM_OPTS='-XX:+HeapDumpOnOutOfMemoryError'

After adding this parameter and restarting the CRX instance, verify that the new jvm option is set. Run ps -ef | grep java from the command-line. Then check whether you see -XX:+HeapDumpOnOutOfMemoryError as a parameter of the CRX java process.

If you need to specify a different directory to generate the heap dump in due to disk space restrictions, then you can add the -XX:HeapDumpPath=/path/to/generate/heapdump parameter to tell the jvm where to put the file.

See here for a reference of debugging related jvm parameters.
Generate a heap dump manually
Sun/Oracle JVM

To manually generate a heap dump, run this command (jmap and jps can be found in the bin folder of your jdk home directory):

IBM JVM

You’ll first have to change the default JVM settings regarding the Dump agents, to generate the correct dumps on user signal. There are several kinds of dumps, but you generally need the full  system dump  to perform a thorough memory analysis. Add the following arguments:

Xdumpopts=PHD+CLASSIC:events=user -Xdumpevents=user

This “user” event happening when the JVM receives the SIGQUIT (Linux, AIX®, z/OS®, and i5/OS™) or SIGBREAK (Windows) signal from the operating system.

For more information, see the vendor’s documentation here.

Warning:  Heap dump files are large and can be up to the same size on disk as your max heap -Xmx jvm parameter configuration. Make sure that you have enough disk space allocated to the directory where the dump file is generated.
Analyze heap dump
A good tool to analyze heap dumps is EclipseMAT (Eclipse Memory Analyzer).

This tool can’t analyze  IBM JVM  generated dumps. For those, there are several possibilities. IBM HeapAnalyzer will do well for the heap dumps in PHD or classic format.

For a complete system dump analysis, use the IBM Support Assistant Workbench, with IBM Monitoring and Diagnostic Tools for Java - Memory Analyzer installed on top. The heap histogram is a simple measurement of the number of live objects and memory used per Java class. Unfortunately, depending on the Java installation, the required tools may not be available or may not always work. To create a heap histogram, you first need the process id of the Java process. To get it, run ps or (if available), run:Heap Histogram
The heap histogram is a simple measurement of the number of live objects and memory used per Java class. Unfortunately, depending on the Java installation, the required tools may not be available or may not always work. To create a heap histogram, you first need the process id of the Java process. To get it, run ps or (if available), run:

jps -l

This Java tool gets the process ids of all running Java processes. Example:

327
3332 sun.tools.jps.Jps
3313 crx-quickstart-....jar

Now run the following command:

jmap -histo 3313

The list is sorted by total memory required (shallow: excluding referenced objects). The first 20 lines of the output are the most interesting. Example output:

JVM version is 1.5.0_20-141
Iterating over heap. This may take a while...
Warning: skipping invalid TLAB for thread t@62211
Warning: skipping invalid TLAB for thread t@62467
...
SizeCountClass description
-------------------------------------------------------
1059290412916byte`[` `]`
1028584075255* ConstMethodKlass
628317658388char`[` `]`
604230414928int`[` `]`
4995752116201* SymbolKlass
422089675255* MethodKlass
41965126969* ConstantPoolKlass
29285606969* InstanceKlassKlass
26310086066* ConstantPoolCacheKlass
2395872149742org.apache.jackrabbit.core.query.lucene.DocId$PlainDocId
14760087003java.util.HashMap$Entry`[` `]`
139612858172java.lang.String
107023244593java.util.HashMap$Entry
75398410036short`[` `]`
73546454org.apache.jackrabbit.core.query.lucene.DocId`[` `]`
7201927502java.lang.Class
64070413348com.day.crx.persistence.tar.index.IndexEntry
...

Additional information

To help analyzing the problem, we also need to know the following information:

References

[ 1] Oracle Help Center > Troubleshooting Guide for HotSpot VM
[ 2]  Oracle.com > javas > DebuggingOptions