Get an .hprof file (Java heap dump) from a service in Puppet Enterprise 2016.2 to 2018.1

When you work with Support to troubleshoot JVM-related memory issues, we might ask you to send us an .hprof file (Java heap dump). A heap dump takes a snapshot of JVM objects and classes in the heap at the time of capture.

Note:

Since a heap dump contains everything in memory at the time it is created, be aware that this file might include sensitive information, including passwords.

This article will help you do the following:

• Get a heap dump from a running service without waiting for an OOM (out of memory) error. This can be useful in some development and debugging situations.

  or

• Get information about why crashes are happening by turning on the setting in Puppet Enterprise that triggers a heap dump during a service crash. By default, this setting is turned off, since heap dumps are very large (multiple GBs), and might fill up the hard drive of the infrastructure node they're generated on (master, compile master, PuppetDB, or console node). When you're finished debugging, turn off the setting triggering heap dumps during crashes using the steps in this article.

For more information about how to use Java heap dumps for debugging, read Chris Price's blog post Puppet Server: advanced memory debugging.

Version and installation information

PE version: 2016.2 to 2018.1
OS: Any *nix
Installation type: Any

Solution

For PE 2018.1 and later: Install Java development tools

In versions of PE earlier than 2018.1, the Java development tools needed for a heap dump were installed by default. If you are using PE 2018.1 or later, you must install the tools before you can get a heap dump.

1. Install the tools on the node you'd like to get a heap dump from. Run:

puppet resource package pe-java-devel ensure=installed

Get a heap dump from a running service without waiting for an OOM

This command use jmap:dump with the live option which triggers a full garbage collection. If you are diagnosing a memory leak, use the live option. If you are diagnosing large objects committed to the heap or want to see what’s in the heap, do not use the live option.

Run the following on the appropriate node, substituting the appropriate values from the table below for <NAME OF SERVICE>:

export DUMP_SERVICE=<NAME OF SERVICE> ; su - $(ls -ld /proc/$(cat /var/run/puppetlabs/$DUMP_SERVICE/$DUMP_SERVICE.pid) |awk '{print $3}') -s /bin/bash -c "$(find /opt/puppetlabs -name jmap) -dump:live,format=b,file=./jvm_heap_dump.hprof $(cat /var/run/puppetlabs/$DUMP_SERVICE/$DUMP_SERVICE.pid)"

For example, to get a heap dump from Puppet Server, run the following on your master node:

export DUMP_SERVICE=puppetserver ; su - $(ls -ld /proc/$(cat /var/run/puppetlabs/$DUMP_SERVICE/$DUMP_SERVICE.pid) |awk '{print $3}') -s /bin/bash -c "$(find /opt/puppetlabs -name jmap) -dump:live,format=b,file=./jvm_heap_dump.hprof $(cat /var/run/puppetlabs/$DUMP_SERVICE/$DUMP_SERVICE.pid)"

Turning on the Java heap dump setting for PE services

The process to turn heap dumps on for each of four services (PE console services, PE Puppet Server, PE orchestration, and PuppetDB) is very similar, so just choose the appropriate variables as you work through. The process uses jmap:dump with the live option which triggers a full garbage collection. If you are diagnosing a memory leak, use the live option. If you are diagnosing large objects committed to the heap or want to see what’s in the heap, do not use the live option.

1. From the console, navigate to the Classification page (Nodes > Classification for PE version prior to 2017.3) and select the relevant node group:

   Heap dump for	PE console services	PE Puppet Server	PE orchestration	PuppetDB
   Node group	PE Console	PE Master	PE Orchestrator	PE PuppetDB

2. Click Classes and scroll down to the relevant class:

   puppet_enterprise::profile::<class name>

   Heap dump for	PE console services	PE Puppet Server	PE orchestration	PuppetDB
   Class name	console	master	orchestrator	puppetdb

3. Click the Parameter name drop-down list and select java_args. Edit the parameter value to add "XX:+HeapDumpOnOutOfMemoryError" : "", "XX:HeapDumpPath": "=<filepath>", and "Djruby.reify.classes":"=true".

   For example: {"Xmx": "4096m", "Xms": "4096m", "XX:+HeapDumpOnOutOfMemoryError" : "", "XX:HeapDumpPath": "=/var/log/puppetlabs/puppetserver/puppetserver.hprof","Djruby.reify.classes":"=true"}

   If the HeapDumpPath is a file, it will be overwritten with each dump, and will thus limit the amount of drive space consumed. If the path is a directory, a new .hprof file will be generated on each out of memory event. If you'd like to capture a series of heap dumps, use a directory instead of a file path.

   Note: Add XX:+HeapDumpOnOutOfMemoryError, XX:HeapDumpPath and Djruby.reify.classes only. Values for Xmx and Xms (memory settings) are shown as an example.

4. Click Add Parameter, and then click the commit button.

5. In the command line on the relevant node, run puppet agent -t to start a Puppet run and apply the change.

   Heap dump for	PE console services	PE Puppet Server	PE orchestration	PuppetDB
   Node	Console node (master in a monolithic installation)	Master node, compile master nodes	Master node	PuppetDB node (master in a monolithic installation)

   Note: On any master node where you'd like to capture a .hprof file for PE Puppet Server, run puppet agent -t. Usually, you'd only want to run it on the master or master of masters, but you can also run it on compile masters.

Turning off the Java heap dump setting for PE services:

As with turning on heap dumps, the process to turn heap dumps off for each of four services (PE console services, PE Puppet Server, PE orchestration, and PuppetDB) is very similar, so just choose the appropriate variables as you work through.

1. From the console, navigate to the Classification page (Nodes > Classification for PE version prior to 2017.3) and select the relevant node group:

   Heap dump for	PE console services	PE Puppet Server	PE orchestration	PuppetDB
   Node group	PE Console	PE Master	PE Orchestrator	PE PuppetDB

2. Click Classes and scroll down to the relevant class

   puppet_enterprise::profile::<class name>

   Heap dump for	PE console services	PE Puppet Server	PE orchestration	PuppetDB
   Class name	console	master	orchestrator	puppetdb

3. Click the Parameter name drop-down list and select java_args. Click Edit. Edit the parameter value to remove "XX:+HeapDumpOnOutOfMemoryError" : "", "XX:HeapDumpPath": "=<filepath>" and "Djruby.reify.classes":"=true". You should be left with, for example: {"Xmx": "4096m", "Xms": "4096m"}.

   Note: Remove XX:+HeapDumpOnOutOfMemoryError, XX:HeapDumpPath and Djruby.reify.classes only. Values for Xmx and Xms (memory settings) are shown as an example.

4. Click the commit button.

5. On the relevant node, edit the JAVA_ARGS setting for the appropriate service and remove XX:+HeapDumpOnOutOfMemoryError, XX:HeapDumpPath and Djruby.reify.classes. For RHEL and SLES, this setting is located in /etc/sysconfig/<service name>. For Ubuntu, the setting is located in /etc/default/<service name>.

   Heap dump for	PE console services	PE Puppet Server	PE orchestration	PuppetDB
   Node	Console node (master in a monolithic installation)	Master node, compile master nodes	Master node	PuppetDB node (master in a monolithic installation)
   Service	pe-console-services	pe-puppetserver	pe-orchestration-services	pe-puppetdb

   For example, for PE Puppet Server on an RHEL node, in /etc/sysconfig/pe-puppetserver:
   
   JAVA_ARGS="-Xms4096m -Xmx4096m -Djava.io.tmpdir=/opt/puppetlabs/server/apps/puppetserver/tmp -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/puppetlabs/puppetserver/puppetserver_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=16 -XX:GCLogFileSize=64m"

   Note: Values for Xmx and Xms (memory settings) are shown as an example.

6. Still on the same node, restart the service.

   Heap dump for	PE console services	PE Puppet Server	PE orchestration	PuppetDB
   Service	pe-console-services	pe-puppetserver	pe-orchestration-services	pe-puppetdb

   For example

   • puppet resource service pe-puppetserver ensure=stopped
   • puppet resource service pe-puppetserver ensure=running