Fix "java.lang.OutOfMemoryError" errors by increasing the Java heap size in Puppet Enterprise 2018.1 to 2019.8

One of the most common issues in Puppet Enterprise is a java.lang.OutOfMemoryError (also called an out of memory or OOM error) in journald or in the daemon.log of PE services. Fix the issue by increasing the Java heap size for that service.

Version and installation information

PE version: 2018.1 to 2019.8

Notes:

• Use the version selector on our docs site to make sure you've got the right version of our docs for your deployment.
• In versions of PE earlier than 2019.8, the primary server was called the master.

Error messages

In journald or the daemon.log for any PE service, errors similar to:

...
java.lang.OutOfMemoryError: Java heap space
...

Or

...
java.lang.OutOfMemoryError: GC overhead limit exceeded
...

If Java has hit a process limit like ulimit -u or a systemd/cgroup limit on tasks, it might also throw an OutOfMemoryError If you get an error message similar to Unable to create new native thread, it is likely the issue is not memory related, and the solution included here won’t fix the issue.

Solution

There are two ways to increase the heap size permanently: in Hiera (preferred) or in the console. If services are down, change the heap size temporarily by editing a service file, and then change the values permanently in Hiera or the console.

Note: Values set in the console override values set in Hiera and the service file. Values set in Hiera override settings in the service file.

To learn more about PE services, their default heap sizes, and the JSON strings used to increase heap size, see our documentation.

Change the heap size in Hiera

There are many benefits to using Hiera in your PE infrastructure, so we recommend that you increase the heap size for PE services in Hiera (instead of in the console). If you set heap size in Hiera, do not set it in the console.

Note: You can only change the AMQ java heap in PE 2018.1.

To change the heap size in Hiera, use the the instructions in our documentation and the following examples.

# Console:
puppet_enterprise::profile::console::java_args:
  Xms: 2048m
  Xmx: 2048m

# PuppetServer:
puppet_enterprise::profile::master::java_args:
  Xms: 15360m
  Xmx: 15360m

# Orchestrator:
puppet_enterprise::profile::orchestrator::java_args:
  Xms: 2048m
  Xmx: 2048m

# Puppetdb:
puppet_enterprise::profile::puppetdb::java_args:
  Xms: 16384m
  Xmx: 16384m

puppet_enterprise::profile::amq::broker::java_args

Change the heap size in the console

To change the heap size for PE services in the console, use the instructions in our documentation. If you set heap size in the console, do not set it in Hiera.

Note: Make sure that you run puppet after making changes to java_args.

For example, to increase the heap size for the pe-puppetserver service:

1. In the console, navigate to the PE Infrastructure node group and select the PE Master node group. Click Configuration. Under Parameter select java_args and change the Value to {"Xmx": "4096m", "Xms": "4096m"}.

2. Commit the change.

3. To complete the change, on the primary server, run puppet agent -t.

When services are down, change the heap size in a service file

Use this method when your services are down and you can’t use the other two methods. Any changes you make to heap size using this method will be reverted during the next Puppet run.

1. Prevent agent runs from reverting your changes by stopping the puppet service. On the primary server, run puppet resource service puppet ensure=stopped

2. Stop the affected PE service. On the primary server, run puppet resource service <SERVICE NAME> ensure=stopped

   Service names:

   • Console: pe-console-services
   • Puppet Server: pe-puppetserver
   • Orchestrator: pe-orchestration-services
   • PuppetDB: pe-puppetdb
   • ActiveMQ: (in PE 2018.1 only): pe-activemq

3. In the file that matches the heap size you want to change, edit the JAVA_ARGS Xms and Xms values. For ActiveMQ, (only in PE 2018.1), change the heap size by editing Xms and Xms values for ACTIVEMQ_OPTS_MEMORY.

   Service files are located here:
   • For Ubuntu: /etc/default/
   • For Red Hat: /etc/sysconfig/

   For example, to change the heap size for Puppet Server on Red Hat, edit /etc/sysconfig/pe-puppetserver.

4. Start the service you stopped. On the primary server run puppet resource service <SERVICE NAME> ensure=running

5. Make the setting change permanent by increasing the heap size in Hiera or the console.
6. Start the puppet service. On the primary server, run puppet resource service puppet ensure=running.