Fix "java.lang.OutOfMemoryError" errors by increasing the Java heap size in Puppet Enterprise

One of the most common issues in Puppet Enterprise (PE) is a java.lang.OutOfMemoryError, also called an out of memory or OOM error, in journald or in the daemon.log of PE services. Fix the issue by increasing the Java heap size for that service.

Version and installation information

PE version: all supported

Links are to the most recent version of our documentation, use the version selector to make sure you have the right version for your deployment.

Error messages

In the syslog for your OS, for example journald or the daemon.log , errors for any PE service like:

java.lang.OutOfMemoryError: Java heap space or java.lang.OutOfMemoryError: GC overhead limit exceeded

If Java has hit a process limit like ulimit -u or a systemd/cgroup limit on tasks, it might also throw an OutOfMemoryError. If you get an error message like Unable to create new native thread, it is likely the issue is not memory related, and the steps included here won’t fix the issue.

Solution

To fix the issue, choose a new heap size and then increase it by changing the java_args setting. There are two ways to increase the heap size permanently: in Hiera (preferred) or in the console. If services are down, change the heap size temporarily by editing a service file, and then change the values permanently in Hiera or the console. Values set in the console override values set in Hiera and the service file. Values set in Hiera override settings in the service file.

Choose a new heap size

The java_args settings specify heap size, which is the amount of memory that each Java process can request from the operating system. Heap size is declared as a JSON hash containing a maximum (Xmx) and minimum (Xms) value. Usually, the maximum and minimum are the same so that the heap size is fixed.

To choose a new value, you can use the puppet infrastructure tune command, which is compatible with most standard PE configurations, including those with compilers, a replica, or standalone PostgreSQL. There is also a table with suggested tuning values for Java heap size for standard or large installations in our documentation.

Learn more about java_args, including the defaults for Puppet services.

• Change the heap size in Hiera
• Change the heap size in the console
• Change the heap size when services are down

Change the heap size in Hiera

If you set heap size in Hiera, do not set it in the console.

To change the heap size in Hiera, use the instructions in our documentation. You can use the following examples for syntax and formatting. Ensure that you update the values so that they are appropriate for the resources available on your infrastructure.

# Puppet Server:
puppet_enterprise::profile::master::java_args:
  Xms: 15360m
  Xmx: 15360m

# PuppetDB:
puppet_enterprise::profile::puppetdb::java_args:
  Xms: 16384m
  Xmx: 16384m

# Console:
puppet_enterprise::profile::console::java_args:
  Xms: 2048m
  Xmx: 2048m

# Orchestrator:
puppet_enterprise::profile::orchestrator::java_args:
  Xms: 2048m
  Xmx: 2048m

Change the heap size in the console

To change the heap size for PE services in the console, use the following steps. If you set heap size in the console, do not set it in Hiera. Make sure that you run Puppet after making changes.

1. In the console, click Node groups, then find the node group of the service you want to change the heap for.

   Node groups

   • Puppet Server: PE Master or PE Compiler (for the primary server and compilers, respectively)
   • PuppetDB: PE PuppetDB
   • Console services: PE Console
   • Orchestrator: PE Orchestrator

2. On the Configuration data tab, specify a Class based on the service you want to change, and select the java_args parameter. Change the Value. For example, {"Xmx": "4096m", "Xms": "4096m"}.

   Classes

   • Puppet Server: puppet_enterprise::profile::master
   • PuppetDB: puppet_enterprise::profile::puppetdb
   • Console services: puppet_enterprise::profile::console
   • Orchestration services: puppet_enterprise::profile::orchestrator

3. Click Add to node group and Commit the changes.

4. To complete the change, on the primary server, run puppet agent -t

When Puppet services are down, change the heap size in a service file

Use this method when services are down, and you can’t use the other two methods. Any changes you make to heap size using this method will be reverted during the next Puppet run. Complete these steps on the primary server unless otherwise noted.

1. Prevent agent runs from reverting your changes by stopping the puppet service. Run puppet resource service puppet ensure=stopped

2. Stop the affected PE service. Run puppet resource service <SERVICE NAME> ensure=stopped

   Service names

   • Puppet Server: pe-puppetserver
   • PuppetDB: pe-puppetdb
   • Console services: pe-console-services
   • Orchestrator: pe-orchestration-services

3. On the primary server, edit the JAVA_ARGS Xms and Xms values in a service file. The location of service files varies based on your OS.

   • For Ubuntu: /etc/default/
   • For Red Hat: /etc/sysconfig/

   Each service file is named after its service. For example, the service file for Puppet Server in Red Hat is /etc/sysconfig/pe-puppetserver

4. Start the service you stopped. Run puppet resource service <SERVICE NAME> ensure=running

5. Make the setting change permanent by increasing the heap size in Hiera or the console using the steps in this article.

6. Start the puppet service. Run puppet resource service puppet ensure=running