One of the most common issues in Puppet Enterprise is a java.lang.OutOfMemoryError
(also called an out of memory or OOM error) in journald or in the daemon.log of PE services. Fix the issue by increasing the Java heap size for that service.
Version and installation information
PE version: 2018.1 to 2019.8
Notes:
- Use the version selector on our docs site to make sure you've got the right version of our docs for your deployment.
- In versions of PE earlier than 2019.8, the primary server was called the master.
Error messages
In journald
or the daemon.log
for any PE service, errors similar to:
... java.lang.OutOfMemoryError: Java heap space ...
Or
... java.lang.OutOfMemoryError: GC overhead limit exceeded ...
If Java has hit a process limit like ulimit -u
or a systemd/cgroup limit on tasks, it might also throw an OutOfMemoryError
If you get an error message similar to Unable to create new native thread
, it is likely the issue is not memory related, and the solution included here won’t fix the issue.
Solution
There are two ways to increase the heap size permanently: in Hiera (preferred) or in the console. If services are down, change the heap size temporarily by editing a service file, and then change the values permanently in Hiera or the console.
Note: Values set in the console override values set in Hiera and the service file. Values set in Hiera override settings in the service file.
To learn more about PE services, their default heap sizes, and the JSON strings used to increase heap size, see our documentation.
Change the heap size in Hiera
There are many benefits to using Hiera in your PE infrastructure, so we recommend that you increase the heap size for PE services in Hiera (instead of in the console). If you set heap size in Hiera, do not set it in the console.
Note: You can only change the AMQ java heap in PE 2018.1.
To change the heap size in Hiera, use the the instructions in our documentation and the following examples.
# Console: puppet_enterprise::profile::console::java_args: Xms: 2048m Xmx: 2048m # PuppetServer: puppet_enterprise::profile::master::java_args: Xms: 15360m Xmx: 15360m # Orchestrator: puppet_enterprise::profile::orchestrator::java_args: Xms: 2048m Xmx: 2048m # Puppetdb: puppet_enterprise::profile::puppetdb::java_args: Xms: 16384m Xmx: 16384m
puppet_enterprise::profile::amq::broker::java_args
Change the heap size in the console
To change the heap size for PE services in the console, use the instructions in our documentation. If you set heap size in the console, do not set it in Hiera.
Note: Make sure that you run puppet after making changes to java_args
.
For example, to increase the heap size for the pe-puppetserver
service:
-
In the console, navigate to the PE Infrastructure node group and select the PE Master node group. Click Configuration. Under Parameter select
java_args
and change the Value to{"Xmx": "4096m", "Xms": "4096m"}
. -
Commit the change.
-
To complete the change, on the primary server, run
puppet agent -t
.
When services are down, change the heap size in a service file
Use this method when your services are down and you can’t use the other two methods. Any changes you make to heap size using this method will be reverted during the next Puppet run.
-
Prevent agent runs from reverting your changes by stopping the puppet service. On the primary server, run
puppet resource service puppet ensure=stopped
-
Stop the affected PE service. On the primary server, run
puppet resource service <SERVICE NAME> ensure=stopped
Service names:
- Console: pe-console-services
- Puppet Server: pe-puppetserver
- Orchestrator: pe-orchestration-services
- PuppetDB: pe-puppetdb
- ActiveMQ: (in PE 2018.1 only): pe-activemq
-
In the file that matches the heap size you want to change, edit the
Service files are located here:JAVA_ARGS
Xms and Xms values. For ActiveMQ, (only in PE 2018.1), change the heap size by editing Xms and Xms values forACTIVEMQ_OPTS_MEMORY
.- For Ubuntu:
/etc/default/
- For Red Hat:
/etc/sysconfig/
For example, to change the heap size for Puppet Server on Red Hat, edit
/etc/sysconfig/pe-puppetserver.
- For Ubuntu:
-
Start the service you stopped. On the primary server run
puppet resource service <SERVICE NAME> ensure=running
- Make the setting change permanent by increasing the heap size in Hiera or the console.
- Start the puppet service. On the primary server, run
puppet resource service puppet ensure=running
.
How can we improve this article?
0 comments
Please sign in to leave a comment.
Related articles