Workaround for hiera-eyaml memory leak and OutOfMemory errors in Puppet Enterprise 3.8 and later

There is a known issue where using hiera-eyaml version 2.0.8 or earlier can cause a memory leak and OutOfMemory errors. Over time, Puppet Server performance will diminish, eventually becoming unresponsive. Restarting it will restore functionality temporarily. This affects you if your hiera.yaml configuration file contains the following backend declaration:

:backends:
    - eyaml

and if the max-requests-per-instance setting in /etc/puppetlabs/puppetserver/conf.d/pe-puppet-server.conf is anything other than 0 (zero). You might experience this issue after upgrading because the default max-requests-per-instance setting changes from 0 (zero) in 3.8 and 2015.2 to 10000 in 2015.3.

Version and installation information

PE version: 3.8 and later
OS: Any *nix

Solution

Normally, each JRuby interpreter that PE uses gets shut down, all of its memory is reclaimed, and a new one is added. However, when you're using a version of hiera-eyaml earlier than 2.1.0, the JRuby instances have JVM threads that prevent garbage collection, so old JRuby instances can't be reclaimed.

To continue using hiera-eyaml and prevent the memory leak, upgrade hiera-eyaml to version 2.1.0 or later.

As a temporary workaround if you are unable to upgrade hiera-eyaml, you can use our documentation to set max-requests-per-instance to 0 to prevent the memory leak. Make sure that you use steps for your version of PE.