Resolve Puppet run failures with "(SystemStackError) stack level too deep" errors for Puppet Enterprise 3.8 to 2019.1

When you run puppet agent -t, you get a (SystemStackError) stack level too deep error.

Error messages and logs

Error message:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Internal Server Error: org.jruby.exceptions.RaiseException: (SystemStackError) stack level too deep

Logs:

In the Puppet Server log,

for PE 2015.2 to 2019.1: /var/log/puppetlabs/puppetserver/puppetserver.log

for PE 3.8: /var/log/pe-puppetserver/puppetserver.log:

2015-11-02 11:17:10,496 ERROR [p.p.ringutils] Exception while handling HTTP request org.jruby.exceptions.RaiseException: (SystemStackError) stack level too deep at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613) ~[puppet-server-release.jar:na] at org.jruby.RubyEnumerable.inject(org/jruby/RubyEnumerable.java:866)

You received this error because the pe-puppetserver process exceeded the JVM stack size.

Version and installation information

PE version: 3.8 to 2019.1

Solution

To resolve the error, increase the JVM stack size from 1MB (default) to 2MB.

Note: Our terminology changed when we released PE 2019.1, master of masters changed to master and compile master changed to compiler.

1. Open /etc/sysconfig/pe-puppetserver (on EL-based systems) or /etc/default/pe-puppetserver (on Ubuntu). Add an -Xss2m option to the existing JAVA_ARGS, for example:

   JAVA_ARGS="-Xms<XMS SIZE> -Xmx<XMX SIZE> -Xss2m"

   Note: Add the -Xss option if it doesn't exist, edit it if it already exists. No other changes to JAVA_ARGS are necessary.

2. Restart pe-puppetserver:

   puppet resource service pe-puppetserver ensure=stopped ; puppet resource service pe-puppetserver ensure=running

3. Verify that the change resolved the error by running puppet agent -t.