Get a JRuby thread dump on Puppet Enterprise 2018.1

If Support asks you for a thread dump (also known as a stack trace or stack dump) to debug a hanging JRuby process, you can enable and get one by following the steps in this article.

Version and installation information

PE version: 2018.1
Installation type: Any

Solution

You can get a thread dump by using JMX, which is exposed through HTTP via Jolokia endpoints.

By default, thread dumps are not enabled. To enable them, you’ll apply a manifest that adds the jruby.management.enabled argument to JAVA_ARGS, gives Jolokia permissions to run the exec command, and restarts pe-puppetserver service. After you’ve gotten the thread dump, remove access to the exec command in Jolokia and disable thread dumps.

Enable JRuby thread dumps

To enable JRuby thread dumps complete the following steps on the master:

1. Cut and paste the following manifest into a file named enable-jruby-thread-dump.pp

   The manifest generates a version of /etc/puppetlabs/puppetserver/jolokia-access.xml that only allows users connecting over loopback to run an exec of threadDump. For more information on securing the connection and adding IP addresses see Jolokia's security documentation.

   pe_hocon_setting {'metrics.metrics-webservice.jolokia.servlet-init-params.policyLocation':
     ensure  => present,
     path    => '/etc/puppetlabs/puppetserver/conf.d/metrics.conf',
     setting => 'metrics.metrics-webservice.jolokia.servlet-init-params.policyLocation',
     value   => 'file:///etc/puppetlabs/puppetserver/jolokia-access.xml',
   }
   
   file {'/etc/puppetlabs/puppetserver/jolokia-access.xml':
     ensure  => file,
     owner   => 'root',
     group   => 'root',
     mode    => '0644',
     content => @("JOLOKIAACCESS"/L)
       <?xml version="1.0" encoding="utf-8"?>
   
       <restrict>
         <remote>
           <host>127.0.0.1</host>
         </remote>
   
         <allow>
           <mbean>
             <name>org.jruby:type=Runtime,service=Runtime</name>
             <operation>threadDump</operation>
           </mbean>
         </allow>
       </restrict>
       | JOLOKIAACCESS
   
   }
   
   $defaults_dir = $facts['os']['family'] ? {
     'Debian'      => '/etc/default',
     /RedHat|Suse/ => '/etc/sysconfig',
     default       => undef,
   }
   
   pe_ini_subsetting { 'pe_puppetserver_jruby.management.enabled':
     ensure            => present,
     path              => "${defaults_dir}/pe-puppetserver",
     key_val_separator => '=',
     section           => '',
     setting           => 'JAVA_ARGS',
     quote_char        => '"',
     subsetting        => '-Djruby.management.enabled',
     value             => '=true',
     notify            => Service['pe-puppetserver'],
   }
   
   service {'pe-puppetserver':
     ensure => running,
   }

2. Apply the manifest by running puppet apply enable-jruby-thread-dump.pp

   The output will be similar to the following:

   # puppet apply enable-jruby-thread-dump.pp
   Notice: Compiled catalog for pe201814cm.test in environment production in 0.25 seconds
   Notice: /Stage[main]/Main/Pe_hocon_setting[metrics.metrics-webservice.jolokia.servlet-init-params.policyLocation]/ensure: created
   Notice: /Stage[main]/Main/File[/etc/puppetlabs/puppetserver/jolokia-access.xml]/ensure: defined content as '{md5}c28a2b47743cfc2bccc4c9cf63eca467'
   Notice: /Stage[main]/Main/Pe_ini_subsetting[pe_puppetserver_jruby.management.enabled]/ensure: created
   Notice: /Stage[main]/Main/Service[pe-puppetserver]: Triggered 'refresh' from 1 event
   Notice: Applied catalog in 35.08 seconds

3. When you’ve replicated the problem, run the following command to get the thread dump and store the output in /tmp/thread-dump.txt.

   for mbean in $(curl -sk https://127.0.0.1:8140/metrics/v2/list |python -m json.tool |grep "service=Runtime,type=Runtime" |cut -d'"' -f2); do curl -sk https://127.0.0.1:8140/metrics/v2/exec/org.jruby:$mbean/threadDump |python -m json.tool; done > /tmp/thread-dump.txt

Attach the generated file to your support ticket.

Disable JRuby thread dumps

When you have the thread dump, remove access to the exec command in Jolokia and disable thread dumps by completing the following steps on the master.

1. Cut and paste the following manifest into a file named disable-jruby-thread-dump.pp:

   pe_hocon_setting {'metrics.metrics-webservice.jolokia.servlet-init-params.policyLocation':
     ensure  => absent,
     path    => '/etc/puppetlabs/puppetserver/conf.d/metrics.conf',
     setting => 'metrics.metrics-webservice.jolokia.servlet-init-params.policyLocation',
   }
   
   file {'/etc/puppetlabs/puppetserver/jolokia-access.xml':
     ensure  => absent,
   }
   
   $defaults_dir = $facts['os']['family'] ? {
     'Debian'      => '/etc/default',
     /RedHat|Suse/ => '/etc/sysconfig',
     default       => undef,
   }
   
   pe_ini_subsetting { 'pe_puppetserver_jruby.management.enabled':
     ensure            => absent,
     path              => "${defaults_dir}/pe-puppetserver",
     key_val_separator => '=',
     section           => '',
     setting           => 'JAVA_ARGS',
     quote_char        => '"',
     subsetting        => '-Djruby.management.enabled',
     value             => '=true',
     notify            => Service['pe-puppetserver'],
   }
   
   service {'pe-puppetserver':
     ensure => running,
   }

2. Apply the manifest by running puppet apply disable-jruby-thread-dump.pp

   The output will be similar to the following:

   # puppet apply disable-jruby-thread-dump.pp
   Notice: Compiled catalog for pe201814cm.test in environment production in 0.25 seconds
   Notice: /Stage[main]/Main/Pe_hocon_setting[metrics.metrics-webservice.jolokia.servlet-init-params.policyLocation]/ensure: removed
   Notice: /Stage[main]/Main/File[/etc/puppetlabs/puppetserver/jolokia-access.xml]/ensure: removed
   Notice: /Stage[main]/Main/Pe_ini_subsetting[pe_puppetserver_jruby.management.enabled]/ensure: removed
   Notice: /Stage[main]/Main/Service[pe-puppetserver]: Triggered 'refresh' from 1 event
   Notice: Applied catalog in 40.26 seconds