If Support asks you for a thread dump (also known as a stack trace or stack dump) to debug a hanging JRuby process, you can enable and get one by following the steps in this article.
Version and installation information
PE version: 2018.1
Installation type: Any
Solution
You can get a thread dump by using JMX, which is exposed through HTTP via Jolokia endpoints.
By default, thread dumps are not enabled. To enable them, you’ll apply a manifest that adds the jruby.management.enabled
argument to JAVA_ARGS
, gives Jolokia permissions to run the exec
command, and restarts pe-puppetserver service. After you’ve gotten the thread dump, remove access to the exec
command in Jolokia and disable thread dumps.
Enable JRuby thread dumps
To enable JRuby thread dumps complete the following steps on the master:
-
Cut and paste the following manifest into a file named
enable-jruby-thread-dump.pp
The manifest generates a version of
/etc/puppetlabs/puppetserver/jolokia-access.xml
that only allows users connecting over loopback to run an exec of threadDump. For more information on securing the connection and adding IP addresses see Jolokia's security documentation.pe_hocon_setting {'metrics.metrics-webservice.jolokia.servlet-init-params.policyLocation': ensure => present, path => '/etc/puppetlabs/puppetserver/conf.d/metrics.conf', setting => 'metrics.metrics-webservice.jolokia.servlet-init-params.policyLocation', value => 'file:///etc/puppetlabs/puppetserver/jolokia-access.xml', } file {'/etc/puppetlabs/puppetserver/jolokia-access.xml': ensure => file, owner => 'root', group => 'root', mode => '0644', content => @("JOLOKIAACCESS"/L) <?xml version="1.0" encoding="utf-8"?> <restrict> <remote> <host>127.0.0.1</host> </remote> <allow> <mbean> <name>org.jruby:type=Runtime,service=Runtime</name> <operation>threadDump</operation> </mbean> </allow> </restrict> | JOLOKIAACCESS } $defaults_dir = $facts['os']['family'] ? { 'Debian' => '/etc/default', /RedHat|Suse/ => '/etc/sysconfig', default => undef, } pe_ini_subsetting { 'pe_puppetserver_jruby.management.enabled': ensure => present, path => "${defaults_dir}/pe-puppetserver", key_val_separator => '=', section => '', setting => 'JAVA_ARGS', quote_char => '"', subsetting => '-Djruby.management.enabled', value => '=true', notify => Service['pe-puppetserver'], } service {'pe-puppetserver': ensure => running, }
-
Apply the manifest by running
puppet apply enable-jruby-thread-dump.pp
The output will be similar to the following:
# puppet apply enable-jruby-thread-dump.pp Notice: Compiled catalog for pe201814cm.test in environment production in 0.25 seconds Notice: /Stage[main]/Main/Pe_hocon_setting[metrics.metrics-webservice.jolokia.servlet-init-params.policyLocation]/ensure: created Notice: /Stage[main]/Main/File[/etc/puppetlabs/puppetserver/jolokia-access.xml]/ensure: defined content as '{md5}c28a2b47743cfc2bccc4c9cf63eca467' Notice: /Stage[main]/Main/Pe_ini_subsetting[pe_puppetserver_jruby.management.enabled]/ensure: created Notice: /Stage[main]/Main/Service[pe-puppetserver]: Triggered 'refresh' from 1 event Notice: Applied catalog in 35.08 seconds
-
When you’ve replicated the problem, run the following command to get the thread dump and store the output in
/tmp/thread-dump.txt
.for mbean in $(curl -sk https://127.0.0.1:8140/metrics/v2/list |python -m json.tool |grep "service=Runtime,type=Runtime" |cut -d'"' -f2); do curl -sk https://127.0.0.1:8140/metrics/v2/exec/org.jruby:$mbean/threadDump |python -m json.tool; done > /tmp/thread-dump.txt
Attach the generated file to your support ticket.
Disable JRuby thread dumps
When you have the thread dump, remove access to the exec
command in Jolokia and disable thread dumps by completing the following steps on the master.
-
Cut and paste the following manifest into a file named
disable-jruby-thread-dump.pp
:pe_hocon_setting {'metrics.metrics-webservice.jolokia.servlet-init-params.policyLocation': ensure => absent, path => '/etc/puppetlabs/puppetserver/conf.d/metrics.conf', setting => 'metrics.metrics-webservice.jolokia.servlet-init-params.policyLocation', } file {'/etc/puppetlabs/puppetserver/jolokia-access.xml': ensure => absent, } $defaults_dir = $facts['os']['family'] ? { 'Debian' => '/etc/default', /RedHat|Suse/ => '/etc/sysconfig', default => undef, } pe_ini_subsetting { 'pe_puppetserver_jruby.management.enabled': ensure => absent, path => "${defaults_dir}/pe-puppetserver", key_val_separator => '=', section => '', setting => 'JAVA_ARGS', quote_char => '"', subsetting => '-Djruby.management.enabled', value => '=true', notify => Service['pe-puppetserver'], } service {'pe-puppetserver': ensure => running, }
-
Apply the manifest by running
puppet apply disable-jruby-thread-dump.pp
The output will be similar to the following:
# puppet apply disable-jruby-thread-dump.pp Notice: Compiled catalog for pe201814cm.test in environment production in 0.25 seconds Notice: /Stage[main]/Main/Pe_hocon_setting[metrics.metrics-webservice.jolokia.servlet-init-params.policyLocation]/ensure: removed Notice: /Stage[main]/Main/File[/etc/puppetlabs/puppetserver/jolokia-access.xml]/ensure: removed Notice: /Stage[main]/Main/Pe_ini_subsetting[pe_puppetserver_jruby.management.enabled]/ensure: removed Notice: /Stage[main]/Main/Service[pe-puppetserver]: Triggered 'refresh' from 1 event Notice: Applied catalog in 40.26 seconds
How can we improve this article?
0 comments
Article is closed for comments.
Related articles