When you have performance issues such as slowdowns due to inefficient code, slow external services, or one specific node that is slow, you can get troubleshooting information by enabling profiling on Puppet Server. Use the Puppet Profile Parser to make profiling information easier to read and understand.
Version and installation information
Puppet Enterprise version: All supported versions
Solution
When profiling is enabled, a large amount of information is written to Puppet Server’s main log file, /var/log/puppetlabs/puppetserver/puppetserver.log
. Because the information is mixed with other information, it’s hard to interpret. You can use the Puppet Profile Parser to extract relevant information, structure it, and send it to graphing tools like FlameGraph and Jaeger.
Note: We cannot troubleshoot third-party software.
Use the following steps to:
Part one: Enable profiling on catalog requests
Profiling is disabled by default in Puppet Server. You can enable it on specific nodes by using agent configuration or on all nodes by using Puppet Server configuration.
Enable profiling on specific nodes
To enable profiling on a few specific agent nodes:
-
On each node, in the
agent
section ofpuppet.conf
setprofile = true
:[agent] profile = true
-
To enable management of this setting, and apply it to a larger number of nodes, use an
ini_setting
resource frompuppetlabs/inifile
.ini_setting { 'puppet agent: enable profiling': ensure => present, path => '/etc/puppetlabs/puppet/puppet.conf', section => 'agent', setting => 'profile', value => 'true', notify => Service['puppet'], }
Enable profiling on all nodes
To profile catalog requests on all nodes, complete the following steps on the primary server.
-
In
puppet.conf
in themaster
section, setprofile = true
.[master] profile = true
-
Restart the Puppet Server service.
puppet resource service pe-puppetserver ensure=stopped puppet resource service pe-puppetserver ensure=running
Part two: Install the Puppet Profile Parser
Complete these steps as root on the primary server unless otherwise noted.
-
To run the Puppet Profile Parser, which is written in Ruby, you need a Ruby interpreter version 2.0 or later. If it is not present, install the system Ruby interpreter. For example:
yum install -y ruby
-
Download a release of Puppet Profile Parser from Github.
For example:
wget -O /usr/local/bin/puppet-profile-parser.rb \ https://github.com/Sharpie/puppet-profile-parser/releases/download/0.3.0/puppet-profile-parser.rb
-
Make the file executable, for example:
chmod +x /usr/local/bin/puppet-profile-parser.rb
Part three: Install graphing tools
Install FlameGraph or Jaeger to visualize output from the Puppet Profile Parser. To take a quick look at metrics, install FlameGraph, a Perl-based tool which is simple to install. Install Jaeger to graph ongoing metrics monitoring.
For ongoing detailed monitoring of Puppet Server performance, use the Profile Parser to post data to Jaeger.
Install FlameGraph
Many people use FlameGraph from its master
branch on Github, but there is also a v1.0 release that you can install.
-
Download FlameGraph. For example:
wget https://github.com/brendangregg/FlameGraph/archive/v1.0.tar.gz
-
On your primary server, as root, extract the archive somewhere appropriate, such as in
/usr/local
. For example:tar zxf v1.0.tar.gz -C /usr/local/
Install Jaeger
You can deploy Jaeger as Docker images, but you can also use orchestrator templates for Kubernetes and OpenShift setups or Windows services via the sc
utility in Windows environments.
Use the instructions on deploying in the Jaeger documentation. If you have a Docker host, you can quickly deploy an all-in-one Jaeger stack, suitable for evaluating Jager, using a Docker image. For example,
[croddy@docker ~]$ docker run -d -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 16686:16686 -p 9411:9411 jaegertracing/all-in-one:latest
(...)
[croddy@docker ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
472b3ecc7add jaegertracing/all-in-one:latest "/go/bin/all-in-one-â¦" About a minute ago Up About a minute 5775/udp, 5778/tcp, 14250/tcp, 0.0.0.0:9411->9411/tcp, 6831-6832/udp, 14268/tcp, 0.0.0.0:16686->16686/tcp ecstatic_galois
Part four: Use the Profile Parser to format and graph profiling data
Format and visualize the profiling data in FlameGraph or Jaeger.
Graph profiling data with FlameGraph
The Profile Parser has built-in support to format profiling information in FlameGraph. Using FlameGraph, you can reveal a lot of information about Puppet Server performance quickly. In the following example, you parse local logs on your primary server. If you have compilers, you can either run the Puppet Profile Parser and FlameGraph on one or more compilers or collect the logs centrally for processing.
To parse the Puppet server logs in to FlameGraph format, run the puppet-profile-parser.rb
command on the primary server as root. In this example, FlameGraph writes its output through shell redirection to /var/tmp/puppet_profile.svg
. You can write the graph to whatever path is suitable for your setup. You can serve it from a web server or copy it locally for viewing.
# /usr/local/bin/puppet-profile-parser.rb -f flamegraph \
/var/log/puppetlabs/puppetserver/puppetserver.log \
| /usr/local/FlameGraph-1.0/flamegraph.pl --countname ms \
> /var/tmp/puppet_profile.svg
The graph is in Scalable Vector Graphics format, you can click on different rows to zoom in and see details.
In this example, the catalog for pe-server-9678d5-0.us-west1-a.c.customer-support-scratchpad.internal
takes longer than the other nodes visible to the left:
In this example, zooming in to the details for one node reveals that Puppet Server spent a lot of time on nested each
functions:
Send profiling data to Jaeger
For ongoing detailed monitoring of Puppet Server performance, use the Profile Parser to post data to Jaeger. Use a cron
job or similar approach to run the Profile Parser periodically and post its formatted results to Jaeger. The following example parses the local logs on your primary server. If you have compilers, run the Puppet Profile Parser on one or more compilers and post the results to Jaeger from the compilers.
To parse the current logs and post them only once, run the puppet-profile-parser.rb
command below as root on the primary server. For example:
/usr/local/bin/puppet-profile-parser.rb -f zipkin \
/var/log/puppetlabs/puppetserver/puppetserver.log \
| curl -X POST -H 'Content-Type: application/json' \
http://jaeger.example.com:9411/api/v2/spans --data @-
To explore the detail of traces and search for traces matching specific conditions, browse the Jaeger web service. For example, to find all catalog requests that took longer than a certain amount of time, browse to puppetserver: /puppet/v3/catalog
:
For more information on deploying and using Jaeger, refer to the Jaeger documentation.
How can we improve this article?
0 comments
Please sign in to leave a comment.
Related articles