When you’re having performance issues such as slowdowns due to inefficient code or slow external services, you can get troubleshooting information by enabling profiling on Puppet Server. Use the Puppet Profile Parser to make the information easier to read and understand.
Version and installation information
Puppet Enterprise version: All supported versions
Note: We cannot troubleshoot third-party software (FlameGraph and Jaeger).
Solution
When profiling is enabled, a large amount of information is written to Puppet Server’s main log file (/var/log/puppetlabs/puppetserver/puppetserver.log
). Because the information is mixed with other information, it’s hard to interpret. You can use the Puppet Profile Parser to extract relevant information, structure it, and send it to graphing tools like FlameGraph and Jaeger.
To use Puppet Profile Parser, use the steps below to:
-
Enable profiling on some or all nodes.
-
Install the Puppet Profile Parser.
-
Install the graphing tool or tools that you want to use.
-
Use the Profile Parser to extract and format profiling information and use it with your graphing tools.
Enable profiling on catalog requests
Profiling is disabled by default in Puppet Server. You can enable it on specific nodes by using agent configuration or on all nodes by using Puppet Server configuration.
Enable profiling on specific nodes
To enable profiling on a few specific agent nodes:
On each node, in the agent
section of puppet.conf
set profile = true
:
[agent]
profile = true
To enable management of this setting, and apply it to a larger number of nodes, use an ini_setting
resource from puppetlabs/inifile
.
ini_setting { 'puppet agent: enable profiling':
ensure => present,
path => '/etc/puppetlabs/puppet/puppet.conf',
section => 'agent',
setting => 'profile',
value => 'true',
notify => Service['puppet'],
}
Enable profiling on all requests
Profile all catalog requests on all nodes.
-
On the master, in
puppet.conf
in themaster
section, setprofile = true
.[master] profile = true
-
On the master, restart the Puppet Server service
pe-puppetserver
.
Install the Puppet Profile Parser
-
To run the Puppet Profile Parser, which is written in Ruby, you need a Ruby interpreter of version 2.0 or later. If it is not present, install the system Ruby interpreter. For example:
[root@master ~]# yum install -y ruby
-
Install the Puppet Profile Parser by downloading a release from its Releases page on Github, and making it executable. For example:
[root@master ~]# wget -O /usr/local/bin/puppet-profile-parser.rb \ https://github.com/Sharpie/puppet-profile-parser/releases/download/0.3.0/puppet-profile-parser.rb (...) [root@master ~]# chmod +x /usr/local/bin/puppet-profile-parser.rb
Install graphing tools
Install the graphing tool or tools you want to use. For a quick and simple installation, you can use FlameGraph, which is written in Perl, to visualize output from the Puppet Profile Parser. Jaeger is a collection of services and tools for performing distributed performance tracing.
Install FlameGraph
Many people use FlameGraph from its master
branch on Github, but there is also a v1.0 release that you can install. Download it from the Releases page and extract the archive somewhere appropriate on your Puppet master, such as in /usr/local
:
[root@master ~]# wget https://github.com/brendangregg/FlameGraph/archive/v1.0.tar.gz
[root@master ~]# tar zxf v1.0.tar.gz -C /usr/local/
Install Jaeger
There are several ways of deploying Jaeger, including a Kubernetes application.
You can find detailed instructions on deploying in the Jaeger documentation. If you have a Docker host, you can quickly deploy an all-in-one Jaeger stack, suitable for evaluating Jager, using a Docker image:
[croddy@docker ~]$ docker run -d -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
-p 16686:16686 -p 9411:9411 jaegertracing/all-in-one:latest
(...)
[croddy@docker ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
472b3ecc7add jaegertracing/all-in-one:latest "/go/bin/all-in-one-â¦" About a minute ago Up About a minute 5775/udp, 5778/tcp, 14250/tcp, 0.0.0.0:9411->9411/tcp, 6831-6832/udp, 14268/tcp, 0.0.0.0:16686->16686/tcp ecstatic_galois
Use the Profile Parser to format and graph profiling data
Format and visualize the profiling data in your graphing tools.
Graph profiling data with FlameGraph
The Profile Parser has built-in support to format profiling information for FlameGraph. Parse the Puppet server logs in to FlameGraph format by using the puppet-profile-parser.rb
command:
[root@master ~]# /usr/local/bin/puppet-profile-parser.rb -f flamegraph \
/var/log/puppetlabs/puppetserver/puppetserver.log \
| /usr/local/FlameGraph-1.0/flamegraph.pl --countname ms \
> /var/tmp/puppet_profile.svg
The graph is a Scalable Vector Graphics file, you can click on various rows to zoom to see details. FlameGraph graphs can reveal a lot about Puppet Server performance quickly.
In this example, FlameGraph wrote its output through shell redirection to /var/tmp/puppet_profile.svg
. You can write the graph to whatever path you like, and serve it from a web server or copy it locally for viewing. It is clear that the catalog for master.example.com
takes longer than the other nodes visible to the left:
In another example, zooming in to the details for one node reveals that Puppet Server spent a lot of time on nested each
functions:
This example command parses the local logs on your master. If you have compilers, either run the Puppet Profile Parser and FlameGraph on one or more compilers or collect the logs centrally for processing.
Send profiling data to Jaeger
For ongoing detailed monitoring of Puppet Server performance, use the Profile Parser to post data to Jaeger. Use a cron
job or similar approach to run the Profile Parser periodically and post its formatted results to Jaeger.
Run the puppet-profile-parser.rb
command below to parse the current logs and post them only once:
[root@master ~]# /usr/local/bin/puppet-profile-parser.rb -f zipkin \
/var/log/puppetlabs/puppetserver/puppetserver.log \
| curl -X POST -H 'Content-Type: application/json' \
http://jaeger.example.com:9411/api/v2/spans --data @-
To explore the detail of traces and search for traces matching specific conditions, browse the Jaeger web service. For example, to find all catalog requests that took longer than a certain amount of time:
This example command parses the local logs on your master. If you have compilers, run the Puppet Profile Parser on one or more compilers and post the results to Jaeger from the compilers.
For extensive details on deploying and using Jaeger, refer to the Jaeger documentation.
How can we improve this article?
0 comments
Please sign in to leave a comment.
Related articles