Use the Puppet Profile Parser to graph and investigate Puppet Server performance in Puppet Enterprise 2018.1 to 2019.2

When you’re having performance issues such as slowdowns due to inefficient code or slow external services, you can get troubleshooting information by enabling profiling on Puppet Server. Use the Puppet Profile Parser to make the information easier to read and understand.

Version and installation information

Puppet Enterprise version: 2018.1 to 2019.2
Note: We cannot troubleshoot third-party software (FlameGraph and Jaeger).

Solution

When profiling is enabled, a large amount of information is written to Puppet Server’s main log file (/var/log/puppetlabs/puppetserver/puppetserver.log). Because the information is mixed with other information, it’s hard to interpret. You can use the Puppet Profile Parser to extract relevant information, structure it, and send it to graphing tools like FlameGraph and Jaeger.

To use Puppet Profile Parser, use the steps below to:

1. Enable profiling on some or all nodes.

2. Install the Puppet Profile Parser.

3. Install the graphing tool or tools that you want to use.

4. Use the Profile Parser to extract and format profiling information and use it with your graphing tools.

Enable profiling on catalog requests

Profiling is disabled by default in Puppet Server. You can enable it on specific nodes by using agent configuration or on all nodes by using Puppet Server configuration.

Enable profiling on specific nodes

To enable profiling on a few specific agent nodes:

On each node, in the agent section of puppet.conf set profile = true:

[agent]
profile = true

To enable management of this setting, and apply it to a larger number of nodes, use an ini_setting resource from puppetlabs/inifile.

ini_setting { 'puppet agent: enable profiling':
  ensure  => present,
  path    => '/etc/puppetlabs/puppet/puppet.conf',
  section => 'agent',
  setting => 'profile',
  value   => 'true',
  notify  => Service['puppet'],
}

Enable profiling on all requests

Profile all catalog requests on all nodes.

1. On the master, in puppet.conf in the master section, set profile = true.

   [master]
   profile = true

2. On the master, restart the Puppet Server service pe-puppetserver.

Install the Puppet Profile Parser

1. To run the Puppet Profile Parser, which is written in Ruby, you need a Ruby interpreter of version 2.0 or later. If it is not present, install the system Ruby interpreter. For example:

   [root@master ~]# yum install -y ruby

2. Install the Puppet Profile Parser by downloading a release from its Releases page on Github, and making it executable. For example:

   [root@master ~]# wget -O /usr/local/bin/puppet-profile-parser.rb \
     https://github.com/Sharpie/puppet-profile-parser/releases/download/0.3.0/puppet-profile-parser.rb
   (...)
   [root@master ~]# chmod +x /usr/local/bin/puppet-profile-parser.rb

Install graphing tools

Install the graphing tool or tools you want to use. For a quick and simple installation, you can use FlameGraph, which is written in Perl, to visualize output from the Puppet Profile Parser. Jaeger is a collection of services and tools for performing distributed performance tracing.

Install FlameGraph

Many people use FlameGraph from its master branch on Github, but there is also a v1.0 release that you can install. Download it from the Releases page and extract the archive somewhere appropriate on your Puppet master, such as in /usr/local:

[root@master ~]# wget https://github.com/brendangregg/FlameGraph/archive/v1.0.tar.gz
[root@master ~]# tar zxf v1.0.tar.gz -C /usr/local/

Install Jaeger

There are several ways of deploying Jaeger, including a Kubernetes application.

You can find detailed instructions on deploying in the Jaeger documentation. If you have a Docker host, you can quickly deploy an all-in-one Jaeger stack, suitable for evaluating Jager, using a Docker image:

[croddy@docker ~]$ docker run -d -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 16686:16686 -p 9411:9411 jaegertracing/all-in-one:latest
(...)
[croddy@docker ~]$ docker ps
CONTAINER ID        IMAGE                             COMMAND                  CREATED              STATUS              PORTS                                                                                                       NAMES
472b3ecc7add        jaegertracing/all-in-one:latest   "/go/bin/all-in-one-â¦"   About a minute ago   Up About a minute   5775/udp, 5778/tcp, 14250/tcp, 0.0.0.0:9411->9411/tcp, 6831-6832/udp, 14268/tcp, 0.0.0.0:16686->16686/tcp   ecstatic_galois

Use the Profile Parser to format and graph profiling data

Format and visualize the profiling data in your graphing tools.

Graph profiling data with FlameGraph

The Profile Parser has built-in support to format profiling information for FlameGraph. Parse the Puppet server logs in to FlameGraph format by using the puppet-profile-parser.rb command:

[root@master ~]# /usr/local/bin/puppet-profile-parser.rb -f flamegraph \
  /var/log/puppetlabs/puppetserver/puppetserver.log \
  | /usr/local/FlameGraph-1.0/flamegraph.pl --countname ms \
  > /var/tmp/puppet_profile.svg

The graph is a Scalable Vector Graphics file, you can click on various rows to zoom to see details. FlameGraph graphs can reveal a lot about Puppet Server performance quickly.

In this example, FlameGraph wrote its output through shell redirection to /var/tmp/puppet_profile.svg. You can write the graph to whatever path you like, and serve it from a web server or copy it locally for viewing. It is clear that the catalog for master.example.com takes longer than the other nodes visible to the left:

In another example, zooming in to the details for one node reveals that Puppet Server spent a lot of time on nested each functions:

This example command parses the local logs on your master. If you have compilers, either run the Puppet Profile Parser and FlameGraph on one or more compilers or collect the logs centrally for processing.

Send profiling data to Jaeger

For ongoing detailed monitoring of Puppet Server performance, use the Profile Parser to post data to Jaeger. Use a cron job or similar approach to run the Profile Parser periodically and post its formatted results to Jaeger.

Run the puppet-profile-parser.rb command below to parse the current logs and post them only once:

[root@master ~]# /usr/local/bin/puppet-profile-parser.rb -f zipkin \
  /var/log/puppetlabs/puppetserver/puppetserver.log \
  | curl -X POST -H 'Content-Type: application/json' \
  http://jaeger.example.com:9411/api/v2/spans --data @-

To explore the detail of traces and search for traces matching specific conditions, browse the Jaeger web service. For example, to find all catalog requests that took longer than a certain amount of time:

This example command parses the local logs on your master. If you have compilers, run the Puppet Profile Parser on one or more compilers and post the results to Jaeger from the compilers.

For extensive details on deploying and using Jaeger, refer to the Jaeger documentation.