Use the Puppet Profile Parser to graph and investigate Puppet Server performance in Puppet Enterprise

When you have performance issues such as slowdowns due to inefficient code, slow external services, or one specific node that is slow, you can get troubleshooting information by enabling profiling on Puppet Server. Use the Puppet Profile Parser to make profiling information easier to read and understand.

Version and installation information

Puppet Enterprise version: All supported versions

Solution

When profiling is enabled, a large amount of information is written to Puppet Server’s main log file, /var/log/puppetlabs/puppetserver/puppetserver.log. Because the information is mixed with other information, it’s hard to interpret. You can use the Puppet Profile Parser to extract relevant information, structure it, and send it to graphing tools like FlameGraph and Jaeger.

Note: We cannot troubleshoot third-party software.

Use the following steps to:

1. Enable profiling on catalog requests for some nodes or all nodes.

2. Install the Puppet Profile Parser.

3. Install graphing tools (FlameGraph or Jaeger).

4. Use the Profile Parser to format and graph profiling data.

Part one: Enable profiling on catalog requests

Profiling is disabled by default in Puppet Server. You can enable it on specific nodes by using agent configuration or on all nodes by using Puppet Server configuration.

Enable profiling on specific nodes

To enable profiling on a few specific agent nodes:

• On each node, in the agent section of puppet.conf set profile = true:

  [agent]
  profile = true

• To enable management of this setting, and apply it to a larger number of nodes, use an ini_setting resource from puppetlabs/inifile.

  ini_setting { 'puppet agent: enable profiling':
    ensure  => present,
    path    => '/etc/puppetlabs/puppet/puppet.conf',
    section => 'agent',
    setting => 'profile',
    value   => 'true',
    notify  => Service['puppet'],
  }

Enable profiling on all nodes

To profile catalog requests on all nodes, complete the following steps on the primary server.

1. In puppet.conf in the master section, set profile = true.

   [master]
   profile = true

2. Restart the Puppet Server service.

   puppet resource service pe-puppetserver ensure=stopped
   puppet resource service pe-puppetserver ensure=running

Part two: Install the Puppet Profile Parser

Complete these steps as root on the primary server unless otherwise noted.

1. To run the Puppet Profile Parser, which is written in Ruby, you need a Ruby interpreter version 2.0 or later. If it is not present, install the system Ruby interpreter. For example:

   yum install -y ruby

2. Download a release of Puppet Profile Parser from Github.

   For example:

   wget -O /usr/local/bin/puppet-profile-parser.rb \
   https://github.com/Sharpie/puppet-profile-parser/releases/download/0.3.0/puppet-profile-parser.rb

3. Make the file executable, for example: chmod +x /usr/local/bin/puppet-profile-parser.rb

Part three: Install graphing tools

Install FlameGraph or Jaeger to visualize output from the Puppet Profile Parser. To take a quick look at metrics, install FlameGraph, a Perl-based tool which is simple to install. Install Jaeger to graph ongoing metrics monitoring.

For ongoing detailed monitoring of Puppet Server performance, use the Profile Parser to post data to Jaeger.

Install FlameGraph

Many people use FlameGraph from its master branch on Github, but there is also a v1.0 release that you can install.

1. Download FlameGraph. For example: wget https://github.com/brendangregg/FlameGraph/archive/v1.0.tar.gz

2. On your primary server, as root, extract the archive somewhere appropriate, such as in /usr/local. For example: tar zxf v1.0.tar.gz -C /usr/local/

Install Jaeger

You can deploy Jaeger as Docker images, but you can also use orchestrator templates for Kubernetes and OpenShift setups or Windows services via the sc utility in Windows environments.

Use the instructions on deploying in the Jaeger documentation. If you have a Docker host, you can quickly deploy an all-in-one Jaeger stack, suitable for evaluating Jager, using a Docker image. For example,

[croddy@docker ~]$ docker run -d -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 16686:16686 -p 9411:9411 jaegertracing/all-in-one:latest
(...)
[croddy@docker ~]$ docker ps
CONTAINER ID        IMAGE                             COMMAND                  CREATED              STATUS              PORTS                                                                                                       NAMES
472b3ecc7add        jaegertracing/all-in-one:latest   "/go/bin/all-in-one-â¦"   About a minute ago   Up About a minute   5775/udp, 5778/tcp, 14250/tcp, 0.0.0.0:9411->9411/tcp, 6831-6832/udp, 14268/tcp, 0.0.0.0:16686->16686/tcp   ecstatic_galois

Part four: Use the Profile Parser to format and graph profiling data

Format and visualize the profiling data in FlameGraph or Jaeger.

Graph profiling data with FlameGraph

The Profile Parser has built-in support to format profiling information in FlameGraph. Using FlameGraph, you can reveal a lot of information about Puppet Server performance quickly. In the following example, you parse local logs on your primary server. If you have compilers, you can either run the Puppet Profile Parser and FlameGraph on one or more compilers or collect the logs centrally for processing.

To parse the Puppet server logs in to FlameGraph format, run the puppet-profile-parser.rb command on the primary server as root. In this example, FlameGraph writes its output through shell redirection to /var/tmp/puppet_profile.svg. You can write the graph to whatever path is suitable for your setup. You can serve it from a web server or copy it locally for viewing.

# /usr/local/bin/puppet-profile-parser.rb -f flamegraph \
   /var/log/puppetlabs/puppetserver/puppetserver.log \
   | /usr/local/FlameGraph-1.0/flamegraph.pl --countname ms \
   > /var/tmp/puppet_profile.svg

The graph is in Scalable Vector Graphics format, you can click on different rows to zoom in and see details.

In this example, the catalog for pe-server-9678d5-0.us-west1-a.c.customer-support-scratchpad.internal takes longer than the other nodes visible to the left:

In this example, zooming in to the details for one node reveals that Puppet Server spent a lot of time on nested each functions:

Send profiling data to Jaeger

For ongoing detailed monitoring of Puppet Server performance, use the Profile Parser to post data to Jaeger. Use a cron job or similar approach to run the Profile Parser periodically and post its formatted results to Jaeger. The following example parses the local logs on your primary server. If you have compilers, run the Puppet Profile Parser on one or more compilers and post the results to Jaeger from the compilers.

To parse the current logs and post them only once, run the puppet-profile-parser.rb command below as root on the primary server. For example:

/usr/local/bin/puppet-profile-parser.rb -f zipkin \
  /var/log/puppetlabs/puppetserver/puppetserver.log \
  | curl -X POST -H 'Content-Type: application/json' \
  http://jaeger.example.com:9411/api/v2/spans --data @-

To explore the detail of traces and search for traces matching specific conditions, browse the Jaeger web service. For example, to find all catalog requests that took longer than a certain amount of time, browse to puppetserver: /puppet/v3/catalog:

For more information on deploying and using Jaeger, refer to the Jaeger documentation.