Prevent a thundering herd in Puppet Enterprise: Run Puppet with cron or the reidmv-puppet_run_scheduler module

If you have a thundering herd, you can prevent future thundering herds by spreading out Puppet runs. You can either use cron, or you can use the reidmv-puppet_run_scheduler module, which uses both cron and Windows Task scheduler.

Version and installation information

PE version: All supported versions
Installation type: All

Solution

If your installation is all Linux nodes, you can use cron to prevent thundering herds. If you have an installation that has both Linux and Windows nodes, you can use the use puppet_run_scheduler module to install a task that uses cron and the Task Scheduler to prevent thundering herds. Use one solution or the other. If you use both of these solutions, it will cause issues.

To use cron

Use this solution if your installation has only Linux nodes.

You can use cron to prevent thundering herds by disabling the puppet service and using a cron job to run puppet agent -t (run Puppet) on each agent at a random time chosen by the fqdn_rand() function. This solution keeps agent check-ins evenly distributed, even after the puppet service is restarted by rebooting agents. Before using this solution, check agent nodes for existing cron jobs that run Puppet.

Add the following Puppet code to a profile class to:

• Stop and disable the puppet service

• Install a Cron job that runs puppet agent -t using a randomized runinterval.

  ...
  $runinterval = 30 #minutes
  
  $first_run = fqdn_rand($runinterval)
  $second_run = $first_run + $runinterval
  
  cron { 'cron.puppet':
    command => "/opt/puppetlabs/bin/puppet agent -t > /dev/null",
    user => "root",
    minute => [ $first_run, $second_run ],
  }
  
  # Make sure we haven't ever started the puppet daemon
  # This might cause the report on an agent that is triggered via a daemonized run to not be submitted
  
  service { 'puppet':
    enable => false,
    ensure => stopped,
    require => Cron['cron.puppet'],
  }
  ...

To use the puppet_run_scheduler module

Use this solution if your installation has both Windows and Linux nodes.

Use the puppet_run_scheduler module’s readme and reference to install a Puppet task that allows you to configure and schedule Puppet runs on Windows nodes using Task Scheduler and on Linux nodes using cron.