Performance issues and many calls to “/puppet/v4/catalog” after enabling impact analysis in Continuous Delivery for Puppet Enterprise

If you have environments with many nodes, it might take a long time for impact analysis to run. When you look at the Puppet Server access log, there are many calls to the /puppet/v4/catalog API endpoint.

Logs

In the Puppet Server access log, /var/log/puppetlabs/puppetserver/puppetserver-access.log, you find many requests to the /puppet/v4/catalog API endpoint, exclusively used by impact analysis:

In $ zgrep -c /puppet/v4/catalog logs/puppetserver/puppetserver-access*

logs/puppetserver/puppetserver-access-2021-07-20.0.log.gz:6475
logs/puppetserver/puppetserver-access-2021-07-21.0.log.gz:0
logs/puppetserver/puppetserver-access-2021-07-22.0.log.gz:43
logs/puppetserver/puppetserver-access-2021-07-23.0.log.gz:3658
logs/puppetserver/puppetserver-access-2021-07-24.0.log.gz:0
logs/puppetserver/puppetserver-access-2021-07-25.0.log.gz:0
logs/puppetserver/puppetserver-access-2021-07-26.0.log.gz:245
logs/puppetserver/puppetserver-access.log:27433

With impact analysis enabled, catalog compiles are generated every time impact analysis detects that a node could be impacted by a Puppet code change. These additional compiles increase the performance load placed on your PE primary server.

Version and installation information

Product: Continuous Delivery for Puppet Enterprise
Version: 4.x

Solution

You can improve performance by dedicating a pool of compilers to impact analysis. See the list of solution options and steps in our documentation.

Depending on your organization’s use case for impact analysis, you might want to adjust how you’re using it. Check out our Solutions Architecture team’s recommendations for running impact analysis at scale.

It is also possible to use steps in our documentation to only analyze a subset of your total nodes. However, there are tradeoffs:

• Not all nodes get analyzed.

• Additional heap space is consumed. To run impact analysis on fewer nodes, you must create one or more dedicated impact analysis environments. Each impact analysis environment has the same code as its corresponding primary environment (for example, production and production-ia). Because environments consume heap space in Puppet Server, adding additional environments consumes additional heap space deploying the same code to multiple environments.