Code deployments timing out in Puppet Enterprise versions earlier than 2021.7.2

If you’re having issues with code deployments in multiple environments timing out in Puppet Enterprise versions earlier than 2021.7.2, you can improve performance by adjusting the deploy-pool-cleanup-interval setting.

Error messages and logs

In the Puppet Server log at /var/log/puppetlabs/puppetserver/puppetserver.log, errors similar to the following:

2022-10-31T01:12:43.670-07:00 ERROR [qtp220205936-90762] [p.c.app] The deploys job failed to resolve before the wait timeout was exceeded.

The deploy-pool-cleanup-interval setting specifies how often deploy-pool workers pause to clean their on-disk caches. The deploy-pool-cleanup-interval` value indicates a 1 out of n percent chance after each environment is deployed. In versions of PE earlier than 2021.7.2, the default setting is 10, so after each environment deploys, there’s a 1 in 10 chance of cleanup. As the number of environments increases, the chance of cleanups increases. Each cleanup can take as long as 20 minutes, causing deployment timeouts.

Version and installation information

PE version: All supported

Solution

Check if you have this issue

Since are other reasons that your deployments might fail, such as connection issues or misconfiguration of timeout settings, check if this is your issue by searching for deployment failures in the Puppet Server log (/var/log/puppetlabs/puppetserver/puppetserver.log).

grep -e 'Queuing deploy' -e 'Finished deploy attempt' -e 'Attempting to deploy' -e 'timeout was exceeded' -e 'Starting periodic worker' -e 'Finished periodic worker' puppetserver.log

If you have this issue, the output shows long delays between starting and finishing cache cleanup, for example,

2022-10-31T01:02:20.546-07:00 INFO [deploy-pool-2] [p.c.core] Starting periodic worker cache cleanup in '/opt/puppetlabs/server/data/code-manager/worker-caches/deploy-pool-2'...

2022-10-31T01:12:26.808-07:00 INFO [deploy-pool-2] [p.c.core] Finished periodic worker cache cleanup in '/opt/puppetlabs/server/data/code-manager/worker-caches/deploy-pool-2'.

Followed shortly after by:

The deploys job failed to resolve before the wait timeout was exceeded 

If you do have this issue, complete the steps in the next section to fix it.

If you do not have this issue, but do have the same error message, you can find troubleshooting information in our Code Manager documentation. If you need more help with Code Manager issues, please open a ticket.

Change the setting to improve the issue

The best way to fix this issue is to upgrade to PE 2021.7.2 or later, since the issue is resolved in those versions.

If you are not able to upgrade, complete the following steps.

Increase the deploy-pool-cleanup-interval to 100 on the primary server in /etc/puppetlabs/puppetserver/conf.d/code-manager.conf as follows:

deploy-pool-cleanup-interval: 100

Restart Puppet Service service on the primary server:

puppet resource service pe-puppetserver ensure=stopped puppet resource service pe-puppetserver ensure=running