After adding and removing many nodes in Puppet Enterprise, certificate-related commands fail and the console fails to load

If you added and removed many nodes from your Puppet Enterprise (PE) installation, you might have many signed certificates (certs) for nodes that are no longer managed by PE. This can cause command line certificate-related commands to fail, and 504 Gateway Timeout errors when accessing the console. You can fix the issue by removing certs for nodes that are no longer being used.

Version and installation information

PE version: All supported
OS: Linux

Solution

Use the steps in the following sections to check if you have this issue and fix it.

Check for the issue

If you have many thousands of signed certs for nodes that are no longer under PE management and have issues reaching the console or running cert commands, it’s likely that you have this issue. Find the number of signed certs for nodes that are no longer being used by completing the following steps on the primary server.

1. Check the number of signed certs by running ls /etc/puppetlabs/puppet/ssl/ca/signed/ | wc -l. Note the number of signed certs.

2. Open /var/log/puppetlabs/orchestration-services/aggregate-node-count.log and note the number of active nodes.

3. Calculate the number of signed certs for inactive nodes:

   signed certs for nodes no longer managed by PE = signed certs - active nodes.

If you have thousands of signed certs that are no longer being used, follow the steps in the next section to fix the issue.

Fix the issue

To fix the issue, complete these steps on the primary server to manually remove the signed certs that are no longer being used from /etc/puppetlabs/puppet/ca/signed. You can look at the list of certs that you’re using ahead of deleting them.

This solution requires installing the jq utility. We cannot provide support for third-party software.

1. Install the jq utility.

2. Back up your certificates. On the primary server, run /opt/puppetlabs/bin/puppet-backup create --scope=certs

3. To create a list all signed certs, run ls -1 /etc/puppetlabs/puppet/ssl/ca/signed/ > certfiles.txt

4. To create a list of active nodes, run the following commands:

   A. Create a list of active certs puppet query 'nodes[certname] {}' | jq -r '.[].certname += ".pem"' | jq -r '.[].certname' > query.txt

   B. Add the contents of the console and SAML certificates to the list of active nodes: echo -e "console-cert.pem \nsaml-cert.pem" >> query.txt

   C. Sort the list of nodes alphabetically: sort -o activenodes.txt query.txt

5. To get a list of nodes that are no longer being used, remove the active nodes from the list of signed certs by running comm -3 certfiles.txt activenodes.txt > filestodelete.txt

6. To remove all certs for nodes that are no longer being used from /etc/puppetlabs/puppet/ca/signed, complete the following steps:

   A. Navigate to the directory with signed certs: cd /etc/puppetlabs/puppet/ssl/ca/signed/

   B. Remove the certs: xargs rm < <PATH TO FILESTODELETE.TXT>, for example, xargs rm < /usr/filestodelete.txt