Prevent PuppetDB from failing after Puppet Enterprise upgrade on an external unmanaged PostgreSQL node

You tried to upgrade Puppet Enterprise. You upgraded your PostgreSQL node before you tried to upgrade the primary server. Upgrading the primary server failed, with pe-puppetdb errors.

Error messages and logs

In the command line, when upgrading:

2022-03-28T20:44:44.686+00:00 - [Error]: Systemd start for pe-puppetdb failed!

In the PuppetDB log (/var/log/puppetlabs/puppetdb/puppetdb.log):

Mar 28 20:44:39 puppetdb[25075]: Execution error (PSQLException) at org.postgresql.core.v3.ConnectionFactoryImpl/doAuthentication (ConnectionFactoryImpl.java:613).
Mar 28 20:44:39 puppetdb[25075]: FATAL: no pg_hba.conf entry for host "10.195.138.36", user "pe-puppetdb-read", database "pe-puppetdb", SSL on

After upgrading the PostgreSQL node, the pe-puppetdb-read user was removed during a Puppet run (prior to upgrading the primary server).The code that manages PostgreSQL authentication was not updated on the primary server, so the user was removed.

Version and installation information

PE version: All supported
Installation type: External unmanaged PostgreSQL

Solution

To prevent this issue, stop Puppet runs between the time that PuppetDB is upgraded and the time that the primary server is upgraded. If you’ve already reached this point, you can use a backup to roll back and follow these steps while upgrading. Alternatively, ask the owner of the PostgreSQL database to re-add the pe-puppetdb-read user manually on PostgreSQL nodes.

1. Prior to running the upgrade on the PuppetDB node, stop Puppet runs by stopping the puppet service on all PostgreSQL nodes by running puppet resource service puppet ensure=stopped

   Note: Do not run puppet agent -t on the PuppetDB node until after the primary server is upgraded.

2. After upgrading the primary server, start puppet service. On the primary server, run puppet resource service puppet ensure=running