Puppet Server fails after “pg_wal” fills up the hard drive in Puppet Enterprise

Puppet Server fails after pg_wal fills up the hard drive that your Puppet Enterprise-managed PostgreSQL instance is on, and your Puppet Enterprise (PE) instance is down.

Version and installation information

PE version: All supported

Solution

If your disk is filling up, and your PE instance is still up, check the size of pg-wal on the primary server by running du -h /opt/puppetlabs/ | grep '/data/pg_wal'

If you have a replica

If you have a replica, this issue is most likely due to your replica being down and not replicating. Unreplicated data consumes disk space on your primary server.

1. To fix the issue, on the primary server, forget the replica by running puppet infra forget replica

2. After you forget the replica, you can create a new one. To ensure the continued health of your primary server, you must maintain syncing to your replica or promptly forget the replica.

3. If you are using PE 2021.7 or later, the max_slot_wal_keep_size helps prevent this issue from happening. If you don’t have enough disk space for the default setting (12288 MB), adjust the setting.

If this doesn’t fix your issue, use the steps included in the following section to give yourself some extra time to solve the problem. If you’re not able to solve the issue, please open a ticket with us.

If you don’t have a replica, or if forgetting your replica didn’t help

If you don’t have a replica, or if you continue to have an issue after you forget your replica, it’s likely that you have an underlying performance issue.

1. Complete one of the included temporary fix steps. Neither option will prevent pg_wal from filling up your hard drive, however, it will give you some time to troubleshoot the issue, and time to work through a ticket with our team to solve the issue.

   Either:

   • Revert to a snapshot or restore a backup prior to pg_wal filling the disk.

   Or:

   • Stop all PE services and increase the disk volume that /opt/puppetlabs is on before restarting services.

2. If you are using PE 2021.7 or later, the max_slot_wal_keep_size helps prevent this issue. If you don’t have enough disk space for the default setting (12288 MB), adjust the setting.

3. Troubleshoot PuppetDB. If you’re not able to solve the issue, please open a ticket with us.