Change the backup timeout parameter when a snapshot fails in Continuous Delivery for Puppet Enterprise

Exceeding the default timeout limit causes the Puppet Application Manager (PAM) snapshot to fail, and you get an error message in Continuous Delivery for Puppet Enterprise.

Error message

You get this error message in PAM when the snapshot fails.

time="2022-08-02T18:13:52Z" level=error msg="Error backing up item" backup=velero/instance-b7w8b error="timed out after 10m0s" ...

Version and installation information

Product: Continuous Delivery for Puppet Enterprise
Version: 4.x

Solution

Determine if a snapshot fails from a timeout and how much time it takes the snapshot to complete on the node. Then, change the backup timeout parameter in Continuous Delivery for PE.

Determine if the default timeout limit is causing the snapshot to fail on the node

To determine if the snapshot fails, generate a support bundle by following these steps:

1. In the PAM UI, select the Troubleshoot tab.

2. To generate a support bundle, select Analyze Continuous Delivery for Puppet Enterprise.

3. Download the bundle.

4. In the velero/velero-*replicasetid*-*podid*/velero.log file, search for the string Error executing hook or Error backing up item followed by the string timed out after.

   For example:

   time="2022-08-02T18:13:52Z" level=error msg="Error backing up item" backup=velero/instance-b7w8b error="timed out after 10m0s" ...

The error in the log example confirms your node is reaching the timeout limit. The default timeout limit is 10m 0s.

Determine how much time it takes the snapshot to complete on the node

To determine how long it takes for the snapshot to complete on the node, run:

kubectl exec -it postgres-0 -- bash ls /backup time pg_dumpall -U postgres --database=${POSTGRESQL_DATABASE:-"${POSTGRES_DB:-postgres}"} | gzip > /backup/backup.sql.gz

An example of the output:

real 12m27.395s user 6m52.251s sys 0m51.119s

In the example, it takes the snapshot 12 minutes and 27.395 seconds to complete on the node. This exceeds the ten-minute default timeout limit in Continuous Delivery for PE.

Change the backup timeout parameter

Change the default timeout limit on the node if the backup takes longer than ten minutes to complete.

1. In PAM UI, select the Config tab.
2. Select Advanced configuration and tuning and then check the We need to adjust the default settings box.
3. In the Continuous Delivery for PE database backup timeout field, enter a value greater than the time it takes the snapshot to complete on the node.
4. Select Save config.
5. On Version history, select Deploy on the latest available version.

If you need help determining the timeout limit in Continuous Delivery for PE based on your environment, submit a ticket to Support.