Backup and restore migration from Puppet Enterprise 2018.1 to 2019.8 for standard and large installations with or without disaster recovery – Puppet Support

When upgrading from Puppet Enterprise 2018.1 to 2019.8, you can test functionality and minimize Puppet service outages by migrating from your old (existing) primary server to a new primary server.

Complete these steps to:

Back up your old PE 2018.1 installation.
Install PE 2018.1 on a new primary server node.
Using the backup you created, restore on the new primary server.
Upgrade the new primary server to PE 2019.8 (with no agents).
Ensure that everything is functional by pointing a few agents at the new primary server and upgrading them.
Point the rest of the agents at the new server and upgrade them.
Decommission the old PE 2018.1 installation.

Version and installation information

PE version: 2018.1.18, 2019.8.5

Installation type: Standard or large, with or without disaster recovery

If you have a standard installation and don’t use disaster recovery (you don’t have a replica), there are steps in our documentation for later versions of PE: 2019.8 and 2021.1.

Solution

Use these steps to migrate from PE 2018.1 to 2019.8.

Note:

You can choose the data that you want to keep during the backup and restore process. If you don’t back up and restore PuppetDB data, it dramatically speeds up the process. If you’re not sure if you want to keep the data, back it up, and when you restore, you can make a decision to keep it or not.
These steps assume that you are using the console to configure PE. If you use Hiera, modify them.
If you use SSH as your Bolt transport, you can automate most of the steps in this process.

Step 1: Back up your old installation

Before you back up, ensure that old hostnames for PE infrastructure will be valid after you restore by adding DNS alt names to puppet.conf on the primary server of your old installation. In [main], add a dns_alt_names entry with the hostnames (FQDN and short name) for your old installation’s primary server and load balancers.
Back up only what you need. You can make the process faster by:
- Backing up only certificates and configuration.
- Not including reports and catalogs stored in PuppetDB in the backup. If your organization requires you to keep PuppetDB data, keep reports, since facts and catalogs are repopulated during the next agent run after restoring the backup.
Back up your infrastructure using the steps in our documentation..

Step 2: Install PE on a new node

Create a new primary server by installing PE 2018.1 on a new node.
Hardcoded hostnames for the backup are not updated until the first Puppet run after the restore process is complete. To ensure that old hostnames are valid throughout the migration process, on the new primary server in /etc/hosts, add DNS alt names (FQDNs and short names) for your old installation’s primary server, compilers, replica, and load balancers.

Step 3: Restore PE

Using the backup you created, restore your installation on the new primary server.

If you backed up PuppetDB data, you can choose to restore it or not during this step.

If you had a replica, the pglogical extension was set up, which causes errors. Clean up steps are in the next section.

Step 4: Clean up installations with disaster recovery (replicas)

If your old installation didn’t use disaster recovery (it didn’t have replicas), skip to the next section.

If your old installation used disaster recovery and had a replica, complete these steps on the new primary server to clean up the replica.

Remove the pglogical extension. Run:

dbs=($(su - pe-postgres -s '/bin/bash' -c "/opt/puppetlabs/server/bin/psql --tuples-only -c \"SELECT spcname FROM pg_tablespace where spcname != 'pe-puppetdb' and spcname like 'pe-%'\";")) for d in "${dbs[@]}"; do su - pe-postgres -s '/bin/bash' -c "/opt/puppetlabs/server/bin/psql --tuples-only -d \"$d\" -c 'drop extension pglogical cascade;'"; done

Restart PE services. Run:

puppet resource service pe-postgresql ensure=stopped
puppet resource service pe-postgresql ensure=running
puppet resource service pe-puppetdb ensure=stopped
puppet resource service pe-puppetdb ensure=running
puppet resource service pe-console-services ensure=stopped
puppet resource service pe-console-services ensure=running
puppet resource service pe-nginx ensure=stopped
puppet resource service pe-nginx ensure=running
puppet resource service pe-puppetserver ensure=stopped
puppet resource service pe-puppetserver ensure=running
puppet resource service pe-orchestration-services ensure=stopped
puppet resource service pe-orchestration-services ensure=running
puppet resource service pxp-agent ensure=stopped
puppet resource service pxp-agent ensure=running

Remove the client tools services.conf file to ensure that the new primary server hostname is used in the new installation. Run:
- rm -rf /etc/puppetlabs/client-tool/services.conf
Finish configuring the new primary server. Run:
- puppet infrastructure configure --no-recover
- puppet agent -t
Forget the old replica. Run:
- puppet access login
- puppet infrastructure forget <OLD_REPLICA_CERTIFICATE_NAME>

Step 5: Clean up classification on the new primary server

Complete these steps on the new primary server.

Clean up the old primary server and compiler hostnames by running puppet node purge <HOSTNAME>.
Install the WhatsARanjit-node_manager module to output classifier information from the default node groups to a Puppet manifest.

Run:

(for i in 'All Environments' 'All Nodes' 'Development environment' 'Development one-time run exception' 'PE Agent' 'PE Certificate Authority' 'PE Console' 'PE Database' 'PE Infrastructure' 'PE Infrastructure Agent' 'PE Master' 'PE Orchestrator' 'PE PuppetDB' 'Production environment'; do puppet resource node_group "${i}";done) > /tmp/classifier_default_node_groups.pp
In the generated manifest, change the following:
- For all settings, replace the old primary server name with the new primary server name.
- Remove rules that refer to the old compilers.
- Remove rules that refer to the old replica.
- In every node_group resource, remove id parameters.
- In the All Nodes node_group resource, remove the parent parameter.
Push changes to classification to ensure that the new hostnames replace the old ones. Run puppet apply against the edited manifest.
Push classification changes to configuration files. Run puppet agent -t

Step 6: Upgrade the new installation to PE 2019.8

Upgrade the new primary server to PE 2019.8. Don’t upgrade your agents or remove /etc/hosts modifications yet. You’ll do both in later steps. Removing DNS overrides too early will cause DNS issues if the old primary server used a CNAME or additional A and/or AAAA records.
Provision new replicas and compilers for the new installation. Set dns_alt_names for compilers using the command line option when running puppet infrastructure provision compiler so that agents communicate with them.
Point a few agents from the old installation to the new installation. An easy way to do this is by running a task with the Bolt task runner.
1. Download and install Bolt, if you don’t already have it installed.
2. Update the puppet.conf file to point all agents at the new primary server by running:
  
  bolt task run puppet_conf action=set section=agent setting=server value=<RESTORE_HOSTNAME> --targets <COMMA-SEPARATED LIST OF NODES>
  
  bolt task run puppet_conf action=set section=main setting=server value=<RESTORE_HOSTNAME> --targets <COMMA-SEPARATED LIST OF NODES>
3. Run puppet agent -t --no-use_cached_catalog on the newly restored primary server twice to apply changes and then restart services.
4. Run puppet agent -t --no-use_cached_catalog on all agent nodes to test connection to the new primary server.
  
  Note: If agents fail to communicate with the new PE 2019.8 infrastructure with SSL errors, TLSv1 might be disabled. It is often disabled in older operating systems. Enable TLSv1 to allow communication.
Upgrade the agents you pointed at the new primary server to PE 2019.8.
Point the rest of the agents at the new primary server by updating DNS. Update static values in the classifier and remove DNS overrides in /etc/hosts. If the agents fail to communicate with the new infrastructure, you might need to enable TLSv1 to allow communication.
Upgrade the remaining agents.
Decommission the old installation.