Upgrade Puppet Enterprise with high availability enabled and no agent downtime

If your agents can’t be down for any period of time and you have high availability enabled, you can upgrade PE with no agent downtime by editing your iptables rules to prevent traffic to the primary server. If this situation doesn’t apply to you, please use our documentation to upgrade.

Version and installation information

PE version: All supported versions
Installation type: High availability, with or without compilers

Notes:

• Our terminology changed. A primary server was formerly called a master of masters and a compiler was formerly called a compile master.

• The links in these instructions are to PE 2018.1 documentation, please choose the right version for your deployment.

Solution

1. Back up your primary server using the steps in our documentation. If your primary server is running as a VM, take a snapshot before upgrading.

2. Back up iptables rules by running iptables-save > iptables-backup

3. Use the following rules to configure iptables to prevent connections to the primary server during the upgrade process and force all traffic to the replica. Make sure to replace with the FQDN for your primary server.

   Note: These rules are in addition to any that are specific to your organization.

    :INPUT DROP [0:0]
    :FORWARD DROP [0:0]
    :OUTPUT DROP [0:0]
    -A OUTPUT -o eth0 -d <PRIMARY_SERVER_FQDN> -j ACCEPT
    -A OUTPUT -p tcp --dport 80 -j ACCEPT
    -A OUTPUT -p tcp --dport 443 -j ACCEPT
    -A OUTPUT -p tcp --dport 8140 -d <PRIMARY_SERVER_FQDN> -j ACCEPT
    -A OUTPUT -o lo -m comment --comment "001 accept all to lo interface" -j ACCEPT
    -A OUTPUT -p udp --dport 53 -j ACCEPT
    -A OUTPUT -p tcp --dport 53 -j ACCEPT
    -A OUTPUT -p udp --dport 67 -j ACCEPT
    -A OUTPUT -m state --state RELATED,ESTABLISHED -m comment --comment "002 accept related established rules" -j ACCEPT
    -A OUTPUT -j LOG --log-prefix "OUTPUT:DROP:" --log-level 6
    -A OUTPUT -j DROP
    -A OUTPUT -m comment --comment "999 drop all" -j DROP
    -A INPUT -p tcp --sport 53 -j ACCEPT
    -A INPUT -p udp --sport 53 -j ACCEPT
    -A INPUT -s <PRIMARY_SERVER_FQDN> -j ACCEPT
    -A INPUT -i lo -m comment --comment "001 accept all to lo interface" -j ACCEPT
    -A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "002 accept related established rules" -j ACCEPT
    -A INPUT -p tcp -m multiport --dports 22 -m comment --comment "100 allow ssh access" -j ACCEPT
    -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
    -A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "002 accept related established rules" -j ACCEPT
    -A INPUT -j LOG --log-prefix "INPUT:DROP:" --log-level 6
    -A INPUT -j DROP
    -A INPUT -m comment --comment "999 drop all" -j DROP
    COMMIT

4. Upgrade your primary server using the steps in our documentation.

5. On the primary server, run puppet agent -t

6. Until everything is upgraded, some compilers must remain on the older version of PE. Upgrade half of your compilers.

7. Remove the added iptables rules. If you backed up itpables, you can restore your backup by running iptables-restore < iptables-backup. If you didn’t back up iptables, use the following commands to remove the added rules:

   iptables --policy INPUT   ACCEPT;
   iptables --policy OUTPUT  ACCEPT;
   iptables --policy FORWARD ACCEPT;
   iptables -Z;
   iptables -F;
   iptables -X;

8. Depending on the version you are upgrading from, you’ll need to either upgrade your existing replica or forget your replica and provision a new one.

   If you’re upgrading from 2017.3 and later: upgrade the replica using the steps in our documentation.

   In earlier versions of PE: forget the replica and provision a new one using the steps in our documentation.

9. Upgrade the other half of your compilers.

10. Upgrade agents using the steps in our documentation.

11. Verify functionality by running puppet agent -t on an agent, primary server, replica, and compiler. To verify that PE services are up and running, on the primary server, run puppet infrastructure status.