Services fail when Bolt was installed before July 2019 using open-source repositories on a primary server node in Puppet Enterprise – Puppet Support

Prior to July 2019, our documentation asked you to install Bolt using an open-source repository. When that repository is installed in Puppet Enterprise on the primary server node, your package manager might download a later version of the agent package that is incompatible with Puppet Enterprise. PE services such as pe-puppetserver might fail when your package manager upgrades OS packages and restarts services.

Even if you are not having issues, we urge you to avoid future issues if you installed Bolt prior to July 2019 by completing the steps in this article.

Error messages and logs

When PE services such as pe-puppetserver restart with the incompatible agent package, they fail with unrelated error messages.

For example:

ERROR [clojure-agent-send-pool-0] [p.t.internal] shutdown-on-error triggered because of exception!
java.lang.IllegalStateException: There was a problem adding a JRubyInstance to the pool.
at puppetlabs.services.jruby_pool_manager.impl.jruby_agents$eval42238$prime_pool_BANG___42243$fn__42247.invoke(jruby_agents.clj:73)

When puppet --test runs on a compiler, it might fail with messages similar to:

Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, This compile master has a PE version of '2018.1.4' and an aio puppet-agent version of '5.5.6', while the master of masters has version '2018.1.4' and an aio puppet-agent version of '6.4.2'. Please ensure that the PE versions are consistent across all masters by following the LEI upgrade documentation.

On an agent, during the agent run:

Error: /Stage[main]/Pe_repo::Platform::El_7_x86_64/Pe_repo::El[el-7-x86_64]/Pe_repo::Repo[el-7-x86_64 2019.0.2]/Pe_staging::Deploy[puppet-agent-el-7-x86_64.tar.gz]/Pe_staging::File[puppet-agent-el-7-x86_64.tar.gz]/Exec[/opt/puppetlabs/server/data/staging/pe_repo-puppet-agent-6.2.0/puppet-agent-el-7-x86_64.tar.gz]/returns: change from 'notrun' to ['0'] failed: 'curl -f -L -o puppet-agent-el-7-x86_64.tar.gz https://pm.puppetlabs.com/puppet-agent/2019.0.2/6.2.0/repos/puppet-agent-el-7-x86_64.tar.gz' returned 22 instead of one of [0]

Version and installation information

PE version: All supported versions
Installation type: Primary servers (formerly called a master or master of masters) including monolithic and split masters (in PE versions earlier than 2019.2), compilers (formerly called compile masters)

Solution

Complete these steps on all of your primary servers, including the master in a monolithic or split deployment (in versions earlier than 2019.2). Complete these steps on all of your compilers (formerly called compile masters).

Check to make sure the correct packages and package versions are installed. If you find open-source repositories, remove them. If an incompatible version of the agent is installed, downgrade it. To continue using Bolt on the primary server, install the new Puppet Tools repository.

Check the packages and package versions on your primary server

List Puppet and PE packages installed by your operating system.

On Red Hat Enterprise Linux (RHEL), CentOS, or other operating systems using RPM packages: Run rpm --query --all | grep --extended-regexp '^pe-|^puppet'

On Ubuntu, Debian, or other operating systems using DEB packages: Run dpkg-query --show --showformat '${Package}\n' | grep --extended-regexp '^pe-|^puppet'
Check the list for the presence of an open-source puppet6-release package, such as puppet6-release.noarch 6.0.0-5.el7 on RHEL 7. The puppet6-release package should be installed only on open-source primary servers — NEVER install it on the primary server in PE.
Each version of PE requires a specific version of the agent to be installed on the primary server. The version of the agent installed on the primary server in PE should never be later than its documented component version. Find the puppet-agent package in the list, confirm it against the documented component version for your version of PE, and note the documented version of the package. For example, the component versions table lists Puppet Agent 6.4.3 for PE 2019.1.1, so the package installed on a PE 2019.1.1 system should be puppet-agent-6.4.3-1.el7.x86_64

If you are having issues and don't have the open-source puppet6-release package and the version of the agent on your primary server is correct, please open a ticket with us.

If you have the puppet6-release package or a later version of the agent on your primary server than the documented component version, continue following the steps below.

Remove open-source platform repositories and downgrade the agent

If you found the puppet6-release package on your primary server, remove it.

On RHEL, CentOS, and other operating systems that use the yum package manager: Run sudo yum remove puppet6-release.

On Ubuntu, Debian, and other operating systems that use the apt package manager: Run sudo apt-get remove --purge puppet6-release, and then update your package repositories by running sudo apt-get update
If the installed puppet-agent package version is later than the documented component version, downgrade the package.

Note: If the installed puppet-agent package version is the same as the documented component for your version of PE, do not downgrade the puppet-agent package.
1. Stop the Puppet agent repository from updating.
  
  On RHEL, CentOS, and other operating systems that use the yum package manager: Add the following line to /etc/yum.conf:
```
exclude=puppet-agent
```
  On Ubuntu, Debian, and other operating systems that use the apt package manager: Run sudo apt-mark hold puppet-agent
2. Find the appropriate agent version. The agent package is located on the primary server in PE in /opt/puppetlabs/server/data/packages/public/<PE VERSION>/<OPERATING SYSTEM>-<AGENT VERSION>
  
  For example, on a primary server for PE 2019.1.1 RHEL 7, the path containing the packages is /opt/puppetlabs/server/data/packages/public/2019.1.1/el-7-x86_64-6.4.3/
3. Downgrade the puppet-agent package.
  
  For example, to downgrade the agent package on a primary server for PE 2019.1.1 running RHEL 7 or CentOS 7, run:
  
  sudo rpm -iUv --oldpackage /opt/puppetlabs/server/data/packages/public/2019.1.1/el-7-x86_64-6.4.3/puppet-agent-6.4.3-1.el7.x86_64.rpm
  
  On a primary server for PE 2019.1.1 running Ubuntu 18.04, run:
  
  sudo dpkg -i /opt/puppetlabs/server/data/packages/public/2019.1.1/ubuntu-18.04-amd64-6.4.3/puppet-agent_6.4.3-1bionic_amd64.deb
  
  When the downgrade is successful, the output of the command confirms the installed version of puppet before and after the downgrade, for example:
```
[root@pe-201911-server centos]# puppet --version
6.7.2

[root@pe-201911-server centos]# sudo rpm -iUv --oldpackage /opt/puppetlabs/server/data/packages/public/2019.1.1/el-7-x86_64-6.4.3/puppet-agent-6.4.3-1.el7.x86_64.rpm

Preparing packages...
puppet-agent-6.4.3-1.el7.x86_64
puppet-agent-6.7.2-1.el7.x86_64

[root@pe-201911-server centos]# puppet --version
6.4.3
```
4. On the primary server, complete the downgrade by running puppet agent --test

Install the Puppet Tools repository to use Bolt

In July of 2019, our documentation was updated so that Bolt is installed from the Puppet Tools repository and no longer causes the issues in this article. To continue using Bolt, install the Puppet Tools repository using the Bolt documentation.