Resolve SSL errors when connecting to GitHub using the r10k shellgit provider in Puppet Enterprise

GitHub has dropped support for the TLSv1.0 and TLSv1.1 protocols as of February, 22nd 2018 leaving TLSv1.2 and newer as the only valid options for HTTPS traffic. Puppet Enterprise has shipped with an OpenSSL library that supports TLSv1.2 since the 2015.2.1 version of PE. However, integrations that use libraries provided by the operating system, such as the shellgit provider for r10k or Code Manager, may be affected on older versions of RedHat 6 and 7 variants (RHEL, CentOS, OEL, etc.) along with SLES 11.

Error messages and logs

Failure when deploying a module:

INFO     -> Deploying Puppetfile content /etc/puppetlabs/code-staging/environments/production/modules/jmx
ERROR    -> Command exited with non-zero exit code:
Command: git --git-dir /opt/puppetlabs/server/data/code-manager/worker-caches/deploy-pool-1/https---ucwfe2agnu9dp8e.delivery.puppetlabs.net-root-puppet-jmx.git fetch origin --prune
Stderr:
error:  while accessing https://ucwfe2agnu9dp8e.delivery.puppetlabs.net/root/puppet-jmx.git/info/refs

fatal: HTTP request failed
Exit code: 128

Failure when syncing a control repository:

ERROR        -> Unable to determine current branches for Git source 'puppet' (/etc/puppetlabs/code-staging/environments)
Original exception:
Command exited with non-zero exit code:
Command: git --git-dir /opt/puppetlabs/server/data/code-manager/git/https---ucwfe2agnu9dp8e.delivery.puppetlabs.net-root-control-repo.git fetch origin --prune
Stderr:
error:  while accessing https://ucwfe2agnu9dp8e.delivery.puppetlabs.net/root/control-repo.git/info/refs

fatal: HTTP request failed
Exit code: 128

In both cases a “HTTP request failed” error is reported. Re-running the Command: with extra logging enabled via the GIT_CURL_VERBOSE environment variable will reveal additional details about the failure.

For example, a CentOS 6.5 node might return:

# sudo -u pe-puppet GIT_CURL_VERBOSE=1 git --git-dir /opt/puppetlabs/server/data/code-manager/git/https---ucwfe2agnu9dp8e.delivery.puppetlabs.net-root-control-repo.git fetch origin --prune

* Closing connection #0
* Couldn't find host ucwfe2agnu9dp8e.delivery.puppetlabs.net in the .netrc file; using defaults
* About to connect() to ucwfe2agnu9dp8e.delivery.puppetlabs.net port 443 (#0)
*   Trying 10.32.125.103... * Connected to ucwfe2agnu9dp8e.delivery.puppetlabs.net (10.32.125.103) port 443 (#0)
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -5961
* Expire cleared
* Closing connection #0
error:  while accessing https://ucwfe2agnu9dp8e.delivery.puppetlabs.net/root/control-repo.git/info/refs

fatal: HTTP request failed

In this case, LibNSS returned an error code of -5961 indicating that the connection was terminated by the remote server.

Version and installation information

PE version: Any.
OS: RedHat variants 7.1 and earlier, RedHat variants 6.7 and earlier, SLES 11.

Solution

Use the rugged provider

If possible, configure Code Manager or r10k to use the rugged git provider instead of the shellgit provider. The rugged provider is linked against the OpenSSL library bundled with PE and will be able to use TLSv1.2.

Fix shellgit by updating OS packages

If the use of shellgit is required, then the operating system may need to be updated to a newer release along with packages that provide the libraries used by git to make HTTPS requests.

Facter can be used to check the operating system release:

/opt/puppetlabs/bin/facter os.release

And Puppet can be used to check package versions:

/opt/puppetlabs/bin/puppet resource package

RedHat 6

Operating systems derived from RedHat 6 will need to be updated to release 6.8 or later in order to access packages that support TLSv1.2. Once the OS is upgraded, the minimum required package versions for RedHat 6 systems are:

• curl-7.19.7-50.el6
• nss-3.19.1-6.el6

RedHat 7

Operating systems derived from RedHat 7 will need to be updated to release 7.2 or later in order to access packages that support TLSv1.2. Once the OS is upgraded, the minimum required package versions for RedHat 7 systems are:

• curl-7.29.0-24.el7
• nss-3.19.1-14.el7

SLES 11

SLES 11 ships with OpenSSL 0.9.8, which is too old to support TLSv1.2. An updated build of OpenSSL that supports TLSv1.2 is available from the Security Module repository for SLES 11 Service Pack 3 and SP4. However, Puppet Support has not been able to verify that this updated Package is picked up by git — consider contacting SUSE support for more details.