Fix a file sync storage crash for ca-data.git after adding certificates in Puppet Enterprise

When you add many certificates as part of a migration, upgrade, or maintenance, file sync storage might crash causing code deployment to fail and Puppet Server to crash.

Any certificates that you add to the primary server are automatically synchronized to the replica by file sync during code deployment. When many certificates are added, the JGit object used to sync files becomes very large. This increases the size of filesync/storage/ca-data.git on the primary server. On the replica, the size of ca-data.git also affects the filesync client requests that happen every 5 seconds as a part of the Code Manager workflow, causing code deployment to fail and Puppet Server to crash.

Error messages and logs

On the primary server, in the Puppet Server log (/var/log/puppetlabs/puppetserver/puppetserver.log):

2020-07-27T20:28:36.537-05:00 WARN [qtp1893915144-225] [o.e.j.s.h.C.file_sync_git] Internal error during upload-pack from /opt/puppetlabs/server/data/puppetserver/filesync/storage/ca-data.git org.eclipse.jgit.errors.LargeObjectException: ad083a706a339e20e77c7ab28744cae3411d7167 exceeds size limit

On the replica:

/opt/puppetlabs/server/data/puppetserver/filesync/client/ca-data.git is very large.

Version and installation information

PE version: All supported

Solution

1. Calculate the size of the JGit object and note it for the following step. On the primary server, run:

   cd /opt/puppetlabs/server/data/puppetserver/filesync/storage/ca-data.git; git cat-file -s <COMMIT HASH>

   For example:

   /opt/puppetlabs/server/data/puppetserver/filesync/storage/ca-data.git; git cat-file -s ad083a706a339e20e77c7ab28744cae3411d71671064849

2. The stream-file-threshold parameter of JGit sets the maximum allowed size for streaming files. By default, it is configured as 1MB (1024). To allow the JGit object to sync, increase the stream-file-threshold parameter so that it is 512KB larger than the JGit object.

3. Increase the value of stream-file-threshold in Hiera. This change will remain in place until you decrease the value. On the primary server, in /etc/puppetlabs/enterprise/conf.d/pe.conf edit the value for file_sync_stream_file_threshold. For example, to increase the parameter to approximately 1.5MB (1536):

   puppet_enterprise::master::file_sync::file_sync_stream_file_threshold: 1536

4. To apply the change, on the primary server, run puppet agent -t.