When you add many certificates as part of a migration, upgrade, or maintenance, file sync storage might crash causing code deployment to fail and Puppet Server to crash.
Any certificates that you add to the primary server are automatically synchronized to the replica by file sync during code deployment. When many certificates are added, the JGit object used to sync files becomes very large. This increases the size of
filesync/storage/ca-data.git on the primary server. On the replica, the size of
ca-data.git also affects the filesync client requests that happen every 5 seconds as a part of the Code Manager workflow, causing code deployment to fail and Puppet Server to crash.
Error messages and logs
On the primary server, in the Puppet Server log (
2020-07-27T20:28:36.537-05:00 WARN [qtp1893915144-225] [o.e.j.s.h.C.file_sync_git] Internal error during upload-pack from /opt/puppetlabs/server/data/puppetserver/filesync/storage/ca-data.git org.eclipse.jgit.errors.LargeObjectException: ad083a706a339e20e77c7ab28744cae3411d7167 exceeds size limit
On the replica:
/opt/puppetlabs/server/data/puppetserver/filesync/client/ca-data.git is very large.
Version and installation information
PE version: All supported
Calculate the size of the JGit object and note it for the following step. On the primary server, run:
cd /opt/puppetlabs/server/data/puppetserver/filesync/storage/ca-data.git; git cat-file -s <COMMIT HASH>
/opt/puppetlabs/server/data/puppetserver/filesync/storage/ca-data.git; git cat-file -s ad083a706a339e20e77c7ab28744cae3411d71671064849
stream-file-thresholdparameter of JGit sets the maximum allowed size for streaming files. By default, it is configured as 1MB (
1024). To allow the JGit object to sync, increase the
stream-file-thresholdparameter so that it is 512KB larger than the JGit object.
Increase the value of
stream-file-thresholdin Hiera. This change will remain in place until you decrease the value. On the primary server, in
/etc/puppetlabs/enterprise/conf.d/pe.confedit the value for
file_sync_stream_file_threshold. For example, to increase the parameter to approximately 1.5MB (
To apply the change, on the primary server, run
puppet agent -t.
How can we improve this article?
Please sign in to leave a comment.