When you add many certificates as part of a migration, upgrade, or maintenance, file sync storage might crash causing code deployment to fail and Puppet Server to crash.
Any certificates that you add to the primary server (called the master in versions before 2019.8) are automatically synchronized to the replica by file sync during code deployment. When many certificates are added, the JGit object used to sync files becomes very large. This increases the size of filesync/storage/ca-data.git
on the primary server. On the replica, the size of ca-data.git
also affects the filesync client requests that happen every 5 seconds as a part of the Code Manager workflow, causing code deployment to fail and Puppet Server to crash.
Error messages and logs
On the primary server, in the Puppet Server log (/var/log/puppetlabs/puppetserver/puppetserver.log
):
2020-07-27T20:28:36.537-05:00 WARN [qtp1893915144-225] [o.e.j.s.h.C.file_sync_git]
Internal error during upload-pack from /opt/puppetlabs/server/data/puppetserver/filesync/storage/ca-data.git
On the replica:
filesync/clinet/ca-data.git
is very large: /opt/puppetlabs/server/data/puppetserver/filesync/client/ca-data.git
Version and installation information
PE version: 2018.1 and 2019.0 and later
Solution
-
Calculate the size of the JGit object and note it for the following step. On the primary server, run:
cd /opt/puppetlabs/server/data/puppetserver/filesync/storage/ca-data.git; git cat-file -s <COMMIT HASH>
For example:
/opt/puppetlabs/server/data/puppetserver/filesync/storage/ca-data.git; git cat-file -s ad083a706a339e20e77c7ab28744cae3411d71671064849
-
The
stream-file-threshold
parameter of JGit sets the maximum allowed size for streaming files. By default, it is configured as 1MB (1024
) in PE 2018.1.8 and later. In previous versions, the default is 512KB (512
). To allow the JGit object to sync, increase thestream-file-threshold
parameter so that it is 512KB larger than the JGit object.Increase the value of
stream-file-threshold
based on the version of PE you're using.For 2018.1.8 (and later 2018.1), and 2019.0.3 and later
Update the value of
stream-file-threshold
in Hiera. This change will remain in place until you decrease the value.-
On the primary server, in
/etc/puppetlabs/enterprise/conf.d/pe.conf
edit the value forfile_sync_stream_file_threshold
. For example, to increase the parameter to approximately 1.5MB (1536
):puppet_enterprise::master::file_sync::file_sync_stream_file_threshold: 1536
-
To apply the change, on the primary server, run
puppet agent -t
.
For 2018.1.0 to 2018.1.7 and 2019.0.0 to 2019.0.2
Update the value of
stream-file-threshold
using thepe_hocon_setting
resource type by creating and applying a manifest. This change will last until the next Puppet run. Complete these steps on the master-
Create a manifest. Here's an example that increases the parameter to approximately 1.5MB (
1536
):file_sync.pp
pe_hocon_setting { 'file-sync.client.stream-file-threshold': path => '/etc/puppetlabs/puppetserver/conf.d/file-sync.conf', setting => 'file-sync.client.stream-file-threshold', value => 1536, }
-
On the master, apply the manifest. For our example manifest, run:
puppet apply /tmp/file_sync.pp
-
To complete the change, restart pe-puppetserver service.
puppet resource service pe-puppetserver ensure=stopped puppet resource service pe-puppetserver ensure=running
-
-
On the replica, remove the large JGit object by cleaning the filesync cache directory.
-
Stop pe-puppetserver service. Run:
puppet resource service pe-puppetserver ensure=stopped
-
Delete the filesync client directory. Run:
rm -rf /opt/puppetlabs/server/data/puppetserver/filesync/client
-
Start pe-puppetserver service. Run:
puppet resource service pe-puppetserver ensure=running
-
-
To fix the issue permanently, clean up unused certificates and ensure the size of the JGit object is less than the default value of 1024kb (or 512kb in PE 2018.1.7 and earlier). Then, decrease the value of
stream-file-threshold
using the steps in this article to change it back to its default value.If you're unable to clean up certificates and reduce the size of the JGit object, please open a ticket.
How can we improve this article?
4 comments
Hi Mahesh,
Is the default value 1024MB (1GB) or 1024KB (1MB)? Your feedback above:
By default, the
stream-file-threshold
parameter of jgit is configured as 1024 (1MB) in PE 2018.1.13 and later versions. In previous versions, it's configured as 512MB.is contradicting.
Kindly confirm if the default value is in MB or KB?
Regards,
Avikal Jain
I would guess 1GB (not 1MB), I will ask Mahes to confirm.
I confirmed with one of my other teammates that this should read: In previous versions, it's configured as 512KB.
I will go ahead and update that in the text of the article, apologies for the wait.
Not a problem, thanks for clarification.
We got further statistics from Puppet support on what this value should inline with number of nodes for others to refer:
For a 1MB threshold, we could have at maximum the following number of certificates at the associated certname length (+.pem).
10 character certnames: 24966
15 character certnames: 22310
20 character certnames: 20164
40 character certnames: 14562
60 character certnames: 11396
100 character certnames: 7942
Thus, it is actually impacted by the length of certname as well as number of certs. If your average certname is around 20 characters, setting file_sync_stream_file_threshold=1536 should be sufficient for 25k nodes.
Regards,
Avikal Jain
Please sign in to leave a comment.
Related articles