Fix a file sync storage crash for ca-data.git after adding certificates in Puppet Enterprise 2018.1 and 2019.0 and later – Puppet Support

When you add many certificates as part of a migration, upgrade, or maintenance, file sync storage might crash causing code deployment to fail and Puppet Server to crash.

Any certificates that you add to the primary server (called the master in versions before 2019.8) are automatically synchronized to the replica by file sync during code deployment. When many certificates are added, the JGit object used to sync files becomes very large. This increases the size of filesync/storage/ca-data.git on the primary server. On the replica, the size of ca-data.git also affects the filesync client requests that happen every 5 seconds as a part of the Code Manager workflow, causing code deployment to fail and Puppet Server to crash.

Error messages and logs

On the primary server, in the Puppet Server log (/var/log/puppetlabs/puppetserver/puppetserver.log):

2020-07-27T20:28:36.537-05:00 WARN [qtp1893915144-225] [o.e.j.s.h.C.file_sync_git] Internal error during upload-pack from /opt/puppetlabs/server/data/puppetserver/filesync/storage/ca-data.git org.eclipse.jgit.errors.LargeObjectException: ad083a706a339e20e77c7ab28744cae3411d7167 exceeds size limit

On the replica:

filesync/clinet/ca-data.git is very large: /opt/puppetlabs/server/data/puppetserver/filesync/client/ca-data.git

Version and installation information

PE version: 2018.1 and 2019.0 and later

Solution

Calculate the size of the JGit object and note it for the following step. On the primary server, run:

cd /opt/puppetlabs/server/data/puppetserver/filesync/storage/ca-data.git; git cat-file -s <COMMIT HASH>

For example:

/opt/puppetlabs/server/data/puppetserver/filesync/storage/ca-data.git; git cat-file -s ad083a706a339e20e77c7ab28744cae3411d71671064849
The stream-file-threshold parameter of JGit sets the maximum allowed size for streaming files. By default, it is configured as 1MB (1024) in PE 2018.1.8 and later. In previous versions, the default is 512KB (512). To allow the JGit object to sync, increase the stream-file-threshold parameter so that it is 512KB larger than the JGit object.

Increase the value of stream-file-threshold based on the version of PE you're using.

For 2018.1.8 (and later 2018.1), and 2019.0.3 and later

Update the value of stream-file-thresholdin Hiera. This change will remain in place until you decrease the value.
1. On the primary server, in /etc/puppetlabs/enterprise/conf.d/pe.conf edit the value for file_sync_stream_file_threshold. For example, to increase the parameter to approximately 1.5MB (1536):
  
  puppet_enterprise::master::file_sync::file_sync_stream_file_threshold: 1536
2. To apply the change, on the primary server, run puppet agent -t.
For 2018.1.0 to 2018.1.7 and 2019.0.0 to 2019.0.2

Update the value of stream-file-threshold using the pe_hocon_setting resource type by creating and applying a manifest. This change will last until the next Puppet run. Complete these steps on the master
1. Create a manifest. Here's an example that increases the parameter to approximately 1.5MB (1536):
  
  file_sync.pp
```
pe_hocon_setting { 'file-sync.client.stream-file-threshold':
path => '/etc/puppetlabs/puppetserver/conf.d/file-sync.conf',
setting => 'file-sync.client.stream-file-threshold',
value => 1536,
}
```
2. On the master, apply the manifest. For our example manifest, run:
  
  puppet apply /tmp/file_sync.pp
3. To complete the change, restart pe-puppetserver service.
```
puppet resource service pe-puppetserver ensure=stopped
puppet resource service pe-puppetserver ensure=running
```
On the replica, remove the large JGit object by cleaning the filesync cache directory.
1. Stop pe-puppetserver service. Run:
  
  puppet resource service pe-puppetserver ensure=stopped
2. Delete the filesync client directory. Run:
  
  rm -rf /opt/puppetlabs/server/data/puppetserver/filesync/client
3. Start pe-puppetserver service. Run:
  
  puppet resource service pe-puppetserver ensure=running
To fix the issue permanently, clean up unused certificates and ensure the size of the JGit object is less than the default value of 1024kb (or 512kb in PE 2018.1.7 and earlier). Then, decrease the value of stream-file-threshold using the steps in this article to change it back to its default value.

If you're unable to clean up certificates and reduce the size of the JGit object, please open a ticket.

How can we improve this article?

4 comments

Avikal Jain

October 24, 2020 11:23
Hi Mahesh,

Is the default value 1024MB (1GB) or 1024KB (1MB)? Your feedback above:

By default, the stream-file-threshold parameter of jgit is configured as 1024 (1MB) in PE 2018.1.13 and later versions. In previous versions, it's configured as 512MB.

is contradicting.

Kindly confirm if the default value is in MB or KB?

Regards,

Avikal Jain
0
Suzie Baunsgard

October 24, 2020 16:10
I would guess 1GB (not 1MB), I will ask Mahes to confirm.
0
Suzie Baunsgard

October 27, 2020 17:01
I confirmed with one of my other teammates that this should read: In previous versions, it's configured as 512KB.

I will go ahead and update that in the text of the article, apologies for the wait.
0
Avikal Jain

October 28, 2020 02:40
Not a problem, thanks for clarification.

We got further statistics from Puppet support on what this value should inline with number of nodes for others to refer:

For a 1MB threshold, we could have at maximum the following number of certificates at the associated certname length (+.pem).
10 character certnames: 24966
15 character certnames: 22310
20 character certnames: 20164
40 character certnames: 14562
60 character certnames: 11396
100 character certnames: 7942

Thus, it is actually impacted by the length of certname as well as number of certs. If your average certname is around 20 characters, setting file_sync_stream_file_threshold=1536 should be sufficient for 25k nodes.

Regards,

Avikal Jain
0

Please sign in to leave a comment.

Error messages and logs

Version and installation information

Solution

Related articles