Resolve a “too many open files” error message in Puppet Enterprise

You get a too many open files error message in your Puppet code or the Puppet Server logs.

PE services can require as much as one file handle per connected client. The default ulimit settings for most operating systems only support up to about 200 clients. When that number is exceeded, you get a Too many open files error. To support more clients, you need to increase the number of file handles allowed.

Error messages and logs

Error messages

2022-10-18T21:09:13.949Z ERROR [clojure-agent-send-off-pool-161265] [p.e.s.f.file-sync-client-core] Fatal error during file sync, requesting shutdown. java.io.FileNotFoundException: /opt/puppetlabs/server/data/puppetserver/filesync/client/puppet-code/prd_verumsre_p1_p01.git/config (Too many open files)

or

[root@URL ~]# r10k deploy environment -m ERROR -> Unable to determine current branches for Git source 'puppet' (/etc/puppetlabs/code/environments) Original exception: failed to create locked file '/opt/puppetlabs/server/data/puppetserver/r10k/ssh---puppet@URL.com-29418-cah-puppet/objects/pack/pack_git2_luidx.lock': Too many open files

Logs

In the Puppet Server log, at /var/log/puppetlabs/puppetserver/puppetserver.log:

2022-10-18T20:25:53.685Z WARN [qtp1713162490-359805-acceptor-1@4f340775-ServerConnector@3fefd630{SSL, (ssl, http/1.1)}{0.0.0.0:8142}] [o.e.j.s.AbstractConnector] java.io.IOException: Too many open files 2022-10-18T20:25:53.700Z ERROR [qtp1282485519-5004569] [puppetserver] Puppet Server Error: Too many open files - Too many open files

Or

ERROR [qtp1282485519-4994318] [puppetserver] Puppet Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Too many open files - Too many open files (file:

Version and installation information

PE version: All supported

Solution

To fix the issue, increase the ulimit for the PE service generating the error. In this case, the error was in the Puppet Server log, so increase the ulimit for Puppet Server.