Code Manager deployments fail with an“Index is locked” error in Puppet Enterprise

Code deployments in Puppet Enterprise fail with an Index is locked error.

Error messages

2017-09-07T11:27:38.813759-04:00 frup7682 puppetserver[6337]: Exception in thread "main" java.lang.IllegalStateException: Index is locked. This can occur if the server crashed the last time it was running, and was unable to clean-up the index.lock file. This can be fixed by removing /opt/puppetlabs/server/data/puppetserver/filesync/storage/puppet-code/test.git/index.lock

To prevent the code staging directory from being overwritten, a lock file is used during code deployment. When Puppet Server shuts down during code deployment, the lock file is not cleaned up causing subsequent deployments to fail.

Version and installation information

PE version: All supported versions

Note: If you’re already using the puppetlabs-support_tasks module, update it to the most recent version.

Solution

Fix the issue by removing the lock file. You can install a module and use a task to remove the file, or you can remove the file manually.

To use a task to fix the issue:

Use these steps to download a module and run a task that resolves the error by stopping the pe-puppetserver service, removing file sync locks, and starting the pe-puppetserver service.

1. Download and install the puppetlabs-support_tasks module which includes the task for this solution.

2. In the console, in the Orchestration section, click Tasks. Click Run a task in the upper right corner of the Tasks page. Under Tasks select support_tasks:st0267_clear_file_sync_locks.

3. In the Code environment field, select the environment where you installed the module containing the task. This defaults to production. Under Select targets select Node list. In the search field, start typing in the name of the primary server node, and click Search. Select the primary server node.

4. Click Run task.

   The task exits with the following output when the lock file is removed: success - filesync lock removed or not present. With the lock file removed, you should be able to deploy code.

   If it’s run on the wrong node, the task exits with success - Agent Node Not Proceeding or success - pe-puppetserver not installed. If you get this message, run the task on the primary server node.

   It might also fail with fail "Could not stop pe-puppetserver ", fail "Could not start pe-puppetserver ", or fail "Could not remove lockfile".

To fix the problem manually:

1. Stop Puppet Server service. On the primary server, run:

   puppet resource service pe-puppetserver ensure=stopped

2. Remove the lock file. On the primary server, run:

   find /opt/puppetlabs/server/data/puppetserver/filesync/ -type f -name 'index.lock' -delete

3. Start Puppet Server service. On the primary server, run:

   puppet resource service pe-puppetserver ensure=running

   With the lock file removed, you should be able to deploy code.