Resolve "Unrecoverable error occurred during database initialization" errors in the orchestrator by purging stuck jobs

After running jobs, if you get Unrecoverable error occurred during database initialization errors in the console or in the orchestration services log, you might have one or more scheduled jobs that are stuck. You can clear them using the steps in this article.

Error messages and logs

Errors in the orchestration services log (/var/log/puppetlabs/orchestration-services/orchestration-services.log) similar to the following:

ERROR [clojure-agent-send-off-pool-1] [p.o.scheduled] Failure occurred while setting up jobs, cancelling scheduled jobs.
java.net.ConnectException: Connection refused
    at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)

ERROR [clojure-agent-send-off-pool-1] [p.j.pool] orchestrator - An error was encountered during database migration.
java.net.ConnectException: Connection refused
    at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)

ERROR [clojure-agent-send-off-pool-232] [p.t.s.s.status-core] Status check for orchestrator-service threw an exception
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Unrecoverable error occurred during database initialization.
    at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)

ERROR [qtp1961250322-172] [p.o.api] Exception while handling HTTP GET request at /orchestrator/v1/jobs
java.lang.RuntimeException: Unrecoverable error occurred during database initialization.
    at java.base/jdk.internal.reflect.GeneratedConstructorAccessor60.newInstance(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)

Version and installation information

PE version: All supported versions

Solution

1. To output a list of jobs in the pe-orchestrator database, in the command line, on the PE-PostgreSQL node (in most cases the primary server), run the following:

   su - pe-postgres -s '/bin/bash' -c "/opt/puppetlabs/server/bin/psql -d pe-orchestrator -c '\x' -c 'select * from scheduled_jobs;'"

   Note: In 2019.2 and later, PE-PostgreSQL might be on the primary server or an external node. In earlier versions, it can be on the master node (in a monolithic installation), PuppetDB node (in a split installation), or on an external node.

2. To remove past jobs, run:

   su - pe-postgres -s '/bin/bash' -c "/opt/puppetlabs/server/bin/psql -d pe-orchestrator -c '\x' -c 'select * from scheduled_jobs where next_run < CURRENT_TIMESTAMP;' -c 'delete from scheduled_jobs where next_run < CURRENT_TIMESTAMP;'"

3. Optional: If you want to clear all scheduled jobs, use the following command. If you have any future jobs scheduled, you need to manually re-add them after running this command.

   su - pe-postgres -s '/bin/bash' -c "/opt/puppetlabs/server/bin/psql -d pe-orchestrator -c 'delete from scheduled_jobs;'"

4. To complete the purge, on the primary server, restart the orchestrator:

   puppet resource service pe-orchestration-services ensure=stopped
   puppet resource service pe-orchestration-services ensure=running

After that, the errors should not recur. If they do, please open a new ticket with Support so that we can help you resolve it.