After running jobs, if you get Unrecoverable error occurred during database initialization
errors in the console or in the orchestration services log, you might have one or more scheduled jobs that are stuck. You can clear them using the steps in this article.
Error messages and logs
Errors in the orchestration services log (/var/log/puppetlabs/orchestration-services/orchestration-services.log
) similar to the following:
ERROR [clojure-agent-send-off-pool-1] [p.o.scheduled] Failure occurred while setting up jobs, cancelling scheduled jobs.
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
ERROR [clojure-agent-send-off-pool-1] [p.j.pool] orchestrator - An error was encountered during database migration.
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
ERROR [clojure-agent-send-off-pool-232] [p.t.s.s.status-core] Status check for orchestrator-service threw an exception
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Unrecoverable error occurred during database initialization.
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
ERROR [qtp1961250322-172] [p.o.api] Exception while handling HTTP GET request at /orchestrator/v1/jobs
java.lang.RuntimeException: Unrecoverable error occurred during database initialization.
at java.base/jdk.internal.reflect.GeneratedConstructorAccessor60.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
Version and installation information
PE version: All supported versions
Solution
-
To output a list of jobs in the pe-orchestrator database, in the command line, on the PE-PostgreSQL node (in most cases the primary server), run the following:
su - pe-postgres -s '/bin/bash' -c "/opt/puppetlabs/server/bin/psql -d pe-orchestrator -c '\x' -c 'select * from scheduled_jobs;'"
Note: In 2019.2 and later, PE-PostgreSQL might be on the primary server or an external node. In earlier versions, it can be on the master node (in a monolithic installation), PuppetDB node (in a split installation), or on an external node.
-
To remove past jobs, run:
su - pe-postgres -s '/bin/bash' -c "/opt/puppetlabs/server/bin/psql -d pe-orchestrator -c '\x' -c 'select * from scheduled_jobs where next_run < CURRENT_TIMESTAMP;' -c 'delete from scheduled_jobs where next_run < CURRENT_TIMESTAMP;'"
-
Optional: If you want to clear all scheduled jobs, use the following command. If you have any future jobs scheduled, you need to manually re-add them after running this command.
su - pe-postgres -s '/bin/bash' -c "/opt/puppetlabs/server/bin/psql -d pe-orchestrator -c 'delete from scheduled_jobs;'"
-
To complete the purge, on the primary server, restart the orchestrator:
puppet resource service pe-orchestration-services ensure=stopped puppet resource service pe-orchestration-services ensure=running
After that, the errors should not recur. If they do, please open a new ticket with Support so that we can help you resolve it.
How can we improve this article?
0 comments
Please sign in to leave a comment.
Related articles