Skip to main content

Puppet Server "Error Reported" and Orchestrator Service "Unreachable" in the console in Puppet Enterprise 2017.2 and later : Directory corrupted when restoring

Gareth McGrillan
  • Updated

In the console, when you get a Puppet Server "Error Reported" and Orchestrator Service "Unreachable", it might be due to:

  • Port 8143 being blocked or
  • When restoring from a backup, the console services SSL directory is corrupted.

Before using the steps in this article, check if port 8143 is blocked. If port 8143 is open and you have the following error messages, use the steps below to fix the issue.

Error messages and logs

After restoring from a backup, the console services directory is corrupted and you get errors similar to the following in /var/log/puppetlabs/console-services/console-services.log:

2017-07-06 13:02:36,858 ERROR [p.o.api] Failed to make a connection to the database
java.sql.SQLTransientConnectionException: Timeout waiting for the database pool to become ready.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at clojure.lang.Reflector.invokeConstructor(Reflector.java:180)
        at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$reify__35232.getConnection(pool.clj:160)
        at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:307)
        at clojure.java.jdbc$get_connection.invoke(jdbc.clj:197)
        at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:878)
        at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:847)
        at clojure.java.jdbc$query.invokeStatic(jdbc.clj:915)
        at clojure.java.jdbc$query.invoke(jdbc.clj:884)
        at clojure.java.jdbc$query.invokeStatic(jdbc.clj:898)
        at clojure.java.jdbc$query.invoke(jdbc.clj:884)
        at puppetlabs.orchestrator.job$list_jobs.invokeStatic(job.clj:94)
        at puppetlabs.orchestrator.job$list_jobs.invoke(job.clj:87)
        at puppetlabs.orchestrator.api.jobs$list_jobs.invokeStatic(jobs.clj:106)
        at puppetlabs.orchestrator.api.jobs$list_jobs.invoke(jobs.clj:92)
        at puppetlabs.orchestrator.api.jobs$app$fn__37263.invoke(jobs.clj:196)
        at ring.middleware.keyword_params$wrap_keyword_params$fn__17935.invoke(keyword_params.clj:35)
        at ring.middleware.params$wrap_params$fn__16133.invoke(params.clj:64)
        at bidi.ring$fn__18750.invokeStatic(ring.clj:21)
        at bidi.ring$fn__18750.invoke(ring.clj:18)
        at bidi.ring$fn__18735$G__18730__18744.invoke(ring.clj:12)
        at puppetlabs.comidi$make_handler$fn__20441.invoke(comidi.clj:249)
        at puppetlabs.orchestrator.api$wrap_base_url$fn__37352.invoke(api.clj:145)
        at puppetlabs.metrics.http$fn__31068$wrap_with_request_metrics__31073$fn__31077$fn__31079$fn__31080$fn__31081.invoke(http.clj:152)
        at puppetlabs.metrics.http.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at puppetlabs.metrics.http$fn__31068$wrap_with_request_metrics__31073$fn__31077$fn__31079$fn__31080.invoke(http.clj:152)
        at puppetlabs.metrics.http.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
        at com.codahale.metrics.Timer.time(Timer.java:101)
        at puppetlabs.metrics.http$fn__31068$wrap_with_request_metrics__31073$fn__31077$fn__31079.invoke(http.clj:148)
        at puppetlabs.comidi$fn__20498$wrap_with_route_metadata__20503$fn__20504$fn__20506.invoke(comidi.clj:336)
        at puppetlabs.orchestrator.api$permission_check$fn__37344.invoke(api.clj:118)
        at puppetlabs.rbac_client.middleware.authentication$wrap_block_anonymous_access$fn__37315.invoke(authentication.clj:131)
        at puppetlabs.rbac_client.middleware.authentication$wrap_token_access_STAR_$fn__37300.invoke(authentication.clj:69)
        at puppetlabs.orchestrator.api$wrap_exception_handling$fn__37336.invoke(api.clj:102)
        at puppetlabs.orchestrator.api$wrap_catch_500s$fn__37347.invoke(api.clj:130)
        at puppetlabs.i18n.core$locale_negotiator$fn__124.invoke(core.clj:357)
        at puppetlabs.trapperkeeper.services.webserver.jetty9_core$ring_handler$fn__24031.invoke(jetty9_core.clj:423)
        at puppetlabs.trapperkeeper.services.webserver.jetty9_core.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1065)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
        at org.eclipse.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:299)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:497)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:748)

And this repeatedly:

2017-07-06 13:49:46,872 WARN  [p.j.pool] orchestrator - Error while attempting to connect to database, retrying.
java.sql.SQLTransientConnectionException: orchestrator - Connection is not available, request timed out after 30000ms.
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:147)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
        at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$fn__35210$fn__35211.invoke(pool.clj:126)
        at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$fn__35210.invoke(pool.clj:124)
        at clojure.core$binding_conveyor_fn$fn__4676.invoke(core.clj:1938)
        at clojure.lang.AFn.call(AFn.java:18)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: SSL error: Received fatal alert: decrypt_error
        at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:67)
        at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:351)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:137)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:215)
        at org.postgresql.Driver.makeConnection(Driver.java:406)
        at org.postgresql.Driver.connect(Driver.java:274)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
        at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:314)
        at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:171)
        at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:436)
        at com.zaxxer.hikari.pool.HikariPool.access$500(HikariPool.java:65)
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:567)
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:560)
        ... 4 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
        at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
        at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:62)
        ... 18 common frames omitted
2017-07-06 13:50:05,344 WARN  [p.j.core] Database status check timed out after 4 seconds.
2017-07-06 13:50:05,344 WARN  [p.j.core] Status check of db failed with error:
java.lang.InterruptedException: null
        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
        at java.util.concurrent.FutureTask.get(FutureTask.java:204)
        at clojure.core$deref_future.invokeStatic(core.clj:2210)
        at clojure.core$future_call$reify__6962.deref(core.clj:6688)
        at clojure.core$deref.invokeStatic(core.clj:2232)
        at clojure.core$deref.invoke(core.clj:2214)
        at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$reify__35232.getConnection(pool.clj:159)
        at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:307)
        at clojure.java.jdbc$get_connection.invoke(jdbc.clj:197)
        at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:878)
        at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:847)
        at clojure.java.jdbc$query.invokeStatic(jdbc.clj:915)
        at clojure.java.jdbc$query.invoke(jdbc.clj:884)
        at clojure.java.jdbc$query.invokeStatic(jdbc.clj:898)
        at clojure.java.jdbc$query.invoke(jdbc.clj:884)
        at puppetlabs.jdbc_util.core$db_up_QMARK_$fn__31952.invoke(core.clj:57)
        at clojure.core$binding_conveyor_fn$fn__4676.invoke(core.clj:1938)
        at clojure.lang.AFn.call(AFn.java:18)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
2017-07-06 13:50:10,107 WARN  [p.j.core] Database status check timed out after 4 seconds.
2017-07-06 13:50:10,107 WARN  [p.j.core] Status check of db failed with error:

Version and installation information

PE version: 2017.2 and later
OS: Any
Installation type:
2019.2 and later: Large
2019.1: Monolithic with compilers
2017.2 to 2019.0: Monolithic with compilers, split with compilers

Solution

Complete these steps on the master to fix the issue.

  1. Stop pe-orchestration-services:

    puppet resource service pe-orchestration-services ensure=stopped

  2. Stop pxp-agent:

    puppet resource service pxp-agent ensure=stopped

  3. Delete the pe-orchestration-services SSL directory. (Console services use this directory.) :

    rm /etc/puppetlabs/orchestration-services/ssl/*

  4. To regenerate the directory, run Puppet:

    puppet agent -t --verbose

  5. Start pe-orchestration-services:

    puppet resource service pe-orchestration-services ensure=running

  6. Start pxp-agent:

    puppet resource service pxp-agent ensure=running

Did this solve your problem?
0 out of 1 found this helpful

How can we improve this article?

0 comments

Please sign in to leave a comment.

Related articles