In the console, when you get a Puppet Server "Error Reported" and Orchestrator Service "Unreachable", it might be due to:
- Port 8143 being blocked or
- When restoring from a backup, the console services SSL directory is corrupted.
Before using the steps in this article, check if port 8143 is blocked. If port 8143 is open and you have the following error messages, use the steps below to fix the issue.
Error messages and logs
After restoring from a backup, the console services directory is corrupted and you get errors similar to the following in /var/log/puppetlabs/console-services/console-services.log
:
2017-07-06 13:02:36,858 ERROR [p.o.api] Failed to make a connection to the database
java.sql.SQLTransientConnectionException: Timeout waiting for the database pool to become ready.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at clojure.lang.Reflector.invokeConstructor(Reflector.java:180)
at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$reify__35232.getConnection(pool.clj:160)
at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:307)
at clojure.java.jdbc$get_connection.invoke(jdbc.clj:197)
at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:878)
at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:847)
at clojure.java.jdbc$query.invokeStatic(jdbc.clj:915)
at clojure.java.jdbc$query.invoke(jdbc.clj:884)
at clojure.java.jdbc$query.invokeStatic(jdbc.clj:898)
at clojure.java.jdbc$query.invoke(jdbc.clj:884)
at puppetlabs.orchestrator.job$list_jobs.invokeStatic(job.clj:94)
at puppetlabs.orchestrator.job$list_jobs.invoke(job.clj:87)
at puppetlabs.orchestrator.api.jobs$list_jobs.invokeStatic(jobs.clj:106)
at puppetlabs.orchestrator.api.jobs$list_jobs.invoke(jobs.clj:92)
at puppetlabs.orchestrator.api.jobs$app$fn__37263.invoke(jobs.clj:196)
at ring.middleware.keyword_params$wrap_keyword_params$fn__17935.invoke(keyword_params.clj:35)
at ring.middleware.params$wrap_params$fn__16133.invoke(params.clj:64)
at bidi.ring$fn__18750.invokeStatic(ring.clj:21)
at bidi.ring$fn__18750.invoke(ring.clj:18)
at bidi.ring$fn__18735$G__18730__18744.invoke(ring.clj:12)
at puppetlabs.comidi$make_handler$fn__20441.invoke(comidi.clj:249)
at puppetlabs.orchestrator.api$wrap_base_url$fn__37352.invoke(api.clj:145)
at puppetlabs.metrics.http$fn__31068$wrap_with_request_metrics__31073$fn__31077$fn__31079$fn__31080$fn__31081.invoke(http.clj:152)
at puppetlabs.metrics.http.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
at com.codahale.metrics.Timer.time(Timer.java:101)
at puppetlabs.metrics.http$fn__31068$wrap_with_request_metrics__31073$fn__31077$fn__31079$fn__31080.invoke(http.clj:152)
at puppetlabs.metrics.http.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
at com.codahale.metrics.Timer.time(Timer.java:101)
at puppetlabs.metrics.http$fn__31068$wrap_with_request_metrics__31073$fn__31077$fn__31079.invoke(http.clj:148)
at puppetlabs.comidi$fn__20498$wrap_with_route_metadata__20503$fn__20504$fn__20506.invoke(comidi.clj:336)
at puppetlabs.orchestrator.api$permission_check$fn__37344.invoke(api.clj:118)
at puppetlabs.rbac_client.middleware.authentication$wrap_block_anonymous_access$fn__37315.invoke(authentication.clj:131)
at puppetlabs.rbac_client.middleware.authentication$wrap_token_access_STAR_$fn__37300.invoke(authentication.clj:69)
at puppetlabs.orchestrator.api$wrap_exception_handling$fn__37336.invoke(api.clj:102)
at puppetlabs.orchestrator.api$wrap_catch_500s$fn__37347.invoke(api.clj:130)
at puppetlabs.i18n.core$locale_negotiator$fn__124.invoke(core.clj:357)
at puppetlabs.trapperkeeper.services.webserver.jetty9_core$ring_handler$fn__24031.invoke(jetty9_core.clj:423)
at puppetlabs.trapperkeeper.services.webserver.jetty9_core.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1065)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:299)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:748)
And this repeatedly:
2017-07-06 13:49:46,872 WARN [p.j.pool] orchestrator - Error while attempting to connect to database, retrying.
java.sql.SQLTransientConnectionException: orchestrator - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:147)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$fn__35210$fn__35211.invoke(pool.clj:126)
at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$fn__35210.invoke(pool.clj:124)
at clojure.core$binding_conveyor_fn$fn__4676.invoke(core.clj:1938)
at clojure.lang.AFn.call(AFn.java:18)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: SSL error: Received fatal alert: decrypt_error
at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:67)
at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:351)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:137)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:215)
at org.postgresql.Driver.makeConnection(Driver.java:406)
at org.postgresql.Driver.connect(Driver.java:274)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:314)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:171)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:436)
at com.zaxxer.hikari.pool.HikariPool.access$500(HikariPool.java:65)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:567)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:560)
... 4 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: decrypt_error
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:62)
... 18 common frames omitted
2017-07-06 13:50:05,344 WARN [p.j.core] Database status check timed out after 4 seconds.
2017-07-06 13:50:05,344 WARN [p.j.core] Status check of db failed with error:
java.lang.InterruptedException: null
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
at java.util.concurrent.FutureTask.get(FutureTask.java:204)
at clojure.core$deref_future.invokeStatic(core.clj:2210)
at clojure.core$future_call$reify__6962.deref(core.clj:6688)
at clojure.core$deref.invokeStatic(core.clj:2232)
at clojure.core$deref.invoke(core.clj:2214)
at puppetlabs.jdbc_util.pool$wrap_with_delayed_init$reify__35232.getConnection(pool.clj:159)
at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:307)
at clojure.java.jdbc$get_connection.invoke(jdbc.clj:197)
at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:878)
at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:847)
at clojure.java.jdbc$query.invokeStatic(jdbc.clj:915)
at clojure.java.jdbc$query.invoke(jdbc.clj:884)
at clojure.java.jdbc$query.invokeStatic(jdbc.clj:898)
at clojure.java.jdbc$query.invoke(jdbc.clj:884)
at puppetlabs.jdbc_util.core$db_up_QMARK_$fn__31952.invoke(core.clj:57)
at clojure.core$binding_conveyor_fn$fn__4676.invoke(core.clj:1938)
at clojure.lang.AFn.call(AFn.java:18)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
2017-07-06 13:50:10,107 WARN [p.j.core] Database status check timed out after 4 seconds.
2017-07-06 13:50:10,107 WARN [p.j.core] Status check of db failed with error:
Version and installation information
PE version: 2017.2 and later
OS: Any
Installation type:
2019.2 and later: Large
2019.1: Monolithic with compilers
2017.2 to 2019.0: Monolithic with compilers, split with compilers
Solution
Complete these steps on the master to fix the issue.
-
Stop pe-orchestration-services:
puppet resource service pe-orchestration-services ensure=stopped
-
Stop pxp-agent:
puppet resource service pxp-agent ensure=stopped
-
Delete the pe-orchestration-services SSL directory. (Console services use this directory.) :
rm /etc/puppetlabs/orchestration-services/ssl/*
-
To regenerate the directory, run Puppet:
puppet agent -t --verbose
-
Start pe-orchestration-services:
puppet resource service pe-orchestration-services ensure=running
-
Start pxp-agent:
puppet resource service pxp-agent ensure=running
Comments
0 comments
Please sign in to leave a comment.