Unblock Kubernetes when creating new pods in Continuous Delivery for Puppet Enterprise

If maintenance activities such as system patching cause the dbus.service to restart on a PAM node running Kubernetes 1.21, Kubernetes gets blocked when creating new pods in Continuous Delivery for Puppet Enterprise.

Error messages and logs

This error message appears in Continuous Delivery for PE as a pod stuck in ContainerCreating with kubectl describe pod output showing a dbus: connection closed by user:

# kubectl describe pods ingress-certs-valid 

Events: 

  Type     Reason                    Age                      From     Message 

  ----     ------                    ----                     ----     ------- 

Warning  FailedCreatePodContainer  3h29m (x73263 over 11d)  kubelet  unable to ensure pod container exists: failed to create container for [kubepods besteffort pod9fddb1ac-87f6-40ad-ad2f-8b4b89d0253b] : dbus: connection closed by user 

Version and installation information

CD for PE version: 4.x
Installation type: Kubernetes 1.21

Solution

To unblock Kubernetes: run systemctl restart kubelet.service on the PAM node.

If a pod is stuck in a ContainerCreating state, run kubectl delete <POD NAME>.