After upgrading Continuous Delivery for Puppet Enterprise you get an “Insufficient cpu” error

After upgrading Continuous Delivery for Puppet Enterprise (PE), you try to log in to Puppet Application Manager (PAM), but it doesn’t load. In the command line you get an Insufficient cpu error from Kubernetes.

Error messages

Warning FailedScheduling 99m (x1 over 100m) default-scheduler 0/1 nodes are available: 1 Insufficient cpu.

During an upgrade, Kubernetes starts up another application instance on your control plane node to migrate data from one copy of the application to the other. You get this message when there are not enough CPU resources to schedule both application instances.

Version and installation information

Product: Continuous Delivery for Puppet Enterprise

Version: 4.x

Solution

To fix the issue, you can either add more CPUs or remove optional PAM components.

Add CPUs

Adding more CPUs is a long-term solution that helps prevent future issues. If the time it takes to add resources causes too much downtime, roll back to a pre-upgrade system snapshot before adding more cores (CPUs).

Remove optional PAM components

Removing an optional PAM component to free up resources is a quick solution but might not resolve the problem the next time you upgrade.

The best components to remove in this case are Prometheus and Grafana, which are used for metrics. Since you are most likely trying to resolve this issue during an upgrade, keep Velero and Restic, the optional components used for snapshots.

To remove Prometheus and Grafana, run kubectl delete ns/monitoring

If removing Prometheus and Grafana doesn’t work, you can add more CPUs to resolve the issue.