Fix a mismatched kurl-registry-ip for an incomplete offline Puppet Application Manager cluster migration in Puppet Comply

When the new cluster is provisioned during Puppet Application Manager migration, its image registry service must be configured with the IP address used by the old cluster that generated the snapshot using the kurl-registry-ip installation option. If this option is not set during installation, restoration is completed, but application pods are in a ErrImagePull, ImagePullBackOff, or Init:ImagePullBackOff status.

To fix the issue, complete the steps in this article.

Error messages and logs

Output similar to:

# kubectl get pods
NAME                                         READY   STATUS                  RESTARTS   AGE
cd4pe-5d94d68574-79bn2                       0/1     ImagePullBackOff        0          8m34s
cd4pe-issuer-54d7988df5-lkgtt                0/1     ImagePullBackOff        0          8m34s
cd4pe-object-store-backup-5795c7f977-6974m   0/1     Init:ImagePullBackOff   0          8m34s
cd4pe-registry-proxy-6d5cd84844-q64p2        0/1     Init:ImagePullBackOff   0          8m34s
kotsadm-849dc5db4c-9q6zx                     1/1     Running                 0          11m
kotsadm-fs-minio-7768949976-f7wxk            1/1     Running                 2          8d
kotsadm-postgres-0                           1/1     Running                 0          11m
kurl-proxy-kotsadm-7f56448d57-6stcv          1/1     Running                 0          11m
minio-0                                      0/1     ImagePullBackOff        0          8m34s
minio-1                                      0/1     ImagePullBackOff        0          8m34s
minio-2                                      0/1     ImagePullBackOff        0          8m34s
minio-3                                      0/1     ImagePullBackOff        0          8m34s
postgres-0                                   0/1     ImagePullBackOff        0          8m34s
query-649bd8758f-jgf2g                       0/1     ImagePullBackOff        0          8m33s
query-postgres-0                             0/1     ImagePullBackOff        0          8m33s
ui-75987955b5-8w572                          0/1     ImagePullBackOff        0          8m33s

Logs for the containerd service show failures to pull images from the old registry service IP address, similar to:

# journalctl -u containerd.service --no-pager --since '5 minutes ago'

... Feb 17 00:23:20 pamnew-928bbc-0 containerd[14234]: time="2022-02-17T00:23:20.880563583Z" level=error msg="PullImage \"10.96.1.64/cd4pe/continuous-delivery-for-puppet-enterprise:4.10.5\" failed" error="failed to pull and unpack image \"10.96.1.64/cd4pe/continuous-delivery-for-puppet-enterprise:4.10.5\": failed to resolve reference \"10.96.1.64/cd4pe/continuous-delivery-for-puppet-enterprise:4.10.5\": failed to do request: Head https://10.96.1.64/v2/cd4pe/continuous-delivery-for-puppet-enterprise/manifests/4.10.5: dial tcp 10.96.1.64:443: i/o timeout"

Version and installation information

Product: Puppet Comply
Version: All supported
Installation type: Puppet Application Manager offline (air-gapped) installations

Solution

To update kurl-registry-ip and restore successfully from the snapshot, complete the following steps on the new Puppet Application Manager cluster:

1. Get the registry service IP address from the snapshot:

   kubectl get backup <BACKUP NAME> -n velero -o jsonpath="{.metadata.annotations['kots\.io/kurl-registry']}{'\n'}"

2. Delete the registry service:

   kubectl delete svc registry -n kurl

3. Rerun the Puppet Application Manager installer with the kurl-registry-ip= and force-reapply-addons options:

   cat install.sh | bash -s <INSTALLATION OPTIONS USED PREVIOUSLY> kurl-registry-ip=<REGISTRY SERVICE IP ADDRESS> force-reapply-addons

4. Restore the snapshot:

   /usr/local/bin/kubectl-kots restore --from-backup=<BACKUP NAME>