Troubleshoot On-Site Enroll and Restore Issues¶

Seed ISO Not Detected ¶

If the system does not detect the seed ISO after you insert it:

Verify that factory installation completed successfully: The file /var/lib/factory-install/complete must exist. The cloud-init-seed.service service checks for this file and does not start if it is missing.
Verify the ISO volume label is CIDATA (case-insensitive). The run-cloud-init-from-seed.sh script uses blkid -L cidata to locate the seed device.
Verify the udev rule exists:
```
/etc/udev/rules.d/99-seediso.rules
```
If the rule is missing or was recently added, reload the rules:
```
$ udevadm control --reload-rules
```
Verify that the ISO contains valid user-data and meta-data files.

Clock Synchronization Issues ¶

Before authenticating to Keystone, the enrollment and restore scripts synchronize the system time using the System Controller HTTP Date header.

If the clock difference exceeds 20 seconds, the system clock and hardware clock are updated automatically.

If time synchronization fails, verify network connectivity to the System Controller’s Keystone port (5000).

Unable to Connect to the System Controller ¶

The enrollment and restore scripts retry OAM connectivity for up to 8 minutes with 30-second intervals. If this times out:
- Verify the network-config file in the seed ISO has the correct OAM IP, gateway, and VLAN settings.
- Verify the IPsec tunnel is established (if vCSR):
```
$ ip xfrm state
```
- Verify that the SSL CA certificate in config/ matches the System Controller’s OAM certificate.
Keystone authentication retries for up to 5 minutes. SSL-related failures are retried automatically (the certificate may not yet be trusted). Other HTTP errors are treated as fatal and cause the operation to fail.
If the script reports:
```
admin_password not found in bootstrap-values
```
Verify that bootstrap-values.yaml includes the admin_password field (the System Controller’s admin password).

Subcloud Already Exists ¶

If enrollment fails because the subcloud already exists, the enrollment script checks the current subcloud state and takes one of the following actions:

Subcloud state	Action
`create-complete`, `enroll-failed`, `pre-enroll-failed`, `pre-init-enroll-failed`, `init-enroll-failed`, `factory-restore-complete`	The script automatically deletes and re-enrolls the subcloud.
Any other state (for example, `deploy-complete`, `managed`)	The script exits with an error. Manually delete the subcloud on the System Controller before retrying.

Restore Request Rejected ¶

The System Controller rejects restore requests for managed subclouds.

Before starting an on-site restore, place the subcloud in the unmanaged state:

~(keystone_admin)]$ dcmanager subcloud unmanage <subcloud-name>

Backup Not Found ¶

For a local restore (local_only: true):

Verify that a backup exists in:

/opt/platform-backup/backups/<sw-version>/

Verify that the directory contains a *_platform_backup_*.tgz file.

For remote restore (local_only: false):

Verify that the System Controller successfully transfers the backup to
```
/opt/platform-backup/auto-restore/
```
If transfer fails, verify available disk space and network connectivity.

On-Site Restore Only Supported on Simplex ¶

On-site restore is not supported for duplex or multi-node systems.

Verify that the following setting is present in /etc/platform/platform.conf:

system_mode=simplex

Factory Restore Fails Because Prestaged Data Is Missing ¶

Verify that the following files and directories exist:

/opt/platform-backup/factory/<sw-version>/

Required artifacts:

ostree_repo/ directory
factory_backup*.tgz file
miniboot.cfg file

If any are missing, the script exits with an error identifying which artifact is absent. These files are created during the original factory install. If the platform-backup partition was wiped, factory restore is not available.

Factory Restore Boot Entry Problems ¶

Before rebooting, the factory restore process creates or updates a UEFI boot entry labeled Local Factory Restore and sets BootNext to it before rebooting.

If efibootmgr reports errors, verify that the EFI partition is mounted at /boot/efi and that boot.env exists at /boot/efi/EFI/BOOT/boot.env.