Restore an AIO-SX Subcloud On-Site

Use this procedure to restore a factory-installed AIO-SX subcloud locally. The following on-site restore scenarios are supported:

• Restore from a local backup stored on the subcloud.

• Restore from a backup stored on the System Controller.

• Restore the system to its factory-installed state.

Restore from a Local Backup

Use this scenario when the subcloud has failed but can be booted and a recent backup is available on the subcloud.

Prerequisites

• A backup archive is available in /opt/platform-backup/.

• The subcloud is unmanaged on the System Controller.

• The subcloud is in its factory default state.

• The factory system must be configured with Rook-ceph or LVM CSI (baremetal ceph is not supported).

• The backup and the factory system must use the same persistent volume storage solution. You cannot restore a Rook-ceph backup onto an LVM CSI factory system or vice versa.

Procedure

1. Prepare the seed ISO as described in Prepare the Seed ISO for On-Site Enrollment, with the following changes:

   • Set MODE in user-data to onsite-restore.

   • Set a new unique value for instance-id in meta-data.

   • Add a config/restore-values.yaml file containing:

     local_only: true
     restore_timeout: 5400
     

2. Insert the seed ISO using one of the methods described in Insert the Seed ISO for On-Site Enrollment.

Results

After the system detects the seed ISO:

1. Cloud-init configures the network and vCSR.

2. The restore script sends a restore request to the System Controller. StarlingX does not transfer backup data for local restore.

3. The restore process runs locally using the backup stored on the subcloud.

4. After the restore completes, the vCSR re-establishes the IPsec tunnel and the System Controller detects the subcloud is back online.

To monitor restore progress, see Monitor Subcloud Restore Progress.

Restore from Remote Backup

Use this scenario when the subcloud hardware has been replaced and factory-installed to its default state, and a recent backup of the subcloud exists on the System Controller.

Prerequisites

• A backup archive exists on the System Controller.

• The /opt/platform-backup partition on the subcloud has sufficient disk space for the backup archive transfer.

• The subcloud is unmanaged on the System Controller.

• The subcloud is in its factory default state.

• The factory system must be configured with Rook-ceph or LVM CSI (baremetal ceph is not supported).

• The backup and the factory system must use the same persistent volume storage solution. You cannot restore a Rook-ceph backup onto an LVM CSI factory system or vice versa.

Procedure

1. Prepare the seed ISO as described in Prepare the Seed ISO for On-Site Enrollment, and set local_only: false in restore-values.yaml.

2. Insert the seed ISO using one of the methods described in Insert the Seed ISO for On-Site Enrollment.

Results

After the system detects the seed ISO:

1. Cloud-init configures the network and vCSR.

2. The restore script sends a restore request to the System Controller.

3. The System Controller transfers the backup archive to the subcloud.

4. The restore playbook runs locally using the transferred backup.

To monitor restore progress, see Monitor Subcloud Restore Progress.

Restore from Factory Backup

Use this scenario to restore the system to its factory-installed state. The restore runs fully autonomously and requires no communication with the System Controller. Start a factory restore using a seed ISO or the GRUB boot menu.

Prerequisites

• Factory backup data is available in /opt/platform-backup/factory/ (created automatically during the original factory install).

• Prestaged software is available on the protected partition.

Restore Using the Seed ISO

Use when the system is operational and cloud-init is running. This follows a similar process to the local and remote backup restore procedures: a seed ISO is inserted, which triggers a reinstall from prestaged data followed by the factory restore.

Procedure

1. Prepare the seed ISO as described in Prepare the Seed ISO for On-Site Enrollment, with the following changes:

   • Set MODE in user-data to onsite-factory-restore.

   • Set a new unique value for instance-id in meta-data.

2. Insert the seed ISO using the same methods described in Insert the Seed ISO for On-Site Enrollment.

Results

After the system detects the seed ISO and processes the user-data:

1. The 10-trigger-reinstall script initiates a reinstall from prestaged software on the protected partition.

2. The system restores from the factory backup.

3. The system returns to its factory-installed state.

Restore Using the GRUB Boot Menu

Use when the system is completely unresponsive or cannot process a seed ISO (for example, cloud-init is not functional). This requires console access to select the restore option from the GRUB boot menu.

Procedure

1. Access the server console through the BMC by using either a serial console or a remote KVM console.

2. Reboot the server and select the Local Factory Restore from the GRUB boot menu.

   Note

   The Factory Restore GRUB entry is created during the original factory install. It is stored at /boot/efi/EFI/install/grub.cfg and is not managed by OSTree, so it persists across software updates.

Results

The system automatically reinstalls from prestaged software on the protected partition, restores from the factory backup, and returns to its factory-installed state.

After the factory restore completes, choose one of the following next steps:

• To re-enroll the subcloud, see Enroll an AIO-SX Subcloud On-Site.

• To restore from a local backup, see Restore from a Local Backup.

• To restore from a remote backup, see Restore from Remote Backup.

To monitor factory restore progress, see Monitor Subcloud Restore Progress.

Monitor Subcloud Restore Progress

• On-site Restore:

  Monitor the following log files on the subcloud:

  /var/log/cloud-init-output.log (for the trigger phase)
  /var/log/auto-restore.log (for the restore playbook)
  

• Factory Restore:

  Monitor restore progress by using either the BMC serial console or through IPMI system event logs:

  $ ipmitool sel list
  

  After the system becomes accessible, monitor detailed restore progress in:

  /var/log/auto-restore.log
  

Verify and Manage the Subcloud After Restore

After the subcloud restore completes, perform the following steps:

Procedure

1. Verify the subcloud is operational:

   $ source /etc/platform/openrc
   ~(keystone_admin)]$ system host-list
   ~(keystone_admin)]$ fm alarm-list
   

2. From the System Controller, manage the subcloud:

   ~(keystone_admin)]$ dcmanager subcloud manage <subcloud-name>
   

3. If you performed a factory restore, delete the subcloud from the System Controller and re-enroll it:

   Warning

   Deleting the subcloud removes all audit and configuration data from the System Controller. Ensure you have the required bootstrap values and deployment configuration before proceeding.

   ~(keystone_admin)]$ dcmanager subcloud delete <subcloud-name>
   ~(keystone_admin)]$ dcmanager subcloud add --enroll --bootstrap-address <address> \
       --bootstrap-values <file> --deployment-config <file> \
       --install-values <file> --sysadmin-password <password> \
       --bmc-password <password>
   

   For more information, see Enroll a Factory Installed Non Distributed Standalone System as a Subcloud.

Related Information

• Troubleshoot On-Site Enroll and Restore Issues