Restore OpenStack from a Backup¶

You can restore StarlingX OpenStack from a backup.

Prerequisites

You must have a backup of your StarlingX OpenStack installation as described in Back up OpenStack.
You must have an operational StarlingX deployment.

Procedure

Delete the old OpenStack application and upload it again.

Note

Images and volumes will remain in Ceph.

Note

To remove the OpenStack application, first you need to remove all OpenStack resources (including VMs) and then remove the OpenStack.

If the application fails to re-apply due to VMs orphaned from the previous remove, check Uninstall OpenStack for more details.
```
~(keystone_admin)$ system application-remove stx-openstack
~(keystone_admin)$ system application-delete stx-openstack
~(keystone_admin)$ system application-upload stx-openstack.tgz
```
Restore StarlingX OpenStack.

You can choose either of the following options:
- Restore a Distributed Cloud subcloud platform and StarlingX OpenStack. This option is applicable only to Distributed Cloud deployments. It restores the subcloud platform and prepares StarlingX OpenStack data by using DCManager from the Centralized Cloud. After the platform restore completes, you must finish the StarlingX OpenStack application restoration on the subcloud using the standard StarlingX OpenStack restore procedure. For more information, see Restore Distributed Cloud Subcloud.
- Restore only StarlingX OpenStack system data. This option will not restore the external storage data (that is, it will not run commands like rbd import). This procedure will preserve any existing storage data at restore-time. For more information, see Restore only application system data.
- Restore StarlingX OpenStack system data and OpenStack storage data. This step is required when the OpenStack storage data backup was done externally to the StarlingX OpenStack backup. For more information, see Restore application system data, cinder volumes and glance images.
Note

Ephemeral (image-booted) VM disks are not always preserved during backup and restore. Before restoring, review the Ephemeral disk preservation note.

Restore Distributed Cloud Subcloud ¶

This section describes how to restore a Distributed Cloud subcloud using backup files generated by the DCManager backup procedure.

Restore a DC Subcloud Platform and StarlingX OpenStack ¶

Use this procedure to restore the subcloud platform and partially restore the StarlingX OpenStack application in a Distributed Cloud deployment. Initiate the restore from the Centralized Cloud using DCManager.

Use the following procedure to restore the subcloud platform and prepares StarlingX OpenStack data. After the platform restore completes, you must finish restoring the StarlingX OpenStack application directly on the subcloud by using the standard StarlingX OpenStack restore steps.

Procedure

Create a restore override file required for StarlingX OpenStack:

cat << EOF > <subcloud>-restore.yaml
restore_extra_items:
  - "etc/dcdbsync/dcdbsync_openstack.conf"
EOF

Unmanage the subcloud before restoring it.

~(keystone_admin)]$ dcmanager subcloud unmanage [subcloud-name]

Run the DCManager restore command using the local backup files:

dcmanager subcloud-backup restore \
  --local-only \
  --subcloud <subcloud> \
  --sysadmin-password <sysadmin-password> \
  --restore-values <subcloud>-restore.yaml

This command restores the subcloud platform and partially restores the StarlingX OpenStack application.

After the restore completes, access the subcloud and complete the StarlingX OpenStack application restoration.

Remove, delete, and upload the StarlingX OpenStack application tarball:

~(keystone_admin)]$ system application-remove stx-openstack
~(keystone_admin)]$ system application-delete stx-openstack
~(keystone_admin)]$ system application-upload stx-openstack

Run the restore playbook to completely restore the StarlingX OpenStack application:

ansible-playbook /usr/share/ansible/stx-ansible/playbooks/restore_openstack.yml \
  -e 'initial_backup_dir=/opt/platform-backup/backups/<release>/
  ansible_become_pass=<password>
  admin_password=<password>
  backup_filename=<backup_file_name.tgz>
  openstack_app_name="stx-openstack"'

Restore Only StarlingX OpenStack on a DC Subcloud ¶

Use this procedure to restore only the StarlingX OpenStack application on a DC subcloud. This restore method is identical to the procedure used for isolated (non-Distributed Cloud) systems.

Procedure

Remove, delete, and upload the StarlingX OpenStack application tarball:

~(keystone_admin)]$ system application-remove stx-openstack
~(keystone_admin)]$ system application-delete stx-openstack
~(keystone_admin)]$ system application-upload stx-openstack

Run the restore playbook to complete the StarlingX OpenStack restoration:

ansible-playbook /usr/share/ansible/stx-ansible/playbooks/restore_openstack.yml \
  -e 'initial_backup_dir=/opt/platform-backup/backups/<release>/
  ansible_become_pass=<password>
  admin_password=<password>
  backup_filename=<backup_file_name.tgz>
  openstack_app_name="stx-openstack"'

Restore Only Application System Data ¶

Run the following command:

~(keystone_admin)$ ansible-playbook /usr/share/ansible/stx-ansible/playbooks/ restore_openstack.yml -e "@localhost-restore.yaml"

cat << EOF > localhost-restore.yaml
---
ansible_become_pass: "<admin_password>"
admin_password: "<admin_password>"
initial_backup_dir: "<location_of_backup_filename>"
backup_filename: "<openstack_tgz_backup_filename>"

Note

VMs with volumes on iSCSI or FC storage backends (for example: NetApp) may enter an error state after the restore completes because storage sessions can expire during the process. VMs using NFS or Ceph storage backends are not affected.

To enable automatic recovery of affected VMs, set both Helm overrides before running the restore playbook. Apply both Helm overrides after uploading the application and before running the restore playbook.

~(keystone_admin)]$ system helm-override-update stx-openstack nova openstack --set conf.nova.DEFAULT.resume_guests_state_on_host_boot=false --set conf.vm_recovery.enabled=true

Disabling resume_guests_state_on_host_boot keeps the guest resume operations from running in the nova-compute boot critical path, allowing the application apply to complete within the Helm timeout. The vm_recovery setting then automatically recovers the affected VMs after the application apply completes.

Monitor VM Recovery ¶

Once the overrides are in place, recovery occurs automatically. Affected VMs return to the ACTIVE state shortly after the final apply completes, typically within 60 seconds. To monitor the recovery progress, check the sysinv log on the active controller. The recovery routine logs detailed LOG.info and LOG.error messages at each step:

~(keystone_admin)]$ tail -f /var/log/sysinv.log | grep -i "VM recovery"

Recovery is complete when the log indicates that all servers have been recovered or that no servers are in an error state. You can also verify successful recovery by listing the servers and confirming that they are in the ACTIVE state:

~(keystone_admin)]$ openstack server list --all-projects

Once recovery has been verified, you can disable the feature if required:

~(keystone_admin)]$ system helm-override-update stx-openstack nova openstack --set conf.vm_recovery.enabled=false

Restore Application System Data and OpenStack Storage Data ¶

Run the following command:

~(keystone_admin)]$ ansible-playbook /usr/share/ansible/stx-ansible/playbooks/ restore_openstack.yml -e "@localhost-restore.yaml"

cat << EOF > localhost-restore.yaml
---
ansible_become_pass: "<admin_password>"
admin_password: "<admin_password>"
initial_backup_dir: "<location_of_backup_filename>"
backup_filename: "<openstack_tgz_backup_filename>"
restore_cinder_glance_data="true"
...
EOF

When this step has completed, the Cinder, Glance and MariaDB services will be up, and Mariadb data restored.

Restore External Storage data.

If StarlingX OpenStack was using an external storage solution for OpenStack storage data when backup was taken and the storage data was saved separetly and now is the time to restore it. Once restore is done, proceed to next steps. See OpenStack Storage Data Backup Considerations for details.

Run the playbook again with the restore_openstack_continue flag set to true to bring up the remaining OpenStack services.

~(keystone_admin)]$ ansible-playbook /usr/share/ansible/stx-ansible/playbooks/ restore_openstack.yml -e "@localhost-restore.yaml"

cat << EOF > localhost-restore.yaml
---
ansible_become_pass: "<admin_password>"
admin_password: "<admin_password>"
initial_backup_dir: "<location_of_backup_filename>"
backup_filename: "<openstack_tgz_backup_filename>"
restore_openstack_continue="true"
...
EOF