Containerized OpenStack Backup Considerations¶

Backup of the containerized StarlingX OpenStack application can be done in isolation or as part of the StarlingX backup procedures. See Back up OpenStack for details.

OpenStack Storage Data Backup Considerations¶

OpenStack storage data such as Glance images, Nova ephemeral disks, Cinder volumes or volume backups are automatically backed up and restored only when:

stx-openstack uses only StarlingX rook-ceph for OpenStack storage data;
stx-openstack uses StarlingX rook-ceph and/or PVCs for OpenStack storage data and the corresponding PVC storage providers support Kubernetes VolumeSnapshot operations;

In both cases, the operator must perform both the platform and the application backup and restore. The platform backup saves the Ceph pools and restores them along with the StarlingX restore.

For details on performing a StarlingX back-up, see Back Up System Data Overview.

Note

If stx-openstack uses PVCs for OpenStack storage data, automatic backup depends on whether the storage provider supports PVC VolumeSnapshot operations.
- For example, PVCs backed by FlashArray NFS storage use a StorageClass that does not support Kubernetes VolumeSnapshot operations.
- Note that using PVCs backed by an external storage for OpenStack storage data is a configuration option that is not recommended precisely to avoid issues related to PVC resizing, backup, and restore. See Supported NetApp Storage Combinations for details.

For any other external storage deployment setup (NetApp, Dell PowerStore, Everpure, etc), OpenStack storage data is not automatically backed up. It must be backed up separately, using any type of appropriate external backup tool, and later restored using the StarlingX OpenStack restore_cinder_glance_data="true" option along with an appropriate external restore tool.

See Restore application system data, cinder volumes and glance images for details.

Note

The preservation of ephemeral (image-booted) VM disks during a backup and restore depends on the chosen storage provider and on the scope of the backup:

Nova ephemeral disks on StarlingX rook-ceph: Ephemeral VM disks are preserved only when both the StarlingX and stx-openstack (application) are backed up and restored together. The disks are included in the platform’s Ceph backup. Backing up and restoring stx-openstack alone does not preserve ephemeral disks.
Nova ephemeral disks on external storage: Ephemeral VM disks are not included in the StarlingX OpenStack backup and restore procedure. To preserve these disks see the section above on backup and restore of OpenStack storage data on external storage using the StarlingX OpenStack restore_cinder_glance_data="true" option along with appropriate external tooling.

Backup Glance / Cinder Backend in HTTP and HTTPS Enabled Environment¶

When StarlingX OpenStack is deployed with TLS enabled (recommended for production environments), a configuration-escaping issue may cause PostgreSQL fail while processing the SQL file during execution of the restore-openstack playbook. As a result, application configuration overrides are not fully restored, and the StarlingX OpenStack restore operation fails during the application reapply phase.

Procedural Changes: Execute the restore-openstack playbook with TLS disabled (HTTP mode) to restore StarlingX OpenStack, then reapply the application with the required configuration overrides to enable TLS after restore completes.

Backup environment.

$ ansible-playbook /usr/share/ansible/stx-ansible/playbooks/backup.yml \
-e "ansible_become_pass=<password> \
admin_password=<password> \
openstack_app_name=wr-openstack \
skip_os_dbs=
['Database','information_schema','performance_schema','mysql','horizon','panko','gnocchi','sys']"

Pre-Restore Cleanup.

Remove VMs. All VMs must be deleted before restore.

$ source /var/opt/openstack/admin-openrc
$ openstack server list --all
$ openstack server delete <vm_uuid>

Remove and re-upload the application.

Warning

Do not proceed unless all VMs are deleted before restore.

$ system application-remove wr-openstack

Wait until status is uploaded.

$ source /etc/platform/openrc
$ watch system application-show wr-openstack
$ system application-delete wr-openstack
$ system application-upload wr-openstack.tgz

Continue waiting until the status is uploaded and run the following command.
```
$ watch system application-show wr-openstack
```

Apply the --reuse-values.

$ sudo find / -path "/restore-openstack" -type f -exec grep -l "system helm-override-update" {} \; 2>/dev/null | while read f; do
tmp=$(mktemp)
sudo sed -e '/--reuse-values/!s/system helm-override-update/system helm-override-update --reuse-values/g' -e 's/show_multiple_locations=True/show_multiple_locations=False/g' "$f" > "$tmp"
chmod 644 "$tmp"
sudo mount --bind "$tmp" "$f"
done

Verify:

grep “helm-override-update” /usr/share/ansible/stx-ansible/playbooks/roles/restore-openstack/restore/tasks/main.yml

Run the following only if the environment is HTTPS.

$ source /etc/platform/openrc
$ system service-parameter-list | grep endpoint
$ system service-parameter-delete <endpoint-id>

Restore (Runs in HTTP Mode).

$ ansible-playbook /usr/share/ansible/stx-ansible/playbooks/restore_openstack.yml \
-e "initial_backup_dir=/opt/backups \
ansible_become_pass=<password> \
admin_password=<password> \
backup_filename=<wr-openstack-backup-tarball>.tgz \
openstack_app_name=wr-openstack"

Note

If running in an HTTP environment, proceed to step “Reattach VMs” below (If needed).

Post-Restore verification.

$ source /var/opt/openstack/admin-openrc
$ openstack endpoint list

Certificate Location. Certificates generated before backup are reused.
```
/home/sysadmin/openstack-certs/
```

Reconfigure HTTPS (Post-Restore). Set Endpoint Domain.

$ system service-parameter-add openstack helm endpoint_domain=<your-lab>.wrs.com

Create clients override.

cat <<EOF > clients_override.yaml
serviceEndpointPattern: "{service_name}-{endpoint_domain}"
EOF

Apply Helm overrides.

$ source /etc/platform/openrc
$ system helm-override-update wr-openstack clients openstack --reuse-values --set openstackCertificateFile=/home/sysadmin/openstack-certs/openstack-helm.pem
$ system helm-override-update wr-openstack clients openstack --reuse-values --set openstackCertificateKeyFile=/home/sysadmin/openstack-certs/openstack-helm.pem
$ system helm-override-update wr-openstack clients openstack --reuse-values --set openstackCertificateCAFile=/home/sysadmin/openstack-certs/openstack-helm-ca.crt
$ system helm-override-update wr-openstack clients openstack --reuse-values --values clients_override.yaml

Verify if the overrides are applied.

$ system helm-override-show wr-openstack clients openstack

Re-apply the application.

$ system application-apply wr-openstack

Verify HTTPS endpoints.

$ source /var/opt/openstack/admin-openrc
$ openstack endpoint list

Endpoints should now show HTTPS.

Reattach VMs (If needed).

Note

Skip if VMs are already ACTIVE

$ source /var/opt/openstack/admin-openrc
$ nova reset-state --active <uuid>
$ openstack server stop <uuid>
$ openstack server shelve <uuid>
$ openstack server unshelve <uuid>

Results

Clear ERROR state
Stop reboot loops
Reattach volumes

Note

Always source the correct environment before running commands. If you are using a new shell, source one of the following:

/etc/platform/openrc
/var/opt/openstack/admin-openrc or
/home/sysadmin/br-snapshot-info.txt

For example:

$ source /var/opt/openstack/admin-openrc

Note

If the environment is HTTPS, the restore process must run in HTTP mode first, and then HTTPS is re-applied afterward.

Do not proceed if VMs still exist
Ensure application state transitions complete before moving to next steps