Containerized OpenStack Backup Considerations¶

Backup of the containerized OpenStack application is performed as part of the StarlingX backup procedures.

Recommendations¶

After executing the backup and restore procedure, including the latest procedural changes, previously backed up OpenStack VMs that were booted from NFS, iSCSI, or FC volumes may enter an ERROR state. In addition, any attempt to create new volumes after the restore fails, with newly created volumes transitioning to an ERROR state.

Note

Replace all instances of wr-openstack with stx-openstack.

Backup Glance / Cinder Backend in HTTP and HTTPS Enabled Environment¶

When StarlingX OpenStack is deployed with TLS enabled (recommended for production environments), a configuration-escaping issue may cause PostgreSQL fail while processing the SQL file during execution of the restore-openstack playbook. As a result, application configuration overrides are not fully restored, and the StarlingX OpenStack restore operation fails during the application reapply phase.

Procedural Changes: Execute the restore-openstack playbook with TLS disabled (HTTP mode) to restore StarlingX OpenStack, then reapply the application with the required configuration overrides to enable TLS after restore completes.

Backup environment.

$ ansible-playbook /usr/share/ansible/stx-ansible/playbooks/backup.yml \
-e "ansible_become_pass=<password> \
admin_password=<password> \
openstack_app_name=wr-openstack \
skip_os_dbs=
['Database','information_schema','performance_schema','mysql','horizon','panko','gnocchi','sys']"

Pre-Restore Cleanup.

Remove VMs. All VMs must be deleted before restore.

$ source /var/opt/openstack/admin-openrc
$ openstack server list --all
$ openstack server delete <vm_uuid>

Remove and re-upload the application.

Warning

Do not proceed unless all VMs are deleted before restore.

$ system application-remove wr-openstack

Wait until status is uploaded.

$ source /etc/platform/openrc
$ watch system application-show wr-openstack
$ system application-delete wr-openstack
$ system application-upload wr-openstack.tgz

Continue waiting until the status is uploaded and run the following command.
```
$ watch system application-show wr-openstack
```

Apply the --reuse-values.

$ sudo find / -path "/restore-openstack" -type f -exec grep -l "system helm-override-update" {} \; 2>/dev/null | while read f; do
tmp=$(mktemp)
sudo sed -e '/--reuse-values/!s/system helm-override-update/system helm-override-update --reuse-values/g' -e 's/show_multiple_locations=True/show_multiple_locations=False/g' "$f" > "$tmp"
chmod 644 "$tmp"
sudo mount --bind "$tmp" "$f"
done

Verify:

grep “helm-override-update” /usr/share/ansible/stx-ansible/playbooks/roles/restore-openstack/restore/tasks/main.yml

Run the following only if the environment is HTTPS.

$ source /etc/platform/openrc
$ system service-parameter-list | grep endpoint
$ system service-parameter-delete <endpoint-id>

Restore (Runs in HTTP Mode).

$ ansible-playbook /usr/share/ansible/stx-ansible/playbooks/restore_openstack.yml \
-e "initial_backup_dir=/opt/backups \
ansible_become_pass=<password> \
admin_password=<password> \
backup_filename=<wr-openstack-backup-tarball>.tgz \
openstack_app_name=wr-openstack"

Note

If running in an HTTP environment, proceed to step “Reattach VMs” below (If needed).

Post-Restore verification.

$ source /var/opt/openstack/admin-openrc
$ openstack endpoint list

Certificate Location. Certificates generated before backup are reused.
```
/home/sysadmin/openstack-certs/
```

Reconfigure HTTPS (Post-Restore). Set Endpoint Domain.

$ system service-parameter-add openstack helm endpoint_domain=<your-lab>.wrs.com

Create clients override.

cat <<EOF > clients_override.yaml
serviceEndpointPattern: "{service_name}-{endpoint_domain}"
EOF

Apply Helm overrides.

$ source /etc/platform/openrc
$ system helm-override-update wr-openstack clients openstack --reuse-values --set openstackCertificateFile=/home/sysadmin/openstack-certs/openstack-helm.pem
$ system helm-override-update wr-openstack clients openstack --reuse-values --set openstackCertificateKeyFile=/home/sysadmin/openstack-certs/openstack-helm.pem
$ system helm-override-update wr-openstack clients openstack --reuse-values --set openstackCertificateCAFile=/home/sysadmin/openstack-certs/openstack-helm-ca.crt
$ system helm-override-update wr-openstack clients openstack --reuse-values --values clients_override.yaml

Verify if the overrides are applied.

$ system helm-override-show wr-openstack clients openstack

Re-apply the application.

$ system application-apply wr-openstack

Verify HTTPS endpoints.

$ source /var/opt/openstack/admin-openrc
$ openstack endpoint list

Endpoints should now show HTTPS.

Reattach VMs (If needed).

Note

Skip if VMs are already ACTIVE

$ source /var/opt/openstack/admin-openrc
$ nova reset-state --active <uuid>
$ openstack server stop <uuid>
$ openstack server shelve <uuid>
$ openstack server unshelve <uuid>

Results

Clear ERROR state
Stop reboot loops
Reattach volumes

Note

Always source the correct environment before running commands. If you are using a new shell, source one of the following:

/etc/platform/openrc
/var/opt/openstack/admin-openrc or
/home/sysadmin/br-snapshot-info.txt

For example:

$ source /var/opt/openstack/admin-openrc

Note

If the environment is HTTPS, the restore process must run in HTTP mode first, and then HTTPS is re-applied afterward.

Do not proceed if VMs still exist
Ensure application state transitions complete before moving to next steps