Subclouds Previous Major Release Management

You can install, restore, and manage subclouds that are running up to two major software releases behind the major software release that their System Controller is running. That is, a System Controller running the current major software release (N) can manage subclouds running the current (N), previous (N-1), or two-releases-back (N-2) major software release. This capability can be used to revert to a previous release (i.e. major release, or patched major release) when an unforeseen issue is encountered during a subcloud upgrade or deployment, or to continue to manage (e.g. update/patch, rehome, etc.) subclouds of previous major releases that have not yet been upgraded.

Note

Subclouds can be deployed or restored to the current (N), previous (N-1), or two-releases-back (N-2) major software release relative to the release that the System Controller is running.

Overview

This section describes the steps for deploying, restoring, and managing subclouds of a previous major release from a System Controller running the current major software release.

Operations Supported for Subclouds of a Previous Release

  • New subcloud deployment

  • Subcloud re-install

  • Subcloud backup removal

  • Subcloud restore from backup

  • Subcloud prestage (single subcloud)

  • Orchestrated subcloud prestage (multiple subclouds)

  • Orchestrated subcloud sw-deploy of major release or patched major release (upgrade) (multiple subclouds)

  • Orchestrated subcloud patch of major release (update/patch) (multiple subclouds)

  • Subcloud rehome

  • Deploy files upload/show

Note

After deployment, subclouds running a previous major release can be managed, unmanaged, audited, updated, upgraded, or deleted in the same way as subclouds running the current release.

Operations with Limited Support for Subclouds of a Previous Major Release

  • Subcloud Backup

    Limitation: The option to exclude the /opt/patching directory from the backup, which helps to reduce the backup size, does not apply to subclouds operating on some previous major releases. For additional details on the list of configurable system backup parameters and to identify the specific parameter(s) that do not apply to previous-release subclouds, see Run Ansible Backup Playbook Locally on the Controller.

    Warning

    Ensure that the RECONCILED state of each host is true before creating a subcloud backup.

    You cannot create a backup for a subcloud in a failed state when running an upgrade from to a newer major release.

    Use the following command to verify the RECONCILED state.

    kubectl get hosts -n deployment

  • Subcloud Error Reporting

    Limitation: In the event of a failure during subcloud install, bootstrap, deployment, backup, restore, or upgrade, you can use the dcmanager subcloud errors command to obtain detailed information about the failure.

    For example, in the case of a backup failure during a subcloud upgrade due to insufficient disk space, the subcloud generates a detailed error report such as, Not enough free space in /opt/platform-backup. It has 2010876KiB. It needs at least 3558912KiB.

  • Orchestrated subcloud patch of major release (update/patch) (multiple subclouds)

Operations Not Supported for Subclouds of a Previous Release

  • Orchestrated subcloud kubernetes upgrade

  • Orchestrated subcloud firmware update

  • Admin network configuration

Prerequisites

To support subcloud install or subcloud restore to either the current (N), previous (N-1), and two releases back (N-2) release, ensure that the following prerequisites are met:

  • The N load (pre-patched ISO) containing this capability must be imported to the System Controller using the software --os-region-name SystemController upload command. The ISO imported should be at the same patch level as the system controller. This requirement ensures that the subcloud boot image is aligned with the patch level of the load to be installed on the subcloud.

    Note

    Each time a new patch is applied on the System Controller, it is necessary to re-upload the new pre-patched ISO containing the same patch(es).

  • The only latest uploaded ISO (i.e. GA version of ISO or Patched ISO) for each major release is kept at the DC System Controller.

    To display latest uploaded ISO for a particular major release, at the DC System Controller run:

    ~(keystone_admin)]$ software list

  • The N-1 and N-2 load (pre-patched ISO) must be uploaded to the System Controller using the following command:

    ~(keystone_admin)]$ software --os-region-name SystemController upload <bootimage>.iso <bootimage>.sig

  • The central registry (or source registry configured for the subclouds) must contain platform and applications container images of the N, N-1, and N-2 releases.

  • If the System Controller’s system-local-ca uses an ECDSA key, attempts to deploy, enroll, or rehome subclouds running releases prior to r13 will be rejected.

    To allow these operations, update system-local-ca to use an RSA key (4096 bits or longer) following the procedure in Update system-local-ca or Migrate Platform Certificates to use Cert Manager.

  • Subcloud install values, bootstrap values, and deploy config files must be compatible with the release the subcloud is being deployed to.

    Warning

    Since orchestrated Kubernetes upgrades are not supported for subclouds running the previous release, these subclouds must be deployed with the highest Kubernetes version for that release. Do not set the kubernetes_version value in subcloud bootstrap-values.yaml when deploying a new N-1 or N-2 subcloud. If this value is not the highest Kubernetes version, the subcloud add request will be rejected.

    Warning

    If a subcloud was initially deployed in one release and re-deployed to a different release, ensure that persistent_size value in the subcloud install values is equal to or greater than persistent_size specified in the previous deployment. Failing to do so may result in a subcloud installation failure, as the persistent /opt/platform-backup partition cannot be reduced in size.

  • If the System Controller was installed with default settings, it will use the latest Kubernetes version for the release it’s running. To support upgrading the subcloud, ensure that the System Controller has locally stored copies (caches) of container images for all the previous versions of Kubernetes relevant to the platform release. For example, if the system controller is installed with Kubernetes 1.24, we need to ensure that all the system images related to Kubernetes 1.21, 1.22, and 1.23 are stored locally (cached) on the System Controller for use when the subcloud tries to upgrade to those versions.

    To obtain more information about the list of images, see the corresponding pre-build ISO and docker images section for each supported major release.

Warning

If the System Controller’s system-local-ca uses an ECDSA key, attempts to deploy, enroll, or rehome subclouds running releases prior to r13 will be rejected.

To allow these operations, update system-local-ca to use an RSA key (4096 bits or longer) following the procedure in Update system-local-ca or Migrate Platform Certificates to use Cert Manager.

Upload Deploy/Prestage files

Use the dcmanager subcloud deploy upload command with the --release option to upload deploy or prestage artifacts for a specific major release version.

Procedure

  1. To upload deploy files use dcmanager subcloud deploy upload command with --deploy-playbook, --deploy-chart and --deploy-overrides options. The following command uploads deploy files for a specific major release:

    ~(keystone_admin)]$ dcmanager subcloud deploy upload
--deploy-playbook <deployment-manager-yaml-file>
--deploy-chart <deployment-manager-chart-tarball>
--deploy-overrides <deployment-manager-overrides-subcloud-yaml-file>
--release <xx.xx>

    Where, <xx.xx> is the target major release version.

    Note

    Run the software --os-region-name SystemController upload <bootimage>.iso <bootimage>.sig command to automatically upload deploy files for N-1 and N-2 releases.

    It is not mandatory to upload deploy files for the N release. If the deploy files are not uploaded, the software will use the default deploy files included in the active load of the System Controller (running the N release).

  2. To upload prestage images list use dcmanager subcloud deploy upload command with --prestage-images option. The following command uploads the images list for a specific major release:

    ~(keystone_admin)]$ dcmanager subcloud deploy upload --prestage-images <images-list-file> --release <xx.xx>

    Where, <xx.xx> is the target major release version.

  3. View uploaded files for a particular release.

    ~(keystone_admin)]$ dcmanager subcloud deploy show
--release <previous_or_current_major_release_number>

Note

If the --release option is not specified, the deploy/images list files are uploaded or shown for the current major release.

Deploy a New Subcloud

Use the dcmanager subcloud add command with the --release option to deploy a subcloud with the specified major release version.

To initiate the deployment of a subcloud running a previous StarlingX release, ensure that the required deploy files for that release are uploaded, and that the subcloud configurations specific to that release are properly prepared, then run the following command:

Note

The install feed (/var/www/pages/feed/rel-<version>) for the release is synchronized to both controllers automatically when the pre-patched ISO is imported using the software --os-region-name SystemController upload command. A manual copy between controllers is no longer required.

~(keystone_admin)]$ dcmanager subcloud add
--bootstrap-address <oam_ip_address_of_subclouds_controller-0>
--bootstrap-values /home/sysadmin/subcloud1-bootstrap-values.yaml
--install-values /home/sysadmin/subcloud1-install-values.yaml
--bmc-password <bmc_password>
--deploy-config /home/sysadmin/subcloud1-deployment-config.yaml
--sysadmin-password <sysadmin_password>
--release <xx.xx>

Where, <xx.xx> is the target major release version.

Note

If the --release option is not specified, the subcloud will be deployed to the current major release of the System Controller.

Warning

If the value of software_version specified in the subcloud install-values.yaml does not align with the --release value in the subcloud add command, the operation will fail. It is recommended to exclude software_version from the subcloud’s install-values.yaml configuration to prevent potential conflicts.

Note

To deploy a patched ISO, you must have previously updated the System Controller with the current ISO for the N-1 or N-2 major release, see prerequisite.

Restore a Subcloud

Note

If a subcloud needs to be reverted to the previous major release after an upgrade failure, it is recommended to take a backup of the subcloud before the upgrade using the dcmanager subcloud-backup create command. For additional information, see Backup a Subcloud/Group of Subclouds using DCManager CLI.

Note

A subcloud cannot be restored from a backup that was taken before it was deleted and redeployed.

To restore a subcloud from a backup of the previous major release, ensure that the pre-patched ISO for the previous major release, that aligns with the backup being restored, has been uploaded at the System Controller (i.e. see line prerequisite), and use the dcmanager subcloud-backup restore command with the --release option.

~(keystone_admin)]$ dcmanager subcloud-backup restore --subcloud <subcloud> --with-install --sysadmin-password <sysadmin_password> --release <xx.xx>

Where, <xx.xx> is the StarlingX r6 major version.

For additional information about the restore, refer to “Restore a single subcloud” section in Restore a Subcloud/Group of Subclouds from Backup Data Using DCManager CLI.

Note

If the --release option is not specified, the subcloud will be installed to the current major release and restored from a backup of the current release.

Warning

The restore operation will fail if the backup is either non-existent or incompatible with the release version to which it is being restored.

Delete a Subcloud Backup

To delete a subcloud backup of a particular major release, use the dcmanager subcloud-backup delete command with the major release number specified. For more information, see Delete Subcloud Backup Data Using DCManager CLI.

  • To delete backup data from central storage, use the following command.

    ~(keystone_admin)]$ dcmanager subcloud-backup delete <xx.xx> --subcloud

    Where, <xx.xx> is the target major release version.

  • To delete backup data from local storage, use the following command

    ~(keystone_admin)]$ dcmanager subcloud-backup delete <xx.xx> --local-only --subcloud --sysadmin-password <sysadmin_password>

    Where, <xx.xx> is the target major release version.

Prestage a Subcloud

Use the dcmanager subcloud prestage command with the --release option to prestage a subcloud for the specified software release. A subcloud can be prestaged with data from the current (N), previous (N-1), or two-releases-back (N-2) release, irrespective of its current software release version.

Prestaged container images are uploaded to the local registry, and a snapshot of the local registry is captured. This format significantly enhances the speed and efficiency of restoration during subcloud upgrades, re-deployment, or restore processes.

Prestage a Subcloud with Current Release Data

Before prestaging the subcloud(s) with N release data, be sure to complete one or both of the following steps:

  1. Import the pre-patched ISO for the N release as an active load using the following command:

    ~(keystone_admin)]$ software --os-region-name SystemController upload <bootimage>.iso <bootimage>.sig

  2. Optional step: Upload the N release images list by using the following command:

    ~(keystone_admin)]$ dcmanager subcloud deploy upload --prestage-images <nn.nn_images.lst>
Table 1. Container Images Prestage Behavior for N Release

Subcloud Software Version

N Release Images List Available

N Release Images List Not Available

N release subcloud

The container images specified in the list are downloaded from the configured source(s) and pushed to the subcloud local registry. A snapshot of the local registry is then taken and saved in the prestage directory.

A snapshot of the subcloud local registry is taken and saved in the prestage directory.

Note

It is recommended to repeat subcloud prestaging for the current release after a new patch of the current release has been applied on the system controller. This requirement ensures that the subcloud boot image is aligned with the patch level of the load to be installed on the subcloud using prestaged data.

Prestage a Subcloud with N-1 or N-2 Release Data

Before prestaging the subcloud(s) with N-1 or N-2 release data, be sure to complete one or both of the following steps:

  1. Import the pre-patched ISO for the N-1 or N-2 release as an inactive load using the following command:

    ~(keystone_admin)]$ software --os-region-name SystemController upload <bootimage>.iso <bootimage>.sig

  2. Optional step: Upload the N-1 or N-2 release images list by using the following command:

    ~(keystone_admin)]$ dcmanager subcloud deploy upload --prestage-images --release <N-1-major-software-version>  <nn.nn_images.lst>
Table 2. Container Images Prestage Behavior for the N-1 or N-2 Release

Subcloud Software Version

N-1 or N-2 Release Images List Available

N-1 or N-2 Release Images List Not Available

N-1 or N-2 release subcloud

The container images specified in the list are downloaded from the configured source(s) and pushed to the subcloud local registry. A snapshot of the local registry is taken and saved in the prestage directory.

A snapshot of the subcloud local registry is taken and saved in the prestage directory.

  • To prestage a single subcloud, use dcmanager subcloud prestage command. The following commands can be used to prestage a single subcloud with data for a specific major release:

    ~(keystone_admin)]$ dcmanager subcloud prestage --release <xx.xx> --sysadmin-password <sysadmin_password> <subcloud-name-or-id>

    or

    ~(keystone_admin)]$ dcmanager subcloud prestage --for-install --sysadmin-password <sysadmin-password> --release <xx.xx> <subcloud-name-or-id>

    Where, <xx.xx> is the target major release version.

  • To prestage a group of subclouds use dcmanager prestage-strategy related commands. The following commands can be used to prestage a group of subclouds with data for a specific major release:

    ~(keystone_admin)]$ dcmanager prestage-strategy create --release <xx.xx> --sysadmin-password <sysadmin_password> --group <group_name>

~(keystone_admin)]$ dcmanager prestage-strategy apply

    or

    ~(keystone_admin)]$ dcmanager prestage-strategy create --for-install --release <xx.xx> --sysadmin-password <sysadmin_password> --group <group_name>

~(keystone_admin)]$ dcmanager prestage-strategy apply

    Where, <xx.xx> is the target major release version.

Rehome a Subcloud

N-1 and N-2 subclouds can be migrated to another System Controller running N software release. Use the dcmanager subcloud add --migrate command with the --release option to rehome a subcloud. It is not possible to migrate a subcloud running N software release to a System Controller running N-1 or N-2 software release.

Note

A backup of the subcloud should be taken prior to the rehoming procedure in the event that a subcloud restore is required to recover from a failed migration.

See: Backup a Subcloud/Group of Subclouds using DCManager CLI

Rehome StarlingX N-1 or N-2 Major Release Subcloud to StarlingX N Major Release System Controller

To migrate a StarlingX N-1 or N-2 subcloud from System Controller A to System Controller B running StarlingX N, ensure that the StarlingX N-1 or N-2 pre-patched ISO has been imported to System Controller B as an inactive load, and follow the StarlingX N subcloud rehome procedure as follows:

  1. Unmanage the subcloud from the previous System Controller.

  2. Update the admin password on the subcloud to match the new System Controller, if required.

  3. Run the dcmanager subcloud add command on the new System Controller with the --migrate and --release option to initiate the migration:

    ~(keystone_admin)]$ dcmanager subcloud add --migrate
--bootstrap-address <oam_ip_address_of_subclouds_controller-0>
--bootstrap-values /home/sysadmin/subcloud1-bootstrap-values.yaml
--release <xx.xx>

    Where, <xx.xx> is the StarlingX N-1 or N-2 version.

  4. On the subcloud, lock/unlock the subcloud controller(s) to enable the new configuration.

  5. Use the dcmanager subcloud list command to check the subcloud’s status. Ensure that the subcloud is online and fully operational before proceeding to manage the subcloud.

  6. On the new System Controller, set the subcloud to managed and wait for it to sync.

  7. Delete the subcloud from the previous System Controller.

For more detailed information, refer to the Rehome-a-Subcloud topic in the StarlingX N documentation.

Rehome StarlingX N Major Release Subcloud to StarlingX N Major Release System Controller

For more detailed information, see Rehome a Subcloud.

Note

You can add the --release <xx.xx> option to the dcmanager subcloud add --migrate command but it is not mandatory.

Where, <xx.xx> is the StarlingX N version.

Enroll a Subcloud

N-1 and N-2 subclouds can be enrolled to a System Controller running the N software release.

To initiate the enrollment of a subcloud running StarlingX release N-1 or N-2, ensure that the required StarlingX N-1 or N-2 release deploy files are uploaded.

Use the dcmanager subcloud add -enroll command with the -release option to enroll a subcloud. It is not possible to enroll a subcloud running the N software release to a System Controller running N-1 or N-2 software release. The subcloud should be factory installed with patches above r10 that supports enrollment without rebooting the subcloud to update the network and password configuration.