Prestage a Subcloud¶
Before you deploy platform software updates on the subcloud or reinstall it to restore the subcloud, the subcloud can be prestaged with OSTree repo (software updates) and container image archives outside the maintenance window. This can be done using the dcmanager CLI. The prestaging reduces the duration of the subsequent software update or reinstall in the maintenance window.
For information on prestaging a batch of subclouds, see, Prestage Subcloud Orchestration.
About this task
The main steps of this task are:
Ensure prestaging prerequisites are met, see Prestaging Requirements.
Upload the list of container images to prestage. This step is relevant to major release software deployment and must be performed after the system controller has been updated with the new major release. See Upload Prestage Image List.
Use dcmanager commands to prestage the subcloud(s).
To increase Subcloud Platform Backup Size using dcmanager CLI, see note.
Prestaging Requirements¶
Prerequisites
Prestaging can be done for a single subcloud or a batch of subclouds via orchestration. See Prestage Subcloud Orchestration.
There are two types of subcloud prestage:
Prestage for software deployment occurs when the subcloud is running an older software than the system controller during prestaging. This prestage transfers software update(s) and container images to the subcloud while preparing for new software deployment. The prestaged data is stored in various locations on the subcloud. For example, metadata is stored in
/opt/software/<sw-version>and new images are added to the local docker registry.Prestage for install occurs when the subcloud is running the same or older software than the system controller during prestaging. This prestage transfers the entire platform software and container images of the specified release while preparing for the subcloud redeployment. This is used if the subcloud needs to be reinstalled and restored to a specific release. The prestaged data is stored in the subcloud persistent filesystem
/opt/platform-backup/<sw-version>.
Pre-conditions common to both types of prestage:
Subclouds to be prestaged must be online, managed, and free of any management affecting alarms.
Note
You can force prestaging using
--forceoption. However, it is not recommended unless it is certain that the prestaging process will not exacerbate the alarm condition on the subcloud.Subcloud
/opt/platform-backupmust have enough available disk space for prestage data.Subcloud
/var/lib/dockermust have enough space for all prestage image pulls and archive file generation. If the total size of prestage images is N GB, available Docker space should be N*2 GB.
Pre-conditions specific to prestage for install:
The active controller must be controller-0.
Subcloud
/opt/platform-backupmust have enough available disk space for prestaged data.
Note
In case of prestage for-install, the ISO to be uploaded using the system software upload command must be at the same patch level as the system controller. If the system controller is patched after prestaging of subclouds, you need to repeat the prestaging of each subcloud. This ensures that the subcloud boot image aligns with the patch level of the load that needs to be installed on the subcloud.
Pre-conditions specific to prestage for the deployment of major release:
The total size of prestage images and custom images restored over upgrade must not exceed docker-distribution capacity.
Prestage images must already exist in the configured source(s) prior to subcloud prestaging. For example, if the subcloud is configured to download images from the central registry, the specified images must already exist in the registry on the system controller.
Upload Prestage Image List¶
The prestage image list specifies what container images are to be pulled from the configured sources and included in the image archive files during prestaging. This list is only used if the prestage is intended for deployment of a major release, that is, subclouds are running different loads at the time of prestaging.
The prestage image list must contain:
Images required for the deployment of major platform update.
Images required for the restore and update or StarlingX applications, currently applied on the subcloud. For example:
cert-manager
OIDC
metrics-server
If the available docker and docker-distribution storage is ample, prestage image list should also contain:
(Optional) Images required for Kubernetes version upgrades post subcloud upgrade.
(Optional) Images required for the update of end users’ Helm applications post major platform update.
Note
It is recommended to determine the total size of all images to be prestaged in advance and resize the docker-distribution (subcloud local registry) filesystem using the system controllerfs-modify command accordingly.
Procedure
To upload the prestage image list, use the following command after the system controller has been upgraded:
~(keystone_admin)]$ dcmanager subcloud deploy upload --prestage-images nn.nn_images.lst +------------------+-----------------+ | Field | Value | +------------------+-----------------+ |deploy_playbook | None | |deploy_overrides | None | |deploy_chart | None | |prestage_images | nn.nn_images.lst| +------------------+-----------------+
Where, the name of the prestage image file can be user defined. However, it is recommended to use the following format <software_version>_images.lst, for example, <21.12_images.lst>.
To confirm that the image list has been uploaded, use the following command.
~(keystone_admin)]$ dcmanager subcloud deploy show +------------------+-------------------------+ | Field | Value | +------------------+-------------------------+ | deploy_playbook | None | | deploy_overrides | None | | deploy_chart | None | | prestage_images | nn.nn_images.lst | +------------------+-------------------------+
Warning
As prestage images will be pulled from Docker registries currently configured for the subcloud, images in the image list file must not contain custom/private registry prefix.
Single Subcloud Prestage¶
See Prestaging Requirements for preconditions prior to prestaging the subcloud.
Subcloud can be prestaged for install of current/previous software version or for software deployment, and the previous release software must be uploaded before prestaging.
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud prestage --for-install --release nn.nn subcloud7
Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm:
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| id | 7 |
| name | subcloud7 |
| description | None |
| location | None |
| software_version | 22.12 |
| management | managed |
| availability | online |
| deploy_status | complete |
| management_subnet | fdff:719a:bf60:4028::/64 |
| management_start_ip | fdff:719a:bf60:4028::2 |
| management_end_ip | fdff:719a:bf60:4028::ffff |
| management_gateway_ip | fdff:719a:bf60:4028::1 |
| systemcontroller_gateway_ip | fd00:8:16::1 |
| group_id | 1 |
| peer_group_id | None |
| created_at | 2024-10-25 06:06:39.527023 |
| updated_at | 2024-10-25 08:08:16.311164 |
| backup_status | None |
| backup_datetime | None |
| prestage_status | prestaging-packages |
| prestage_versions | |
| prestage_software_version | 22.12 |
+-----------------------------+----------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud prestage --for-install subcloud5
Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm:
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| id | 8 |
| name | subcloud5 |
| description | None |
| location | None |
| software_version | 24.09 |
| management | managed |
| availability | online |
| deploy_status | complete |
| management_subnet | fd00:8:28::/64 |
| management_start_ip | fd00:8:28::2 |
| management_end_ip | fd00:8:28::10 |
| management_gateway_ip | fd00:8:28::1 |
| systemcontroller_gateway_ip | fd00:8:16::1 |
| group_id | 1 |
| peer_group_id | None |
| created_at | 2024-10-25 06:28:57.197765 |
| updated_at | 2024-10-25 08:08:38.433790 |
| backup_status | None |
| backup_datetime | None |
| prestage_status | prestaging-packages |
| prestage_versions | |
| prestage_software_version | 24.09 |
+-----------------------------+----------------------------+
Note
If the
--releaseoption is not specified, the subcloud will be prestaged for install with the same software as the system controller. Also, the release format must be MM.mm and any other format is not valid.If the release is not in deployed/available state and if it tries to do prestaging for the
--for-sw-deployoption, prestaging will throw an error.To verify the release, run the software list command.
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud prestage --for-sw-deploy subcloud5
Enter the sysadmin password for the subcloud:
Re-enter sysadmin password to confirm:
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| id | 8 |
| name | subcloud5 |
| description | None |
| location | None |
| software_version | 24.09 |
| management | managed |
| availability | online |
| deploy_status | complete |
| management_subnet | fd00:8:28::/64 |
| management_start_ip | fd00:8:28::2 |
| management_end_ip | fd00:8:28::10 |
| management_gateway_ip | fd00:8:28::1 |
| systemcontroller_gateway_ip | fd00:8:16::1 |
| group_id | 3 |
| peer_group_id | None |
| created_at | 2024-09-21 15:21:20.418706 |
| updated_at | 2024-09-22 08:44:30.937440 |
| backup_status | None |
| backup_datetime | None |
| prestage_status | prestaging-packages |
| prestage_versions | |
| prestage_software_version | 24.09 |
+-----------------------------+----------------------------+
Note
To perform the software deployment prestaging, system controller should have higher patch level than the subcloud. Post prestaging, the release will be in available state in the subcloud. If the subcloud has higher patch level than system controller, prestaging will be skipped.
Rerun Subcloud Prestage¶
A subcloud can be prestaged multiple times. Software prestaging will be skipped if the subcloud already has the specified software version. Prestaging of container images will be repeated.
Verify Subcloud Prestage¶
After a subcloud is successfully prestaged, prestage_status will change to
complete and prestage_versions will show the software version(s) the
subcloud has prestaged data for. Use the dcmanager subcloud show
command to verify the status and versions.
Troubleshoot Subcloud Prestage¶
If the subcloud prestage fails, check /var/log/dcmanager/dcmanager.log and
/var/log/dcmanager/ansible/<subcloud-name>_playbook_output.log for the reason
of failure. Once the issue has been resolved, prestage can be retried using
the dcmanager subcloud prestage command.
Verifying Usage of Prestaged Data¶
To verify that the prestaged data is used over subcloud update with new software or subcloud redeployment/restore:
Search for the the subcloud name in the log file, for example, subcloud1 from
/www/var/log/lighttpd-access.log. There should not be GET requests to download packages from/iso/<sw_version>/nodes/subcloud1/Packages/.Check subcloud ansible log in the
/var/log/dcmanager/ansibledirectory. Images are imported from local archives and no images in the prestage image list need to be downloaded from the configured sources.If we reinstall the subcloud, then we can check
/root/install.logto validate if the prestage data was used during installation.
