Manual Combined Kubernetes and Host Software Upgrade for AIO-SX¶

For AIO-SX deployments only, you can combine the host software upgrade and the Kubernetes version upgrade into a single manual procedure, instead of running them as two separate, sequential operations.

Note

This procedure applies to AIO-SX (All-in-one Simplex) configurations only. On all other configurations (AIO-DX and Standard), host software deployments and Kubernetes upgrades must be performed sequentially as two separate operations. See Manual Host Software Deployment and Manual Kubernetes Upgrade in AIO-SX.

Overview

In the legacy (default) upgrade flow, you first complete the host software deployment across all hosts (software deploy complete), and only afterwards do you run the Kubernetes version upgrade (system kube-upgrade-start through system kube-upgrade-complete) as a separate operation.

On AIO-SX, because there is a single controller, these two operations can be combined so that the host software upgrade and the Kubernetes upgrade are carried out together on controller-0. Combining the two operations provides:

a reduced overall upgrade duration, since the operations are performed together rather than one after the other, and
a reduced service outage, since the additional disruption of running the Kubernetes upgrade as a separate operation after the host software deployment is avoided.

This procedure uses the following two command families:

software …: The software commands (software upload, software deploy precheck, software deploy start, software deploy host, software deploy activate, software deploy complete, software deploy delete, software system-deploy init, software system-deploy show, and software system-deploy delete) manage the host software (Major or Patch Release) deployment.
system kube-upgrade …: The system kube-upgrade-* and related system kube-* commands (system kube-upgrade-start, system kube-upgrade-download-images, system kube-pre-application-update, system kube-upgrade-networking, system kube-upgrade-storage, system kube-host-upgrade, system kube-upgrade-complete, system kube-upgrade-delete, system kube-post-application-update) manage the Kubernetes version upgrade.

Prerequisites

A recent full backup is available. It is not explicitly required, but it is a good practice to have a recent full backup available prior to performing major changes to the system.
The system must be clear of alarms. It is highly recommended not to have any active alarms on the system, otherwise the upgrade will not proceed.
All Kubernetes pods must be ready, and all hosts must be unlocked, enabled, and available.
The installed applications must be compatible with both the new software release and the new Kubernetes version that the system will upgrade to.
If you are using a private container image registry, it must be populated with any new container images required for both the new software release and the target Kubernetes version.
The platform issuer (system-local-ca) must be configured with an RSA certificate/private key. If it was configured with a different type of certificate/private key, use the Update system-local-ca or Migrate Platform Certificates to use Cert Manager procedure to reconfigure it.

Note

The sysadmin and admin passwords must be set to the same value prior to starting an upgrade from StarlingX Release r11 to StarlingX Release r13.

Procedure

Transfer the new software release files to the active controller-0.

For a major release, this includes the major release install ISO, the software signature file, and the license file.

Install the license file for the release you are upgrading to.

~(keystone_admin)]$ system license-install <new-release-license-file>

Upload the new software release into the system, then confirm it was uploaded.

~(keystone_admin)]$ software upload [ --local ] <new-release>.iso <new-release>.sig

~(keystone_admin)]$ software list

Identify the target Kubernetes version.
```
~(keystone_admin)]$ system kube-version-list
```
Note the target vN.N.N version (for example, v1.30.6) for use in the next step.
Initialize the combined system deployment, binding the target software release and the target Kubernetes version in a single operation.
```
~(keystone_admin)]$ software system-deploy init <release-id> --kube-upgrade <k8s-version>
```
Where:
- <release-id> is the software release to deploy to (the release uploaded in the previous steps).
- --kube-upgrade <k8s-version> specifies the target Kubernetes version. The version must be in vN.N.N form (an optional leading v followed by three dot-separated numbers, for example v1.30.6).
  
  Note
  
  The --kube-upgrade option is what makes this a combined deployment. If it is omitted, only the host software deployment is initialized and no Kubernetes upgrade is performed.
Note

This command only supported for major releases upgrades runs the precheck (if it has not already been executed) and creates LVM snapshots. These snapshots can later be restored using either activate-rollback or host-rollback, whichever is executed first.

Confirm the initialized target release and Kubernetes version.
```
~(keystone_admin)]$ software system-deploy show
```
Confirm that the system is healthy for the Kubernetes upgrade.
```
~(keystone_admin)]$ system health-query-kube-upgrade
```
Resolve any reported issues and re-check until all System Health fields are OK.
Start the Kubernetes upgrade to the target version.
```
~(keystone_admin)]$ system kube-upgrade-start <k8s-version>
```
<k8s-version> is optional, if it is not added, the system infers the value from the software system-deploy init step (e.g. system kube-upgrade-start).

Warning

The command system kube-upgrade-start --force causes the upgrade process to ignore non-management-affecting alarms. Kubernetes cannot be upgraded if there are management-affecting alarms.
Download the Kubernetes images, then confirm the download completed.
```
~(keystone_admin)]$ system kube-upgrade-download-images
```
```
~(keystone_admin)]$ system kube-upgrade-show
```
Wait for the state downloaded-images.
Update the applications that require updating before the Kubernetes upgrade (applications with the timing: pre metadata setting).
```
~(keystone_admin)]$ system kube-pre-application-update
```
The state changes to pre-updated-apps when the update completes.
Upgrade Kubernetes networking.
```
~(keystone_admin)]$ system kube-upgrade-networking
```
Wait for the state upgraded-networking.
Upgrade the Kubernetes storage components.
```
~(keystone_admin)]$ system kube-upgrade-storage
```
Wait for the state upgraded-storage.
Upgrade the control plane on controller-0.
```
~(keystone_admin)]$ system kube-host-upgrade controller-0 control-plane
```
Verify with system kube-host-upgrade-list that the control plane status returns to None.

Note

Repeat the steps for each Kubernetes version (up to three minor versions) until you reach the target version or hit the version skew policy limit.
Start the host software deployment.
```
~(keystone_admin)]$ software deploy start [-f|--force] <release-id>
Deployment for <new-release-id> started
```
Note

<release-id> is optional, if it is not added, the system infers the value from the software system-deploy init step (e.g. software deploy start).

Monitor progress until the state reaches deploy-start-done.
```
~(keystone_admin)]$ software deploy show
```
Deploy the new software release to controller-0.
1. Lock controller-0.
```
~(keystone_admin)]$ system host-lock controller-0
```
2. Deploy the new software release to controller-0.
```
~(keystone_admin)]$ software deploy host controller-0
Host installation request sent to controller-0.
Host installation was successful on controller-0.
```
3. Unlock controller-0.
```
~(keystone_admin)]$ system host-unlock controller-0
```
  The host reboots into the new software release. Wait for the host to finish rebooting and become enabled. This may take 3-5 mins depending on hardware.
  
  Note
  
  This command will also upgrade the kubelets.
Activate the software deployment.
```
~(keystone_admin)]$ software deploy activate
Deploy activate has started
```
Wait for the state to change from deploy-activate to deploy-activate-done.

Complete the software deployment.

~(keystone_admin)]$ software deploy complete
Deployment has been completed

Complete the Kubernetes upgrade.
```
~(keystone_admin)]$ system kube-upgrade-complete
```
Wait for the state upgrade-complete.
Optional step: Update the applications that require updating after the Kubernetes upgrade (applications with the timing: post metadata setting).
```
~(keystone_admin)]$ system kube-post-application-update
```
Delete the temporary resources associated with the Kubernetes upgrade. This also helps clear Alarm 900.007 if it is still present after the upgrade.
```
~(keystone_admin)]$ system kube-upgrade-delete
```
Delete the host software deployment.

Note

For a major release deployment, after this command is executed, the major release software deployment can no longer be rolled back.
```
~(keystone_admin)]$ software deploy delete
Deployment has been deleted
```

Delete the combined system deployment.

~(keystone_admin)]$ software system-deploy delete

Confirm there is no deployment in progress.

~(keystone_admin)]$ software deploy show
No deploy in progress
~(keystone_admin)]$ software system-deploy show
No system deploy in progress.

Rollback¶

The combined deployment can be rolled back at any time while a deploy is in progress. Because the host software deployment and the Kubernetes upgrade are combined, the rollback unwinds them in reverse order, and the exact procedure depends on how far the deployment has progressed when you decide to roll back.

Important

The deploy is finalized with software deploy delete followed by software system-deploy delete. Once software deploy delete has run, the deployment can no longer be rolled back.

Note

Once a host software deployment is in progress (that is, after software deploy start), the Kubernetes upgrade cannot be aborted on its own with system kube-upgrade-abort. Use software deploy abort, which aborts the host software and Kubernetes upgrades together, as described in the stage-specific procedures below.

Important

If the KubeVirt application is installed, stop all running VMs before initiating the platform rollback. Running VMs during a rollback will cause live migration and VM management failures after the rollback completes. For details, see Stop VMs Before Platform Rollback.

Use the procedure that matches the stage you have reached.

Rollback After `software system-deploy init`¶

Use this procedure when you have run software system-deploy init but have not yet run system kube-upgrade-start. Nothing has been applied, so only the system deploy needs to be deleted.

Procedure

Delete the combined system deployment.

~(keystone_admin)]$ software system-deploy delete

Rollback After `system kube-upgrade-start`¶

Use this procedure when you have run system kube-upgrade-start but have not yet run software deploy start. No host software deployment is in progress, so the Kubernetes upgrade can be aborted directly.

Procedure

Abort the Kubernetes upgrade.

~(keystone_admin)]$ system kube-upgrade-abort

Delete the Kubernetes upgrade resources.

~(keystone_admin)]$ system kube-upgrade-delete

Delete the combined system deployment.

~(keystone_admin)]$ software system-deploy delete

Rollback After `software deploy start`¶

Use this procedure when you have run software deploy start but have not yet run software deploy host. If any host is locked, unlock it before proceeding.

Procedure

Delete the host software deployment.

~(keystone_admin)]$ software deploy delete

Abort the Kubernetes upgrade.

~(keystone_admin)]$ system kube-upgrade-abort

Delete the Kubernetes upgrade resources.

~(keystone_admin)]$ system kube-upgrade-delete

Delete the combined system deployment.

~(keystone_admin)]$ software system-deploy delete

Rollback After `software deploy host`¶

Use this procedure when you have run software deploy host but have not yet run software deploy activate.

Procedure

Abort the host software deployment.

~(keystone_admin)]$ software deploy abort

Roll back controller-0.
```
~(keystone_admin)]$ system host-lock <node>
~(keystone_admin)]$ software deploy host-rollback <node>
~(keystone_admin)]$ system host-unlock <node>
```
Note

The host-rollback command restores the LVM snapshots and rolls back both Kubernetes and the software. It also continues rolling back the OSTree deployment.

If snapshot restoration fails, the behavior depends on whether --kube-upgrade was used:
- With --kube-upgrade: host-rollback fails. You must review the logs, identify and correct the issue, and then retry the operation. If snapshot restoration cannot be completed, the recommended fallback is a backup-and-restore procedure.
- Without --kube-upgrade: host-rollback continues with the legacy software rollback mechanism.

Complete the software deployment.

~(keystone_admin)]$ software deploy complete

Delete the host software deployment.

~(keystone_admin)]$ software deploy delete

Delete the combined system deployment.

~(keystone_admin)]$ software system-deploy delete

Rollback After `software deploy activate`¶

Use this procedure when you have run software deploy activate.

Procedure

Abort the software deployment.

~(keystone_admin)]$ software deploy abort

Perform the activate-rollback.
```
~(keystone_admin)]$ software deploy activate-rollback
```
Note

The activate-rollback command restores the LVM snapshots and rolls back both Kubernetes and the software.

If snapshot restoration fails, the behavior depends on whether --kube-upgrade was used:
- With --kube-upgrade: activate-rollback fails. You must review the logs, correct the issue, and retry the operation. If snapshot restoration cannot be completed, the recommended fallback is a backup-and-restore procedure.
- Without --kube-upgrade: activate-rollback automatically falls back to the legacy software rollback mechanism.
Roll back controller-0.
```
~(keystone_admin)]$ system host-lock <node>
~(keystone_admin)]$ software deploy host-rollback <node>
~(keystone_admin)]$ system host-unlock <node>
```
Note

Since the LVM snapshots were already restored during the activate-rollback phase, this step skips snapshot restoration and only rolls back the OSTree deployment.

Complete the software deployment.

~(keystone_admin)]$ software deploy complete

Delete the host software deployment.

~(keystone_admin)]$ software deploy delete

Delete the combined system deployment.

~(keystone_admin)]$ software system-deploy delete

The system is now rolled back to the original software release and Kubernetes version, and is ready for the next host software deployment.