Upgrade Orchestration Overview¶

Upgrade Orchestration automates much of the upgrade procedure, leaving a few manual steps for operator oversight.

Note

Upgrading of Distributed Cloud is distinct from upgrading other StarlingX configurations.

Note

The upgrade orchestration commands are prefixed with sw-manager. To use upgrade orchestration commands, you need administrator privileges. You must log in to the active controller as user sysadmin and source the /etc/platform/openrc script to obtain administrator privileges. Do not use sudo.

~(keystone_admin)]$ sw-manager upgrade-strategy --help
usage: sw-manager upgrade-strategy [-h]  ...

optional arguments:
  -h, --help  show this help message and exit

Software Upgrade Commands:

    create    Create a strategy
    delete    Delete a strategy
    apply     Apply a strategy
    abort     Abort a strategy
    show      Show a strategy

Upgrade Orchestration Requirements ¶

Upgrade orchestration can only be done on a system that meets the following conditions:

The system is clear of alarms (with the exception of the alarm upgrade in progress).
All hosts must be unlocked, enabled, and available.
The system is fully redundant (two controller nodes available, at least one complete storage replication group available for systems with Ceph backend).
An upgrade has been started, and controller-1 has been upgraded and is active.
No orchestration strategy exists. Patch, upgrade, firmware, kubernetes, and kube rootca are all types of orchestration. An upgrade cannot be orchestrated while another orchestration is in progress.
Sufficient free capacity or unused worker resources must be available across the cluster. A rough calculation is:

Required spare capacity ( %) = (<Number-of-hosts-to-upgrade-in-parallel> / <total-number-of-hosts>) * 100

The Upgrade Orchestration Process ¶

Upgrade orchestration can be initiated after the initial controller host has been manual upgraded and returned to a stability state. Upgrade orchestration automatically iterates through the remaining hosts, installing the new software load on each one: first the other controller host, then the storage hosts, and finally the worker hosts. During worker host upgrades, pods are automatically moved to alternate worker hosts.

You first create an upgrade orchestration strategy, or plan, for the automated upgrade procedure. This customizes the upgrade orchestration, using parameters to specify:

the host types to be upgraded
whether to upgrade hosts serially or in parallel

Based on these parameters, and the state of the hosts, upgrade orchestration creates a number of stages for the overall upgrade strategy. Each stage generally consists of moving pods, locking hosts, installing upgrades, and unlocking hosts for a subset of the hosts on the system.

After creating the upgrade orchestration strategy, the you can either apply the entire strategy automatically, or apply individual stages to control and monitor their progress manually.

Update and upgrade orchestration are mutually exclusive; they perform conflicting operations. Only a single strategy (sw-patch or sw-upgrade) is allowed to exist at a time. If you need to update during an upgrade, you can abort/delete the sw-upgrade strategy, and then create and apply a sw-patch strategy before going back to the upgrade.

Some stages of the upgrade could take a significant amount of time (hours). For example, after upgrading a storage host, re-syncing the OSD data could take 30 minutes per TB (assuming 500MB/s sync rate, which is about half of a 10G infrastructure link).

Upgrade Orchestration Workflow ¶

The Upgrade Orchestration procedure has several major parts:

Manually upgrade controller-1.
Orchestrate the automatic upgrade of the remaining controller, all the storage nodes, and all the worker nodes.
Manually complete the upgrade.