Perform an Orchestrated Upgrade Using the CLI¶

The upgrade orchestration CLI is sw-manager.

About this task

Note

To use upgrade orchestration commands, you need administrator privileges. You must log in to the active controller as user sysadmin and source the /etc/platform/openrc script to obtain administrator privileges. Do not use sudo.

The upgrade strategy options are shown in the following output:

~(keystone_admin)]$ sw-manager upgrade-strategy --help
usage: sw-manager upgrade-strategy [-h]  ...

optional arguments:
  -h, --help  show this help message and exit

Software Upgrade Commands:

    create    Create a strategy
    delete    Delete a strategy
    apply     Apply a strategy
    abort     Abort a strategy
    show      Show a strategy

You can perform a partially orchestrated upgrade using the CLI. Upgrade orchestration of other StarlingX nodes can be initiated after the initial controller host has been manually upgraded and returned to a stability state.

Note

Management-affecting alarms cannot be ignored at the indicated severity level or higher by using relaxed alarm rules during an orchestrated upgrade operation. For a list of management-affecting alarms, see StarlingX Fault Management: Alarm Messages. To display management-affecting active alarms, use the following command:

~(keystone_admin)]$ fm alarm-list --mgmt_affecting

During an orchestrated upgrade, the following alarms are ignored even when strict restrictions are selected:

900.005, Upgrade in progress
900.201, Software upgrade auto apply in progress

Prerequisites

See Upgrading All-in-One Duplex / Standard, and perform Steps 1-8, to manually upgrade the initial controller node before doing the upgrade orchestration described below to upgrade the remaining nodes of the StarlingX.

The subclouds must use the Redfish platform management service if it is an All-in-one Simplex subcloud.
Duplex (AIODX/Standard) upgrades are supported, and they do not require remote install via Redfish.

Procedure

Create a update strategy using the sw-manager upgrade-strategy create command.
```
~(keystone_admin)]$ sw-manager upgrade-strategy create

strategy-uuid:                          5435e049-7002-4403-acfb-7886f6da14af
controller-apply-type:                  serial
storage-apply-type:                     serial
worker-apply-type:                      serial
default-instance-action:                migrate
alarm-restrictions:                     strict
current-phase:                          build
current-phase-completion:               0%
state:                                  building
inprogress:                             true
```
Create an upgrade strategy, specifying the following parameters:
- storage-apply-type:
  - serial (default): storage hosts will be upgraded one at a time
  - parallel: storage hosts will be upgraded in parallel, ensuring that only one storage node in each replication group is patched at a time.
  - ignore: storage hosts will not be upgraded
- worker-apply-type:
  
  serial (default)
  Worker hosts will be upgraded one at a time.
  
  ignore
  Worker hosts will not be upgraded.
- Alarm Restrictions
  
  This option lets you determine how to handle alarm restrictions based on the management affecting statuses of any existing alarms, which takes into account the alarm type as well as the alarm’s current severity. If set to relaxed, orchestration will be allowed to proceed if there are no management affecting alarms present.
  
  Performing management actions without specifically relaxing the alarm checks will still fail if there are any alarms present in the system (except for a small list of basic alarms for the orchestration actions such as an upgrade operation in progress alarm not impeding upgrade orchestration).
  
  You can use the CLI command fm alarm-list --mgmt_affecting to view the alarms that are management affecting.
  
  Strict
  Maintains alarm restrictions.
  
  Relaxed
  Relaxes the usual alarm restrictions and allows the action to proceed if there are no alarms present in the system with a severity equal to or greater than its management affecting severity.
The upgrade strategy consists of one or more stages, which consist of one or more hosts to be upgraded at the same time. Each stage will be split into steps (for example, query-alarms, lock-hosts, upgrade-hosts). Following are some notes about stages:
- Controller-0 is upgraded first, followed by storage hosts and then worker hosts.
- Worker hosts with no instances are upgraded before worker hosts with application pods.
- Pods will be relocated off each worker host before it is upgraded.
- The final step in each stage is one of:
  
  system-stabilize
  This waits for a period of time (up to several minutes) and ensures that the system is free of alarms. This ensures that we do not continue to upgrade more hosts if the upgrade has caused an issue resulting in an alarm.
  
  wait-data-sync
  This waits for a period of time (up to many hours) and ensures that data synchronization has completed after the upgrade of a controller or storage node.
Examine the upgrade strategy. Pay careful attention to:
- The sets of hosts that will be upgraded together in each stage.
- The sets of pods that will be impacted in each stage.
  
  Note
  
  It is likely that as each stage is applied, pods will be relocated to worker hosts that have not yet been upgraded. That means that later stages will be relocating more pods than those originally listed in the upgrade strategy. The upgrade strategy is NOT updated, but any additional pods on each worker host will be relocated before it is upgraded.

Run sw-manager upgrade-strategy show command, to display the current-phase-completion percentage progress indicator in various increments. Once at 100%, it returns:

~(keystone_admin)]$ sw-manager upgrade-strategy show

strategy-uuid:                          5435e049-7002-4403-acfb-7886f6da14af
controller-apply-type:                  serial
storage-apply-type:                     serial
worker-apply-type:                      serial
default-instance-action:                migrate
alarm-restrictions:                     strict
current-phase:                          build
current-phase-completion:               100%
state:                                  ready-to-apply
build-result:                           success
build-reason:

Apply the upgrade strategy. You can optionally apply a single stage at a time.
```
~(keystone_admin)]$ sw-manager upgrade-strategy apply

strategy-uuid:                          5435e049-7002-4403-acfb-7886f6da14af
controller-apply-type:                  serial
storage-apply-type:                     serial
worker-apply-type:                      serial
default-instance-action:                migrate
alarm-restrictions:                     strict
current-phase:                          apply
current-phase-completion:               0%
state:                                  applying
inprogress:                             true
```
While an upgrade strategy is being applied, it can be aborted. This results in:
- The current step will be allowed to complete.
- If necessary an abort phase will be created and applied, which will attempt to unlock any hosts that were locked.
After an upgrade strategy has been applied (or aborted) it must be deleted before another upgrade strategy can be created. If an upgrade strategy application fails, you must address the issue that caused the failure, then delete/re-create the strategy before attempting to apply it again.

Run sw-manager upgrade-strategy show command, to display the current-phase-completion displaying the field goes from 0% to 100% in various increments. Once at 100%, it returns:

~(keystone_admin)]$ sw-manager upgrade-strategy show

strategy-uuid:                          b91d8332-9ece-4578-b4dd-e9cf87b73f18
controller-apply-type:                  serial
storage-apply-type:                     serial
worker-apply-type:                      serial
default-instance-action:                migrate
alarm-restrictions:                     strict
current-phase:                          apply
current-phase-completion:               100%
state:                                  applied
apply-result:                           success
apply-reason:

Activate the upgrade.

During the running of the upgrade-activate command, new configurations are applied to the controller. 250.001 (hostname Configuration is out-of-date) alarms are raised and are cleared as the configuration is applied. The upgrade state goes from activating to activation-complete once this is done.
```
~(keystone_admin)]$ system upgrade-activate
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state        | activating                           |
| from_release | nn.nn                                |
| to_release   | nn.nn                                |
+--------------+--------------------------------------+
```
The following states apply when this command is executed.

activation-requested
State entered when system upgrade-activate is executed.

activating
State entered when we have started activating the upgrade by applying new configurations to the controller and compute hosts.

activation-complete
State entered when new configurations have been applied to all controller and compute hosts.

Check the status of the upgrade again to see it has reached activation-complete

~(keystone_admin)]$ system upgrade-show
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state        | activation-complete                  |
| from_release | nn.nn                                |
| to_release   | nn.nn                                |
+--------------+--------------------------------------+

Complete the upgrade.

~(keystone_admin)]$ system upgrade-complete
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | 61e5fcd7-a38d-40b0-ab83-8be55b87fee2 |
| state        | completing                           |
| from_release | nn.nn                                |
| to_release   | nn.nn                                |
+--------------+--------------------------------------+

Delete the imported load.

~(keystone_admin)]$ system load-list
+----+----------+------------------+
| id | state    | software_version |
+----+----------+------------------+
| 1  | imported | nn.nn            |
| 2  | active   | nn.nn            |
+----+----------+------------------+

~(keystone_admin)]$ system load-delete 1
Deleted load: load 1