Create Kubernetes Version Upgrade Cloud Orchestration Strategy¶
You can configure Kubernetes Version Upgrade Orchestration Strategy using the sw-manager CLI.
Note
You require administrator privileges to use sw-manager. You must log in to the active controller as user sysadmin and source the script by using the command, source /etc/platform/openrc to obtain administrator privileges. Do not use sudo.
Note
Management-affecting alarms cannot be ignored using relaxed alarm rules during an orchestrated Kubernetes version upgrade operation. For a list of management-affecting alarms, see StarlingX Fault Management: Alarm Messages. To display management-affecting active alarms, use the following command:
~(keystone_admin)$ fm alarm-list --mgmt_affecting
During an orchestrated Kubernetes version upgrade operation, the following alarms are ignored even when the default strict restrictions are selected:
100.103: Memory threshold exceeded.
200.001: Locked host.
280.001: Subcloud resource off-line.
280.002: Subcloud resource out-of-sync.
700.004: VM stopped.
750.006: Configuration change requires reapply of cert-manager.
900.001: Patch in progress.
900.007: Kube upgrade in progress.
900.401: kube-upgrade-auto-apply-inprogress.
You can use help for the overall commands and also for each sub-command.
For example:
~(keystone_admin)$ sw-manager kube-upgrade-strategy –help
usage: sw-manager kube-upgrade-strategy [-h] ...
optional arguments:
-h, --help show this help message and exit
Kubernetes Update Commands:
create Create a strategy
delete Delete a strategy
apply Apply a strategy
abort Abort a strategy
show Show a strategy
Prerequisites
Hosts that need to be upgraded must be in the
unlocked-enabledstate.If you are using NetApp Trident, ensure that your NetApp version is compatible with Trident 21.04 before upgrading Kubernetes to version 1.19 and after updating StarlingX to version 6.0. For more information, see Upgrade the NetApp Trident Software.
Procedure
List available upgrades.
~(keystone_admin)$ system kube-version-list +-----------------+--------+-------------+ | version | target | state | +-----------------+--------+-------------+ | v1.18.1 | True | active | | v1.19.13 | False | available | | v1.20.9 | False | unavailable | | v1.21.8 | False | unavailable | +-----------------+--------+-------------+
Confirm that the system is healthy.
Check the current system health status, resolve any alarms and other issues reported by the system health-query-kube-upgrade command then recheck the system health status to confirm that all System Health fields are set to OK.
By default, the upgrade process cannot be run and is not recommended to be run with active alarms present. Use the system kube-upgrade-start --force command to force the upgrade process to start and ignore non-management-affecting alarms.
Note
It is strongly recommended that you clear your system of any and all alarms before doing an upgrade. While the --force option is available to run the upgrade, it is a best practice to clear any alarms.
~(keystone_admin)]$ system health-query-kube-upgrade System Health: All hosts are provisioned: [OK] All hosts are unlocked/enabled: [OK] All hosts have current configurations: [OK] All hosts are patch current: [OK] Ceph Storage Healthy: [OK] No alarms: [OK] All kubernetes nodes are ready: [OK] All kubernetes control plane pods are ready: [OK] Required patches are applied: [OK] License valid for upgrade: [OK] No instances running on controller-1: [OK] All kubernetes applications are in a valid state: [OK] Active controller is controller-0: [OK]Create the strategy.
The Kubernetes Version Upgrade Orchestration Strategy create command creates a series of stages with steps that apply the Kubernetes version upgrade.
Kubernetes Version upgrade requires a reboot. Therefore, the created strategy includes steps that automatically lock and unlock the host to bring the new image function into service.
~(keystone_admin)$ sw-manager kube-upgrade-strategy create --to-version v1.19.13 Strategy Kubernetes Upgrade Strategy: strategy-uuid: f7585178-cea6-4d2f-bda0-e0972145ebcf controller-apply-type: serial storage-apply-type: ignore worker-apply-type: serial default-instance-action: migrate alarm-restrictions: strict current-phase: build current-phase-completion: 0% state: building inprogress: true
where:
--to-versionThe version of Kubernetes to upgrade to. For example,
v1.19.13. This argument is required.--controller-apply-typeand--storage-apply-typeThese options cannot be changed from
serialbecause Kubernetes upgrade concurrency is only supported for worker hosts.Note
Kubernetes version upgrade is currently only supported for hosts with worker function. Any attempt to modify the controller or storage apply type is rejected.
--worker-apply-typeThis option specifies the host concurrency of the Kubernetes version upgrade strategy:
serial (default): worker hosts will be patched one at a time
parallel: worker hosts will be upgraded in parallel
At most,
parallelwill be upgraded at the same timeAt most, half of the hosts in a host aggregate will be upgraded at the same time
ignore: worker hosts will not be upgraded; strategy create will fail
Worker hosts with no instances are upgraded before worker hosts with instances.
--max-parallel-worker-hostsThis option applies to the parallel worker apply type selection to specify the maximum worker hosts to upgrade in parallel (minimum: 2, maximum: 10).
–instance-actionThis option only has significance when the stx-openstack application is loaded and there are instances running on worker hosts. It specifies how the strategy deals with worker host instances over the strategy execution.
stop-start(default)Instances will be stopped before the host lock operation following the upgrade and then started again following the host unlock.
Warning
Using the
stop-startoption will result in an outage for each instance, as it is stopped while the worker host is locked/unlocked. In order to ensure this does not impact service, instances MUST be grouped into anti-affinity (or anti-affinity best effort) server groups, which will ensure that only a single instance in each server group is stopped at a time.migrateInstances will be migrated off a host before it is patched (this applies to reboot patching only).
--alarm-restrictionsThis option sets how the how the Kubernetes version upgrade orchestration behaves when alarms are present.
To display management-affecting active alarms, use the following command:
~(keystone_admin)$ fm alarm-list --mgmt_affecting
strict(default)The default strict option will result in patch orchestration failing if there are any alarms present in the system (except for a small list of alarms).
relaxedThis option allows orchestration to proceed if alarms are present, as long as none of these alarms are management affecting.
~(keystone_admin)]$ sw-manager kube-upgrade-strategy create --help usage:sw-manager kube-upgrade-strategy [-h] --to-version <kubernetesVersion> [--controller-apply-type {ignore}] [--storage-apply-type {ignore}] [--worker-apply-type {serial,parallel,ignore}] [--max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10}] [--instance-action {migrate,stop-start}] [--alarm-restrictions {strict,relaxed}] optional arguments: -h, --help show this help message and exit --controller-apply-type {serial,ignore} defaults to serial --storage-apply-type {serial,ignore} defaults to serial --worker-apply-type {serial,parallel,ignore} defaults to serial --max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10} maximum worker hosts to update in parallel --instance-action {migrate,stop-start} defaults to stop-start --alarm-restrictions {strict,relaxed} defaults to strictOptional: Display the strategy in summary, if required. The Kubernetes upgrade strategy show command displays the strategy in a summary.
~(keystone_admin)$ sw-manager kube-upgrade-strategy show Strategy Kubernetes Upgrade Strategy: strategy-uuid: f7585178-cea6-4d2f-bda0-e0972145ebcf controller-apply-type: serial storage-apply-type: ignore worker-apply-type: serial default-instance-action: migrate alarm-restrictions: strict current-phase: build current-phase-completion: 100% state: ready-to-apply build-result: success build-reason:
The show strategy subcommand displays a summary of the current state of the strategy. A complete view of the strategy can be shown using the
--detailsoption.The strategy steps and stages are displayed using the
--detailsoption.Apply the strategy.
Kubernetes Version Upgrade Orchestration Strategy apply command executes the strategy stages and steps consecutively until the Kubernetes upgrade on all the hosts in the strategy is complete.
Use the
-stage-idoption to specify a specific stage to apply; one at a time.Note
When applying a single stage, only the next stage will be applied; you cannot skip stages.
~(keystone_admin)$ sw-manager kube-upgrade-strategy apply Strategy Kubernetes upgrade Strategy: strategy-uuid: 3e43c018-9c75-4ba8-a276-472c3bcbb268 controller-apply-type: ignore storage-apply-type: ignore worker-apply-type: serial default-instance-action: stop-start alarm-restrictions: strict current-phase: apply current-phase-completion: 0% state: applying inprogress: true
Use the kube-upgrade-show command to monitor Kubernetes upgrade state and percentage completion.
~(keystone_admin)$ system kube-upgrade-show +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | uuid | 3d2da123-bff4-4b3a-a64a-b320c3b498cc | | from_version | v1.18.1 | | to_version | v1.19.13 | | state | downloading-images | | created_at | 2021-02-23T00:08:24.579257+00:00 | | updated_at | 2021-02-23T00:09:35.413307+00:00 | +--------------+--------------------------------------+
You will see the
stateproperty transition through values such asdownloading-images,downloaded-images,upgrading-first-master,upgraded-first-master, etc.Optional: Abort the strategy, if required. This is only used to stop, and abort the entire strategy.
The Kubernetes version upgrade strategy abort command can be used to abort the Kubernetes version upgrade strategy after the current step of the currently applying stage is completed.
Confirm that the upgrade has completed successfully.
~(keystone_admin)$ system kube-upgrade-show +--------------+--------------------------------------+ | Property | Value | +--------------+--------------------------------------+ | uuid | 426d7e11-2de2-40ba-b482-ed3691625383 | | from_version | v1.18.1 | | to_version | v1.19.13 | | state | upgrade-complete | | created_at | 2021-04-12T17:58:36.492523+00:00 | | updated_at | 2021-04-12T18:49:11.673259+00:00 | +--------------+--------------------------------------+ ~(keystone_admin)$ system kube-version-list +-----------------+--------+-------------+ | version | target | state | +-----------------+--------+-------------+ | v1.18.1 | True | unavailable | | v1.19.13 | False | active | | v1.20.9 | False | available | | v1.21.8 | False | unavailable | +-----------------+--------+-------------+
Delete the strategy.
Note
After the Kubernetes Version Upgrade Orchestration Strategy has been applied (or aborted) it must be deleted before another Kubernetes version upgrade strategy can be created. If a Kubernetes version upgrade strategy application fails, you must address the issue that caused the failure, then delete and re-create the strategy before attempting to apply it again.
~(keystone_admin)$ sw-manager kube-upgrade-strategy delete Strategy deleted.
