Manual Kubernetes Upgrade in Multi-Node¶

You can upgrade the Kubernetes version on a running system from one supported version to another.

Note

Each StarlingX release supports two or more consecutive Kubernetes releases. The default version on a fresh install will always be the latest Kubernetes release supported by a StarlingX release. Upgrades from previous releases will always start with the same Kubernetes version as the highest version from the release you are upgrading from.

Kubernetes upgrades previously required sequential version updates, one at a time. The upgrade system now checks for Kubernetes version skew, allowing kubelet components to run up to three minor versions behind the control plane. This enables multi-version upgrades in a single cycle, eliminating the need to upgrade kubelet through each intermediate version. The system marks all valid versions as available, simplifying version selection and reducing the number of upgrades steps.

About this task

To complete this task, you will apply the following updates (patches) and upgrade various systems.

Platform update: The platform update contains metadata for the new Kubernetes version and the Kubernetes networking pods templates for the new Kubernetes version.
Kubernetes Component Packages: Each supported Kubernetes version is packaged separately and includes the corresponding versions of kubeadm, kubelet, and kubectl binaries. The system uses Debian packages to manage Kubernetes components.

Prerequisites

The system must be clear of alarms.
All hosts must be unlocked, enabled and available.
All Kubernetes pods must be ready.
The installed applications must be compatible with the new Kubernetes version.
All updates required for the new Kubernetes version must be transferred to the active controller.
If you are using NetApp Trident in StarlingX r11.0 and have upgraded from the StarlingX previous version, ensure that your NetApp backend version is compatible with Trident 25.02.1 and follow the steps in Upgrade the NetApp Trident Software to upgrade the Trident drivers to 25.02.1 before upgrading Kubernetes to version 1.32.

Note

The sysadmin and admin passwords must be set to the same value prior to starting an upgrade from StarlingX Release r9 to StarlingX Release r10.

Procedure

Upload, apply and install the platform update.

Use the standard sw-patch, upload, apply and install commands to perform these operations.

Confirm that the system is healthy.

Check the current system health status, resolve any alarms and other issues reported by the system health-query-kube-upgrade command then recheck the system health status to confirm that all System Health fields are set to OK.

~(keystone_admin)]$ system health-query-kube-upgrade
    System Health:
    All hosts are provisioned: [OK]
    All hosts are unlocked/enabled: [OK]
    All hosts have current configurations: [OK]
    All hosts are patch current: [OK]
    Ceph Storage Healthy: [OK]
    No alarms: [OK]
    All kubernetes nodes are ready: [OK]
    All kubernetes control plane pods are ready: [OK]
    All kubernetes applications are in a valid state: [OK]

List the available Kubernetes versions.

On a fresh install of StarlingX r11.0, the following output appears, for example:

~(keystone_admin)]$ system kube-version-list
+---------+--------+-------------+
| version | target | state       |
+---------+--------+-------------+
| v1.29.2 | False  | unavailable |
| v1.30.6 | False  | unavailable |
| v1.31.5 | False  | unavailable |
| v1.32.2 | True   | active      |
+---------+--------+-------------+

If StarlingX was upgraded to r11.0, the following appears:

~(keystone_admin)]$ system kube-version-list
+---------+--------+-------------+
| version | target | state       |
+---------+--------+-------------+
| v1.29.2 | True   | active      |
| v1.30.6 | False  | available   |
| v1.31.5 | False  | available   |
| v1.32.2 | False  | available   |
+---------+--------+-------------+

The following meanings apply to the output shown:

Target

A value of True means that the target is currently selected for installation.

State

Can be one of:

active: The version is running everywhere.
partial: The version is running somewhere.
available: The version can be upgraded to.
unavailable: The version is not available for upgrading. Either it is a downgrade or it requires an intermediate upgrade first. Kubernetes can be only upgraded one version at a time.

Start the Kubernetes upgrade.

For example:

~(keystone_admin)]$ system kube-upgrade-start v1.32.2
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | upgrade-started                      |
+--------------+--------------------------------------+

The upgrade process checks the applied/available updates, the upgrade path, the health of the system, the installed applications compatibility and validates the system is ready for an upgrade.

Warning

The command system kube-upgrade-start --force causes the upgrade process to ignore non-management-affecting alarms. Kubernetes cannot be upgraded if there are management-affecting alarms.

Download the Kubernetes images.

For example:

~(keystone_admin)]$ system kube-upgrade-download-images
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | downloading-images                   |
| created_at   | 2025-03-26T18:44:46.854319+00:00     |
| updated_at   | None                                 |
+--------------+--------------------------------------+

Confirm that the download has completed.

~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | downloaded-images                    |
| created_at   | 2025-03-26T18:44:46.854319+00:00     |
| updated_at   | 2025-03-26T18:51:29.486448+00:00     |
+--------------+--------------------------------------+

Update all applications that require updating before performing the Kubernetes version upgrade. This will update all applications that have the timing: pre metadata setting.

~(keystone_admin)]$ system kube-pre-application-update
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | pre-updating-apps                    |
| created_at   | 2025-03-26T18:44:46.854319+00:00     |
| updated_at   | 2025-03-26T18:51:29.486448+00:00     |
+--------------+--------------------------------------+

Note

The state will change to pre-updated-apps when the app update has completed.

Upgrade Kubernetes networking, for example:

~(keystone_admin)]$ system kube-upgrade-networking
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | upgrading-networking                 |
| created_at   | 2025-03-26T18:44:46.854319+00:00     |
| updated_at   | 2025-03-26T18:51:29.486448+00:00     |
+--------------+--------------------------------------+

The state upgraded-networking will be entered when the networking upgrade has completed.

Upgrade the storage, for example:

~(keystone_admin)]$ system kube-upgrade-storage
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | upgrading-storage                    |
| created_at   | 2025-03-13T18:44:46.854319+00:00     |
| updated_at   | 2025-03-13T18:51:29.486448+00:00     |
+--------------+--------------------------------------+

The state upgraded-storage will be entered when the storage upgrade has completed.

Note

Repeat the following upgrade steps for each Kubernetes version (up to three minor versions) until you reach the target version or hit the version skew policy limit.

Upgrade the control plane on the first controller (controller-1), for example:

~(keystone_admin)]$ system kube-host-upgrade controller-1 control-plane
+-----------------------+-------------------------+
| Property              | Value                   |
+-----------------------+-------------------------+
| control_plane_version | v1.29.2                 |
| hostname              | controller-1            |
| id                    | 2                       |
| kubelet_version       | v1.29.2                 |
| personality           | controller              |
| status                | upgrading-control-plane |
| target_version        | v1.32.2                 |
+-----------------------+-------------------------+

The state upgraded-first-master will be entered when the first control plane upgrade has completed.

Upgrade the control plane on the second controller (controller-0), for example:

~(keystone_admin)]$ system kube-host-upgrade controller-0 control-plane
+-----------------------+-------------------------+
| Property              | Value                   |
+-----------------------+-------------------------+
| control_plane_version | v1.29.2                 |
| hostname              | controller-0            |
| id                    | 1                       |
| kubelet_version       | v1.29.2                 |
| personality           | controller              |
| status                | upgrading-control-plane |
| target_version        | v1.32.2                 |
+-----------------------+-------------------------+

The state upgraded-second-master will be entered when the upgrade has completed.

Note

The upgrade order of control plane nodes (Controller 0 and Controller 1) is not critical; either node can be upgraded first. However, both must be upgraded to the target version before proceeding with the next version.

Show the Kubernetes upgrade status for all hosts, for example:

~(keystone_admin)]$ system kube-host-upgrade-list
+----+--------------+-------------+----------------+-----------------------+-----------------+--------+
| id | hostname     | personality | target_version | control_plane_version | kubelet_version | status |
+----+--------------+-------------+----------------+-----------------------+-----------------+--------+
| 1  | controller-0 | controller  | v1.32.2        | v1.32.2               | v1.29.2         | None   |
| 2  | controller-1 | controller  | v1.32.2        | v1.32.2               | v1.29.2         | None   |
| 3  | worker-0     | worker      | v1.29.2        | N/A                   | v1.29.2         | None   |
| 4  | worker-1     | worker      | v1.29.2        | N/A                   | v1.29.2         | None   |
+----+--------------+-------------+----------------+-----------------------+-----------------+--------+

The control planes of both controllers are now upgraded to v1.32.2.

Upgrade kubelet on both controllers.

Either controller can be upgraded first.

The kubelets on all controller hosts must be upgraded before upgrading kubelets on worker hosts.

For each controller, do the following.

Lock the controller.

For example:

~(keystone_admin)]$ system host-lock controller-1

Apply the upgrade.

For example:

~(keystone_admin)]$ system kube-host-upgrade controller-1 kubelet
+-----------------------+-------------------+
| Property              | Value             |
+-----------------------+-------------------+
| control_plane_version | v1.32.2           |
| hostname              | controller-1      |
| id                    | 2                 |
| kubelet_version       | v1.29.2           |
| personality           | controller        |
| status                | upgrading-kubelet |
| target_version        | v1.32.2           |
+-----------------------+-------------------+

Confirm the kubelet upgrade status before proceeding.

For example:

~(keystone_admin)]$ system kube-host-upgrade-list
+----+--------------+-------------+----------------+-----------------------+-----------------+--------------------+
| id | hostname     | personality | target_version | control_plane_version | kubelet_version | status             |
+----+--------------+-------------+----------------+-----------------------+-----------------+--------------------+
| 1  | controller-0 | controller  | v1.32.2        | v1.32.2               | v1.29.2         | None               |
| 2  | controller-1 | controller  | v1.32.2        | v1.32.2               | v1.29.2         | upgraded-kubelet   |
| 3  | worker-0     | worker      | v1.29.2        | N/A                   | v1.29.2         | None               |
| 4  | worker-1     | worker      | v1.29.2        | N/A                   | v1.29.2         | None               |
+----+--------------+-------------+----------------+-----------------------+-----------------+--------------------+

Unlock the controller.

For example:

~(keystone_admin)]$ system host-unlock controller-1

Switch the active controller.
```
~(keystone_admin)]$ swact
```

Lock the controller-0.

~(keystone_admin)]$ system host-lock controller-0

Apply the upgrade.

~(keystone_admin)]$ system kube-host-upgrade controller-0 kubelet

Unlock the controller-0.

For example:

~(keystone_admin)]$ system host-unlock controller-0

Show the Kubernetes upgrade status.

~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | upgrading-kubelets                   |
| created_at   | 2025-03-26T18:44:46.854319+00:00     |
| updated_at   | 2025-03-26T19:41:57.936617+00:00     |
+--------------+--------------------------------------+

Upgrade the kubelet on each worker node by repeating the following steps for each worker host in the system:

Multiple worker hosts can be upgraded simultaneously provided there is sufficient capacity remaining on other worker hosts.
1. Lock the host.
  
  For example:
```
~(keystone_admin)]$ system host-lock worker-1
```
2. Perform the upgrade.
  
  For example:
```
~(keystone_admin)]$ system kube-host-upgrade worker-1 kubelet
+-----------------------+-------------------+
| Property              | Value             |
+-----------------------+-------------------+
| control_plane_version | v1.32.2           |
| hostname              | worker-1          |
| id                    | 3                 |
| kubelet_version       | v1.29.2           |
| personality           | worker            |
| status                | upgrading-kubelet |
| target_version        | v1.32.2           |
+-----------------------+-------------------+
```
3. Unlock the host.
  
  For example:
```
~(keystone_admin)]$ system host-unlock worker-1
```
  Note
  
  If your upgrade includes four or more Kubernetes versions, return to the “Upgrade the control plane” step after every three minor version upgrades. For example, if your target version is 1.32.2, and your starting version is more than three versions behind, you will need to repeat the control plane upgrade process to stay within the Kubernetes version skew policy.

Complete the Kubernetes upgrade.

~(keystone_admin)]$ system kube-upgrade-complete
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | upgrade-complete                     |
| created_at   | 2025-03-26T18:44:46.854319+00:00     |
| updated_at   | 2025-03-26T19:06:47.515747+00:00     |
+--------------+--------------------------------------+

Update all applications that require updating after the K8S Version Upgrade. This will update all applications that have the timing: post metadata setting.
```
~(keystone_admin)]$ system kube-post-application-update
```
Delete temporary resources associated with a Kubernetes upgrade procedure. This step will also help clear Alarm 900.007 if it is still running after the upgrade.
```
~(keystone_admin)]$ system kube-upgrade-delete
```