Manual Kubernetes Multi-Version Upgrade in AIO-SX

You can upgrade the Kubernetes version on a running system from one supported version to another.

AIO-SX now supports multi-version Kubernetes upgrades. In this model, Kubernetes is upgraded by two or more versions after disabling applications and then applications are enabled again. This is faster than upgrading Kubernetes one version at a time. Also, the upgrade can be aborted and reverted to the original version. This feature is supported only for AIO-SX.

AIO-SX supports the Kubernetes multi-version upgrade. Thus, Kubernetes can be upgraded from the lowest version to the highest version available in the system. This feature is not supported in the system which is not AIO-SX.

Note

Each StarlingX release supports two or more consecutive Kubernetes releases. The default version on a fresh install will always be the latest Kubernetes release supported by a StarlingX release. Upgrades from previous releases will always start with the same Kubernetes version as the highest version from the release you are upgrading from.

Prerequisites

  • The system must be clear of alarms.

  • All hosts must be unlocked, enabled, and available.

  • All Kubernetes pods must be ready.

  • The installed applications must be compatible with the new Kubernetes versions that the system will upgrade to.

  • If you are using NetApp Trident in StarlingX r9.0 and if you have upgraded from the StarlingX previous version, ensure that your NetApp backend version is compatible with Trident 22.07. Follow the steps in Upgrade the NetApp Trident Software to upgrade Trident to 22.07 before upgrading Kubernetes to version 1.24.

Procedure

  1. List the available Kubernetes versions.

    On a fresh install of StarlingX r9.0, the following output appears:

    ~(keystone_admin)]$ system kube-version-list
+---------+--------+-------------+
| Version | Target | State       |
+---------+--------+-------------+
| v1.21.8 | False  | unavailable |
| v1.22.5 | False  | unavailable |
| v1.23.1 | False  | unavailable |
| v1.24.4 | True   | active      |
+---------+--------+-------------+

    If StarlingX was upgraded to r9.0, the following output appears:

    ~(keystone_admin)]$ system kube-version-list
+---------+--------+-------------+
| Version | Target | State       |
+---------+--------+-------------+
| v1.21.8 | True   | active      |
| v1.22.5 | False  | available   |
| v1.23.1 | False  | available   |
| v1.24.4 | False  | available   |
+---------+--------+-------------+

    The following meanings apply to the output shown:

    Target

    Target is either true or false. Target will be true only for the active Kubernetes version.

    State

    State can be one of the following:

    active: The version is running everywhere.

    partial: The version is running somewhere.

    available: The version can be upgraded.

    unavailable: The version is not available for upgrading.

  2. Confirm that the system is healthy.

    Check the current system health status, resolve any alarms and other issues reported by the system health-query-kube-upgrade command, and recheck the system health status to confirm that all the System Health fields are set to OK.

    ~(keystone_admin)]$ system health-query-kube-upgrade
    System Health:
    All hosts are provisioned: [OK]
    All hosts are unlocked/enabled: [OK]
    All hosts have current configurations: [OK]
    All hosts are patch current: [OK]
    Ceph Storage Healthy: [OK]
    No alarms: [OK]
    All kubernetes nodes are ready: [OK]
    All kubernetes control plane pods are ready: [OK]
    Required patches are applied: [OK]
    License valid for upgrade: [OK]
    All kubernetes applications are in a valid state: [OK]
    Active controller is controller-0: [OK]

  3. Start the Kubernetes multi-version upgrade.

    Specify the desired target version available to upgrade.

    For example:

    ~(keystone_admin)]$ system kube-upgrade-start v1.24.4
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| from_version | v1.21.8                              |
| to_version   | v1.24.4                              |
| state        | upgrade-started                      |
+--------------+--------------------------------------+

    The upgrade process checks the applied/available updates, the upgrade path, the system health, the installed applications compatibility, and validates that the system is ready for an upgrade.

    Warning

    The command system kube-upgrade-start --force causes the upgrade process to ignore non-management-affecting alarms. Kubernetes cannot be upgraded if there are management-affecting alarms.

  4. Download the Kubernetes images.

    For example:

    ~(keystone_admin)]$ system kube-upgrade-download-images
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8                              |
| to_version   | v1.24.4                              |
| state        | downloading-images                   |
| created_at   | 2023-08-24T02:33:47.049826+00:00     |
| updated_at   | None                                 |
+--------------+--------------------------------------+

  5. Confirm that the download has completed.

    ~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8                              |
| to_version   | v1.24.4                              |
| state        | downloaded-images                    |
| created_at   | 2023-08-24T02:33:47.049826+00:00     |
| updated_at   | 2023-08-24T02:38:16.374677+00:00     |
+--------------+--------------------------------------+

  6. Upgrade Kubernetes networking.

    ~(keystone_admin)]$ system kube-upgrade-networking
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.21.8                              |
| to_version   | v1.24.4                              |
| state        | upgrading-networking                 |
| created_at   | 2023-08-24T02:33:47.049826+00:00     |
| updated_at   | 2023-08-24T02:38:16.374677+00:00     |
+--------------+--------------------------------------+

    The state upgraded-networking will be entered when the networking upgrade has completed.

    ~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8                              |
| to_version   | v1.24.4                              |
| state        | upgraded-networking                  |
| created_at   | 2023-08-24T02:33:47.049826+00:00     |
| updated_at   | 2023-08-24T02:42:40.543522+00:00     |
+--------------+--------------------------------------+

  7. Optional step: Cordon

    The kube-host-cordon command will evict the regular pods from the host. This command will prevent the application from running on intermediate versions.

    Note

    This command will permanently evict the pods which are not in namespaces.

    ~(keystone_admin)]$ system kube-host-cordon controller-0
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8                              |
| to_version   | v1.24.4                              |
| state        | cordon-started                       |
| created_at   | 2023-08-24T02:45:32.257231+00:00     |
| updated_at   | 2023-08-24T02:45:32.257231+00:00     |
+--------------+--------------------------------------+

    ~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8                              |
| to_version   | v1.24.4                              |
| state        | cordon-complete                      |
| created_at   | 2023-08-24T02:45:32.257231+00:00     |
| updated_at   | 2023-08-24T11:47:56.178266+00:00     |
+--------------+--------------------------------------+

    The state cordon-complete will be entered when the host cordon has completed.

  8. Upgrade the control plane on controller-0.

    ~(keystone_admin)]$ system kube-host-upgrade controller-0 control-plane
+-----------------------+-------------------------+
| Property              | Value                   |
+-----------------------+-------------------------+
| control_plane_version | v1.21.8                 |
| hostname              | controller-0            |
| id                    | 1                       |
| kubelet_version       | v1.21.8                 |
| personality           | controller              |
| status                | upgrading-control-plane |
| target_version        | v1.24.4                 |
+-----------------------+-------------------------+

    Check if the control plane version upgrade status is changed to None. This verifies that the control plane has been successfully upgraded to the next version.

    ~(keystone_admin)]$ system kube-host-upgrade-list
+----+---------------+------------+----------------+-----------------------+-----------------+--------------+
| id | hostname      | personality| target_version | control_plane_version | kubelet_version | status       |
+----+---------------+------------+----------------+-----------------------+-----------------+--------------+
| 1  | controller-0  | controller | v1.22.5        | v1.22.5               | v1.21.8         | None         |
+----+---------------+---+--------+----------------+-----------------------+-----------------+--------------+

  9. Upgrade kubelet on controller-0.

    ~(keystone_admin)]$ system kube-host-upgrade controller-0 kubelet
+-----------------------+-------------------------+
| Property              | Value                   |
+-----------------------+-------------------------+
| control_plane_version | v1.22.5                 |
| hostname              | controller-0            |
| id                    | 1                       |
| kubelet_version       | v1.21.8                 |
| personality           | controller              |
| status                | upgrading-kubelet       |
| target_version        | v1.22.5                 |
+-----------------------+-------------------------+

    Check the status of the kubelet upgrade.

    ~(keystone_admin)]$ system kube-host-upgrade-list
+----+---------------+------------+----------------+-----------------------+-----------------+------------------+
| id | hostname      | personality| target_version | control_plane_version | kubelet_version | status           |
+----+---------------+------------+----------------+-----------------------+-----------------+------------------+
| 1  | controller-0  | controller | v1.22.5        | v1.22.5               | v1.22.5         | upgraded-kubelet |
+----+---------------+---+--------+----------------+-----------------------+-----------------+------------------+

    The status upgraded-kubelet will be entered when the kubelet upgrade has completed.

    Repeat steps Cordon and Upgrade the control plane to reach the target Kubernetes version. For example, in this case, we need to repeat steps Cordon and Upgrade the control plane twice for the remaining versions v1.23.1 and v1.24.4.

  10. Optional step: Uncordon

    Skip this step if you did not perform step Upgrade Kubernetes Networking.

    The kube-host-uncordon command will allow the regular pods on the host again.

       ~(keystone_admin)]$ system kube-host-uncordon controller-0
   +--------------+--------------------------------------+
   | Property     | Value                                |
   +--------------+--------------------------------------+
   | uuid         | 065e683a-13a3-4229-b3c7-701f90216a3d |
   | from_version | v1.21.8                              |
   | to_version   | v1.24.4                              |
   | state        | uncordon-started                     |
   | created_at   | 2023-08-24T11:56:56.178266+00:00     |
   | updated_at   | 2023-08-24T11:56:56.178266+00:00     |
   +--------------+--------------------------------------+

.. code-block:: none

    ~(keystone_admin)]$ system kube-upgrade-show
    +--------------+--------------------------------------+
    | Property     | Value                                |
    +--------------+--------------------------------------+
    | uuid         | 065e683a-13a3-4229-b3c7-701f90216a3d |
    | from_version | v1.21.8                              |
    | to_version   | v1.24.4                              |
    | state        | uncordon-complete                    |
    | created_at   | 2023-08-24T11:56:56.178266+00:00     |
    | updated_at   | 2023-08-24T11:58:35.136866+00:00     |
    +--------------+--------------------------------------+

The state **uncordon-complete** will be entered when the host uncordon has
completed.

  11. Complete the Kubernetes upgrade.

    ~(keystone_admin)]$ system kube-upgrade-complete
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8                              |
| to_version   | v1.24.4                              |
| state        | upgrade-complete                     |
| created_at   | 2023-08-24T02:33:47.049826+00:00     |
| updated_at   | 2023-08-24T02:55:18.122620+00:00     |
+--------------+--------------------------------------+

  12. Remove the alarm 900.007 (Kubernetes upgrade in progress) if it is still running after the upgrade.

    ~(keystone_admin)]$ system kube-upgrade-delete

Kubernetes Upgrade Abort

If you want to abort the Kubernetes upgrade after the upgrade has started, run the following command:

~(keystone_admin)]$ system kube-upgrade-abort
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8                              |
| to_version   | v1.22.4                              |
| state        | upgrade-aborting                     |
| created_at   | 2023-06-26T18:44:46.854319+00:00     |
| updated_at   | 2023-08-24T02:55:18.122620+00:00     |
+--------------+--------------------------------------+

To check the status of the abort operation, run the following command:

~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8                              |
| to_version   | v1.24.4                              |
| state        | upgrade-aborted                      |
| created_at   | 2023-08-24T07:10:02.578787+00:00     |
| updated_at   | 2023-08-24T07:24:00.429794+00:00     |
+--------------+--------------------------------------+

Note

  • The upgrade abort operation reverts all the Kubernetes version upgrades and shows the same state the Kubernetes was in before the upgrade started.

  • Once the Kubernetes upgrade is completed, it cannot be aborted.