StarlingX patching

This section describes how to update StarlingX systems using patches.

Introduction

Deployed StarlingX systems can be updated using the StarlingX patching feature, which provides the capability to update StarlingX systems before the next major version is released. It can be used for bug fixes, security vulnerabilities, feature enhancements, etc.

StarlingX patching supports two kinds of patches:

  • In-service patches, which do not require a reboot for those hosts to be patched. Only affected processes need to be restarted.

  • Reboot-required patches, which require a host reboot operation to make the changes take effect. Hosts to be patched should be locked before applying the patch and unlocked after the patch is applied.

This document describes the patching process from the developer’s perspective, rather than being a product user guide. The patching process can be divided into two stages: patch generation and patch applying, which are described in the following sections.

Generate a patch

A StarlingX patch includes one or more rpms that contain some updates to the system. Before starting to generate a patch, you must identify rpms to be installed against the deployed system.

Starting at StarlingX 10.0, the patches will now be built and applied using USM, which means the patch process will be simplified as there is no need to deliver ostree deltas anymore and just debian packages (.deb’s). Legacy patching will no longer be supported.

Identify debs to be Installed

First, figure out the software version of the deployed system, using either the StarlingX Horizon GUI or using the CLI.

Use Horizon GUI

  1. Access the StarlingX Horizon GUI by entering the OAM floating IP address in your browser: https://<oam-floating-ip-address>:8443

    Note

    If https is not enabled, the URL is http://<oam-floating-ip-address>:8080.

    Discover your OAM floating IP address with the system oam-show command.

  2. Log in to Horizon with an admin/<sysadmin-password>.

  3. Identify the software version of the deployed system by clicking Admin > Platform > System Configuration and clicking the Systems tab in the left-hand pane.

Use StarlingX CLI

  1. In the CLI, use the system show command on the active controller node to show the software version. In the example below, the software version is “19.09”.

    controller-0:~$ . /etc/platform/openrc
    [sysadmin@controller-0 ~(keystone_admin)]$ system show
    +----------------------+--------------------------------------+
    | Property             | Value                                |
    +----------------------+--------------------------------------+
    | contact              | None                                 |
    | created_at           | 2019-10-14T03:10:50.862114+00:00     |
    | description          | None                                 |
    | https_enabled        | False                                |
    | location             | None                                 |
    | name                 | 608dfe48-9a05-4b21-afc1-ea122574caa7 |
    | region_name          | RegionOne                            |
    | sdn_enabled          | False                                |
    | security_feature     | spectre_meltdown_v1                  |
    | service_project_name | services                             |
    | software_version     | 19.09                                |
    | system_mode          | duplex                               |
    | system_type          | All-in-one                           |
    | timezone             | UTC                                  |
    | updated_at           | 2019-10-14T03:12:41.983029+00:00     |
    | uuid                 | 2639ad15-08a7-4f1b-a372-f927a5e4ab31 |
    | vswitch_type         | none                                 |
    +----------------------+--------------------------------------+
    

After you figure out the software version, check the latest StarlingX build to identify updated debs against that version. Select one or more debs for patch generation and continue to the next section.

Build Patches

After you determine the debs to be installed, set up an environment for patch build. As a StarlingX developer, the easiest way to set up this type of environment is to use a StarlingX build container, with a few changes.

Prerequisites

  1. Set up a StarlingX build container following the build guide.

  2. Download StarlingX source code.

  3. Populate your build using build-pkgs --reuse or build-pkgs --all. Using the reuse flag should be faster if you already have the running env.

  4. Make your changes, usually by cherry picking some commit.

  5. Apply the changes to the necessary files and than build the packages related to these files with:

    $  build-pkgs -p <package name>
    
  6. Build all necessary packages. To find out the package name, look for a debian meta_data.yaml file.

Procedure

Set up the patch build environment using the procedure below.

Note

This environment is intended for developers working on this feature and is not intended to be used for a StarlingX-based product.

Note

Patch requirements are automatically installed when containers are created.

  1. Create your patch recipe. The recipe file is used as input and it has all the patch metadata. Packages that will be included in the patch need to be added to the packages object (“stx” are packages we build, “binary” are packages we download).

    <patch_recipe>
            <!-- Software Version -->
            <sw_version>1.0.0</sw_version>
            <!-- Component -->
            <component>starlingx</component>
            <!-- Summary: Short text to give a summary about the patch -->
            <summary>sample patch test</summary>
            <!-- Description: Patch description. Usually it has a list of fixes -->
            <description>Sample description</description>
            <!-- Install Instructions: Any instructions to be done before the patch installation -->
            <install_instructions>Sample instructions</install_instructions>
            <!-- Warnings: Any warnings that this patch can trigger -->
            <warnings>Sample warning</warnings>
            <!-- Reboot required: Y (Yes) or N (No) for in service patch -->
            <reboot_required>Y</reboot_required>
            <!-- Unremovable: Y (Yes)/ N (No), specifices if the patch can be removed -->
            <unremovable>N</unremovable>
            <!-- Patch Status: Supported values are DEV (development) and REL (released) -->
            <status>DEV</status>
            <!-- Requires: List of patches that are required by this patch -->
            <requires>
                    <!--
                    <id>PATCH_XYZ_01</id>
                    <id>PATCH_XYZ_02</id>
                    -->
            </requires>
            <semantics></semantics>
            <!--
                    Pre and Post install hook scripts that are executed before/after patch installation.
                    Leave if blank if no scripts are required.
            -->
            <pre_install>scripts/pre-install.sh</pre_install>
            <post_install>scripts/post-install.sh</post_install>
            <!-- List Packages to be included in the patch -->
            <stx_packages>
                    <!-- Starlingx packages list -->
                    <package>sysvinv</package>
                    <package>linux</package>
                    <package>linux-rt</package>
            </stx_packages>
            <!-- Binary packages list to be included in the patch (Packages that we download from 3rd party sources) -->
            <binary_packages>
                    <!-- 3rd party packages list -->
                    <package>curl</package>
            </binary_packages>
    </patch_recipe>
    

Patch build

  1. Create the patch.

    $ patch-builder --recipe EXAMPLES/patch-recipe-sample.xml
     ...
    2024-06-12 13:12:49,566 - patch_builder - DEBUG: Copying pre-install script: /localdisk/designer/USER/master-env/cgcs-root/build-tools/stx/patch/scripts/pre-install.sh
    2024-06-12 13:12:49,567 - patch_builder - INFO: Renaming /localdisk/designer/USER/master-env/cgcs-root/build-tools/stx/patch/scripts/pre-install.sh to pre-install.sh
    2024-06-12 13:12:49,567 - patch_builder - DEBUG: Copying post-install script: /localdisk/designer/USER/master-env/cgcs-root/build-tools/stx/patch/scripts/post-install.sh
    2024-06-12 13:12:49,567 - patch_builder - INFO: Renaming /localdisk/designer/USER/master-env/cgcs-root/build-tools/stx/patch/scripts/post-install.sh to post-install.sh
    2024-06-12 13:12:49,567 - patch_builder - DEBUG: Generating metadata file
    2024-06-12 13:12:49,569 - patch_builder - DEBUG: Generating signature for patch files ['metadata.tar', 'software.tar', 'pre-install.sh', 'post-install.sh']
    2024-06-12 13:12:49,888 - patch_builder - INFO: Formal signing status: False
    2024-06-12 13:12:49,888 - patch_builder - INFO: Saving file software.tar
    2024-06-12 13:12:51,441 - patch_builder - INFO: Saving file post-install.sh
    2024-06-12 13:12:51,441 - patch_builder - INFO: Saving file metadata.tar
    2024-06-12 13:12:51,442 - patch_builder - INFO: Saving file signature
    2024-06-12 13:12:51,442 - patch_builder - INFO: Saving file signature.v2
    2024-06-12 13:12:51,442 - patch_builder - INFO: Saving file pre-install.sh
    2024-06-12 13:12:51,442 - patch_builder - INFO: Patch file created /localdisk/loadbuild/USER/master-env/patch_output/starlingx-1.0.0.patch
    
  2. Verify patch content.

    $ tar -tvf /localdisk/loadbuild/USER/master-env/patch_output/starlingx-1.0.0.patch
                    USER/cgts 74772480 2024-06-12 13:12 software.tar
                    USER/cgts      440 2024-06-12 13:12 post-install.sh
                    USER/cgts    10240 2024-06-12 13:12 metadata.tar
                    USER/cgts       32 2024-06-12 13:12 signature
                    USER/cgts      256 2024-06-12 13:12 signature.v2
                    USER/cgts      433 2024-06-12 13:12 pre-install.sh
    
  3. Verify patch metadata.

    $ tar xf /localdisk/loadbuild/USER/stx-master/patch_output/sample_patch.patch -O metadata.tar | tar x -O
    <?xml version="1.0" ?>
    <patch>
      <id>starlingx-1.0.0</id>
      <sw_version>1.0.0</sw_version>
      <component>starlingx</component>
      <summary>sample patch test</summary>
      <install_instructions>Sample instructions</install_instructions>
      <warnings>Sample warning</warnings>
      <status>DEV</status>
      <unremovable>N</unremovable>
      <reboot_required>Y</reboot_required>
      <semantics/>
      <requires/>
      <pre_install>pre-install.sh</pre_install>
      <post_install>post-install.sh</post_install>
      <packages>
            <deb>curl_7.74.0-1.3+deb11u11_amd64.deb</deb>
            <deb>linux-compiler-gcc-10-x86_5.10.205-1.stx.79_amd64.deb</deb>
            <deb>linux-headers-5.10.0-6-amd64_5.10.205-1.stx.84_amd64.deb</deb>
            <deb>linux-headers-5.10.0-6-common_5.10.205-1.stx.84_all.deb</deb>
            <deb>linux-image-5.10.0-6-amd64-unsigned_5.10.205-1.stx.84_amd64.deb</deb>
            <deb>linux-kbuild-5.10_5.10.205-1.stx.84_amd64.deb</deb>
            <deb>linux-libc-dev_5.10.205-1.stx.84_amd64.deb</deb>
            <deb>linux-perf-5.10_5.10.205-1.stx.84_amd64.deb</deb>
            <deb>linux-rt-headers-5.10.0-6-rt-amd64_5.10.205-1.stx.79_amd64.deb</deb>
            <deb>linux-rt-headers-5.10.0-6-rt-common_5.10.205-1.stx.79_all.deb</deb>
            <deb>linux-rt-image-5.10.0-6-rt-amd64-unsigned_5.10.205-1.stx.79_amd64.deb</deb>
            <deb>linux-rt-kbuild-5.10_5.10.205-1.stx.79_amd64.deb</deb>
      </packages>
    </patch>
    
  4. Verify patch content (software.tar).

    $ tar xf /localdisk/loadbuild/USER/stx-master/patch_output/sample_patch.patch -O software.tar | tar t
    
               linux-kbuild-5.10_5.10.205-1.stx.84_amd64.deb
                    linux-compiler-gcc-10-x86_5.10.205-1.stx.79_amd64.deb
                    linux-rt-image-5.10.0-6-rt-amd64-unsigned_5.10.205-1.stx.79_amd64.deb
                    linux-rt-kbuild-5.10_5.10.205-1.stx.79_amd64.deb
                    linux-perf-5.10_5.10.205-1.stx.84_amd64.deb
                    linux-rt-headers-5.10.0-6-rt-common_5.10.205-1.stx.79_all.deb
                    linux-rt-headers-5.10.0-6-rt-amd64_5.10.205-1.stx.79_amd64.deb
                    linux-headers-5.10.0-6-amd64_5.10.205-1.stx.84_amd64.deb
                    linux-image-5.10.0-6-amd64-unsigned_5.10.205-1.stx.84_amd64.deb
                    linux-libc-dev_5.10.205-1.stx.84_amd64.deb
                    curl_7.74.0-1.3+deb11u11_amd64.deb
                    linux-headers-5.10.0-6-common_5.10.205-1.stx.84_all.deb
    

Apply a patch

Once patches are generated, you can manually apply them to an applicable StarlingX system. Both the StarlingX Horizon GUI and the CLI support the patch applying operation. This example uses the CLI to show more detail.

The life cycle of a patch consists of the following states:

  • Available: A patch in the Available state means it has been put into the patch storage area, but it has not been put into the software update repository and installed on any host yet.

  • Partial-Apply: A patch in the Partial-Apply state means the patching process has been triggered by the sw-patch apply command, but the patch has not been installed on all hosts that require it. It may have been installed on some hosts, but not all.

  • Applied: A patch in the Applied state means it has been installed on all hosts that require it.

  • Partial-Remove: A patch in the Partial-Remove state means the removing process has been triggered by the sw-patch remove command, but the patch has not been removed from all hosts that installed it. It may have been removed from some hosts, but not all.

Before applying a patch, you must upload it to the file system of the active controller. Uploading can be performed in many ways. Here is an example using scp.

scp <patch-id>.patch sysadmin@<oam_ip>:~/

The StarlingX patching system provides the client tool sw-patch, which can perform all types of patching operations. The many operations supported by sw-patch are listed below.

controller-0:~$ sw-patch --help
usage: sw-patch [--debug]
                  <subcommand> ...

Subcommands:
    upload:         Upload one or more patches to the patching system.

    upload-dir:     Upload patches from one or more directories to the
                    patching system.

    apply:          Apply one or more patches. This adds the specified
                    patches to the repository, making the update(s)
                    available to the hosts in the system. Use --all to
                    apply all available patches.
                    Patches are specified as a space-separated list of
                    patch IDs.

    remove:         Remove one or more patches. This removes the specified
                    patches from the repository.
                    Patches are specified as a space-separated list of
                    patch IDs.

    delete:         Delete one or more patches from the patching system.
                    Patches are specified as a space-separated list of
                    patch IDs.

    query:          Query system patches. Optionally, specify 'query
                    applied' to query only those patches that are applied,
                    or 'query available' to query those that are not.

    show:           Show details for specified patches.

    what-requires:  List patches that require the specified patches.

    query-hosts:    Query patch states for hosts in the system.

    host-install:   Trigger patch install/remove on specified host. To
                    force install on unlocked node, use the --force option.

    host-install-async: Trigger patch install/remove on specified host. To
                    force install on unlocked node, use the --force option.
                    Note: This command returns immediately upon dispatching
                    installation request.

    install-local:  Trigger patch install/remove on the local host. This
                    command can only be used for patch installation prior
                    to initial configuration.

    drop-host:      Drop specified host from table.

    query-dependencies: List dependencies for specified patch. Use
                    --recursive for recursive query.

    is-applied:     Query Applied state for list of patches. Returns True
                    if all are Applied, False otherwise.

    report-app-dependencies: Report application patch dependencies,
                    specifying application name with --app option, plus a
                    list of patches. Reported dependencies can be dropped
                    by specifying app with no patch list.

    query-app-dependencies: Display set of reported application patch
                    dependencies.

    commit:         Commit patches to free disk space. WARNING: This
                    action is irreversible!

    --os-region-name: Send the request to a specified region

The following example demonstrates how to apply a patch to the system using sw-patch commands. This example applies an in-service patch to be installed on all hosts in the system, and the StarlingX system is in a 2+2+2 configuration.

  1. Upload the patch to the patching storage area using the sw-patch upload command.

    controller-0:~$ sudo sw-patch upload 001.patch
    001 is now available
    

    Check the status of the patch with the sw-patch query command. The “Patch State” value indicates that the patch is available in the system.

    controller-0:~$ sudo sw-patch query
    Patch ID  RR  Release  Patch State
    ========  ==  =======  ===========
    001       N    19.09    Available
    

    Check the status of all hosts in the cluster with the sw-patch query-hosts command.

    controller-0:/$ sudo sw-patch query-hosts
    Hostname      IP Address      Patch Current  Reboot Required  Release State
    ============  ==============  =============  ===============  ======  =====
    worker-0      192.178.204.7        Yes             No          19.09   idle
    worker-1      192.178.204.9        Yes             No          19.09   idle
    controller-0  192.178.204.3        Yes             No          19.09   idle
    controller-1  192.178.204.4        Yes             No          19.09   idle
    storage-0     192.178.204.12       Yes             No          19.09   idle
    storage-1     192.178.204.11       Yes             No          19.09   idle
    

    The “Patch Current” value indicates whether or not there are patches pending for installation or removal on the host. “Yes” means no patch pending, and “No” means there is at least one patch pending.

  2. Once the patch is in the Available state, trigger patch applying using the sw-patch apply command.

    controller-0:/$ sudo sw-patch apply 001
    001 is now in the repo
    

    Check the status of the patch and the hosts again. As shown below, the patch is in the Partial-Apply state because it has not been installed on any host yet. The “Patch Current” value for all hosts are “No”.

    controller-0:~$ sudo sw-patch query
    Patch ID  RR  Release   Patch State
    ========  ==  =======  =============
    001       N    19.09   Partial-Apply
    
    controller-0:~$ sudo sw-patch query-hosts
    Hostname      IP Address      Patch Current  Reboot Required  Release State
    ============  ==============  =============  ===============  ======  =====
    worker-0      192.178.204.7        No              No          19.09   idle
    worker-1      192.178.204.9        No              No          19.09   idle
    controller-0  192.178.204.3        No              No          19.09   idle
    controller-1  192.178.204.4        No              No          19.09   idle
    storage-0     192.178.204.12       No              No          19.09   idle
    storage-1     192.178.204.11       No              No          19.09   idle
    
  3. Install the patch on each host. In this case, it is an in-service patch, so you don’t need to lock hosts. (If the patch is a reboot-required patch, each node must be locked before the patch can be installed.)

    controller-0:~$ sudo sw-patch host-install controller-0
    ...
    Installation was successful.
    

    Check the host status again. The “Patch Current” value of controller-0 has changed to “Yes” and the other “Patch Current” values are still “No”, which is expected.

    controller-0:~$ sudo sw-patch query-hosts
    Hostname      IP Address    Patch Current    Reboot Required  Release State
    ============  ==============  =============  ===============  ======  =====
    worker-0      192.178.204.7        No              No          19.09   idle
    worker-1      192.178.204.9        No              No          19.09   idle
    controller-0  192.178.204.3        Yes             No          19.09   idle
    controller-1  192.178.204.4        No              No          19.09   idle
    storage-0     192.178.204.12       No              No          19.09   idle
    storage-1     192.178.204.11       No              No          19.09   idle
    

    To install the patch on all hosts, execute the command against each host.

    controller-0:~$ sudo sw-patch host-install controller-1
    ....
    Installation was successful.
    controller-0:~$ sudo sw-patch host-install worker-0
    ....
    Installation was successful.
    controller-0:~$ sudo sw-patch host-install worker-1
    ....
    Installation was successful.
    controller-0:~$ sudo sw-patch host-install storage-0
    ...
    Installation was successful.
    controller-0:~$ sudo sw-patch host-install storage-1
    ...
    Installation was successful.
    

    Check the status of the patch and the hosts. The patch has been installed on all hosts as shown below. The “Patch Current” value of the hosts changed to “Yes” and the “Patch State” value changed to “Applied”.

    controller-0:~$ sudo sw-patch query
    Patch ID  RR  Release  Patch State
    ========  ==  =======  ===========
    001       N    19.09     Applied
    
    controller-0:~$ sudo sw-patch query-hosts
    Hostname      IP Address      Patch Current Reboot Required  Release  State
    ============  ==============  ============  ===============  =======  =====
    worker-0      192.178.204.7        Yes             No          19.09   idle
    worker-1      192.178.204.9        Yes             No          19.09   idle
    controller-0  192.178.204.3        Yes             No          19.09   idle
    controller-1  192.178.204.4        Yes             No          19.09   idle
    storage-0     192.178.204.12       Yes             No          19.09   idle
    storage-1     192.178.204.11       Yes             No          19.09   idle
    

    This output confirms that the patch 001.patch has been applied to the whole system.

StarlingX patching also supports patch removal, using the sw-patch remove and sw-patch host-install commands. The procedure is similar to that of patch applying and is not described here.

Patch orchestration

In the example shown above, the hosts in the cluster were updated one by one. For a case where the cluster size is very large, the updating process takes a long time, and the situation takes longer for reboot-required patches. The updating operation becomes very inefficient and can be a burden to the cluster administrator. StarlingX has an advanced feature called patch orchestration. It allows the whole system to be patched with a few operations, which simplifies update tasks for the cluster administrator. The operations can be performed using the CLI, the Horizon GUI, or the VIM RESTful API.

  1. Using the StarlingX CLI client tool sw-manager, you can create and apply a patch strategy, which then updates the whole system. The supported options are listed below.

    controller-0:~$ sw-manager patch-strategy -h
    usage: sw-manager patch-strategy [-h]  ...
    
    optional arguments:
      -h, --help  show this help message and exit
    
    Software Patch Commands:
    
        create    Create a strategy
        delete    Delete a strategy
        apply     Apply a strategy
        abort     Abort a strategy
        show      Show a strategy
    
    controller-0:~$ sw-manager patch-strategy create -h
    usage: sw-manager patch-strategy create [-h]
                            [--controller-apply-type {serial,ignore}]
                            [--storage-apply-type {serial,parallel,ignore}]
                            [--worker-apply-type {serial,parallel,ignore}]
                            [--max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10,
                            11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,
                            28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,
                            45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,
                            62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,
                            79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,
                            96,97,98,99,100}]
                            [--instance-action {migrate,stop-start}]
                            [--alarm-restrictions {strict,relaxed}]
    
    optional arguments:
      -h, --help            show this help message and exit
      --controller-apply-type {serial,ignore}
                            defaults to serial
      --storage-apply-type {serial,parallel,ignore}
                            defaults to serial
      --worker-apply-type {serial,parallel,ignore}
                            defaults to serial
      --max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,
                    17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,
                    37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,
                    57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,
                    77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,
                    97,98,99,100}
                            maximum worker hosts to patch in parallel
      --instance-action {migrate,stop-start}
                            defaults to stop-start
      --alarm-restrictions {strict,relaxed}
                            defaults to strict
    
  2. Using the Horizon GUI, click Admin > Platform > Software Management and click the Patch Orchestration tab.

  3. Using the VIM API, http://<oam_ip>:4545

    Method

    URI

    Description

    Post

    /api/orchestration/sw-update/strategy

    Create a patch strategy

    Delete

    /api/orchestration/sw-update/strategy

    Delete current patch strategy

    Get

    /api/orchestration/sw-update/strategy

    Get detailed information of current patch strategy

    Post

    /api/orchestration/sw-update/strategy/ actions

    Apply or abort a patch strategy

    The system must be in good condition to use patch orchestration. For example:

    • All hosts must be in the state of unlocked-enabled-available.

    • The system is clear of alarms.

    • Enough spare worker nodes for VM migration.

Current status

  • The whole patching source code is already in StarlingX repositories, across several projects, like “update” and “nfv”.

  • Patch generation and manual patch application have been roughly verified for both in-service patches and reboot-required patches. They are working.

  • Patch orchestration has not been verified yet.