StarlingX patching¶
This section describes how to update StarlingX systems using patches.
Introduction¶
Deployed StarlingX systems can be updated using the StarlingX patching feature, which provides the capability to update StarlingX systems before the next major version is released. It can be used for bug fixes, security vulnerabilities, feature enhancements, etc.
StarlingX patching supports two kinds of patches:
In-service patches, which do not require a reboot for those hosts to be patched. Only affected processes need to be restarted.
Reboot-required patches, which require a host reboot operation to make the changes take effect. Hosts to be patched should be locked before applying the patch and unlocked after the patch is applied.
This document describes the patching process from the developer’s perspective, rather than being a product user guide. The patching process can be divided into two stages: patch generation and patch applying, which are described in the following sections.
Generate a patch¶
A StarlingX patch includes one or more rpms that contain some updates to the system. Before starting to generate a patch, you must identify rpms to be installed against the deployed system.
Starting at StarlingX 10.0, the patches will now be built and applied using USM, which means the patch process will be simplified as there is no need to deliver ostree deltas anymore and just debian packages (.deb’s). Legacy patching will no longer be supported.
Identify debs to be Installed¶
First, figure out the software version of the deployed system, using either the StarlingX Horizon GUI or using the CLI.
Use Horizon GUI¶
Access the StarlingX Horizon GUI by entering the OAM floating IP address in your browser: https://<oam-floating-ip-address>:8443
Discover your OAM floating IP address with the system oam-show command.
Log in to Horizon with an
admin/<sysadmin-password>.Identify the software version of the deployed system by clicking and clicking the Systems tab in the left-hand pane.
Use StarlingX CLI¶
In the CLI, use the system show command on the active controller node to show the software version. In the example below, the software version is “19.09”.
controller-0:~$ . /etc/platform/openrc [sysadmin@controller-0 ~(keystone_admin)]$ system show +----------------------+--------------------------------------+ | Property | Value | +----------------------+--------------------------------------+ | contact | None | | created_at | 2019-10-14T03:10:50.862114+00:00 | | description | None | | https_enabled | False | | location | None | | name | 608dfe48-9a05-4b21-afc1-ea122574caa7 | | region_name | RegionOne | | sdn_enabled | False | | security_feature | spectre_meltdown_v1 | | service_project_name | services | | software_version | 19.09 | | system_mode | duplex | | system_type | All-in-one | | timezone | UTC | | updated_at | 2019-10-14T03:12:41.983029+00:00 | | uuid | 2639ad15-08a7-4f1b-a372-f927a5e4ab31 | | vswitch_type | none | +----------------------+--------------------------------------+
After you figure out the software version, check the latest StarlingX build to identify updated debs against that version. Select one or more debs for patch generation and continue to the next section.
Build Patches¶
After you determine the debs to be installed, set up an environment for patch build. As a StarlingX developer, the easiest way to set up this type of environment is to use a StarlingX build container, with a few changes.
Prerequisites
Set up a StarlingX build container following the build guide.
Download StarlingX source code.
Populate your build using
build-pkgs --reuseorbuild-pkgs --all. Using the reuse flag should be faster if you already have the running env.Make your changes, usually by cherry picking some commit.
Apply the changes to the necessary files and than build the packages related to these files with:
$ build-pkgs -p <package name>
Build all necessary packages. To find out the package name, look for a debian
meta_data.yamlfile.
Procedure
Set up the patch build environment using the procedure below.
Note
This environment is intended for developers working on this feature and is not intended to be used for a StarlingX-based product.
Note
Patch requirements are automatically installed when containers are created.
Create your patch recipe. The recipe file is used as input and it has all the patch metadata. Packages that will be included in the patch need to be added to the packages object (“stx” are packages we build, “binary” are packages we download).
<patch_recipe> <!-- Software Version --> <sw_version>1.0.0</sw_version> <!-- Component --> <component>starlingx</component> <!-- Summary: Short text to give a summary about the patch --> <summary>sample patch test</summary> <!-- Description: Patch description. Usually it has a list of fixes --> <description>Sample description</description> <!-- Install Instructions: Any instructions to be done before the patch installation --> <install_instructions>Sample instructions</install_instructions> <!-- Warnings: Any warnings that this patch can trigger --> <warnings>Sample warning</warnings> <!-- Reboot required: Y (Yes) or N (No) for in service patch --> <reboot_required>Y</reboot_required> <!-- Unremovable: Y (Yes)/ N (No), specifices if the patch can be removed --> <unremovable>N</unremovable> <!-- Patch Status: Supported values are DEV (development) and REL (released) --> <status>DEV</status> <!-- Requires: List of patches that are required by this patch --> <requires> <!-- <id>PATCH_XYZ_01</id> <id>PATCH_XYZ_02</id> --> </requires> <semantics></semantics> <!-- Pre and Post install hook scripts that are executed before/after patch installation. Leave if blank if no scripts are required. --> <pre_install>scripts/pre-install.sh</pre_install> <post_install>scripts/post-install.sh</post_install> <!-- List Packages to be included in the patch --> <stx_packages> <!-- Starlingx packages list --> <package>sysvinv</package> <package>linux</package> <package>linux-rt</package> </stx_packages> <!-- Binary packages list to be included in the patch (Packages that we download from 3rd party sources) --> <binary_packages> <!-- 3rd party packages list --> <package>curl</package> </binary_packages> </patch_recipe>
Patch build
Create the patch.
$ patch-builder --recipe EXAMPLES/patch-recipe-sample.xml ... 2024-06-12 13:12:49,566 - patch_builder - DEBUG: Copying pre-install script: /localdisk/designer/USER/master-env/cgcs-root/build-tools/stx/patch/scripts/pre-install.sh 2024-06-12 13:12:49,567 - patch_builder - INFO: Renaming /localdisk/designer/USER/master-env/cgcs-root/build-tools/stx/patch/scripts/pre-install.sh to pre-install.sh 2024-06-12 13:12:49,567 - patch_builder - DEBUG: Copying post-install script: /localdisk/designer/USER/master-env/cgcs-root/build-tools/stx/patch/scripts/post-install.sh 2024-06-12 13:12:49,567 - patch_builder - INFO: Renaming /localdisk/designer/USER/master-env/cgcs-root/build-tools/stx/patch/scripts/post-install.sh to post-install.sh 2024-06-12 13:12:49,567 - patch_builder - DEBUG: Generating metadata file 2024-06-12 13:12:49,569 - patch_builder - DEBUG: Generating signature for patch files ['metadata.tar', 'software.tar', 'pre-install.sh', 'post-install.sh'] 2024-06-12 13:12:49,888 - patch_builder - INFO: Formal signing status: False 2024-06-12 13:12:49,888 - patch_builder - INFO: Saving file software.tar 2024-06-12 13:12:51,441 - patch_builder - INFO: Saving file post-install.sh 2024-06-12 13:12:51,441 - patch_builder - INFO: Saving file metadata.tar 2024-06-12 13:12:51,442 - patch_builder - INFO: Saving file signature 2024-06-12 13:12:51,442 - patch_builder - INFO: Saving file signature.v2 2024-06-12 13:12:51,442 - patch_builder - INFO: Saving file pre-install.sh 2024-06-12 13:12:51,442 - patch_builder - INFO: Patch file created /localdisk/loadbuild/USER/master-env/patch_output/starlingx-1.0.0.patch
Verify patch content.
$ tar -tvf /localdisk/loadbuild/USER/master-env/patch_output/starlingx-1.0.0.patch USER/cgts 74772480 2024-06-12 13:12 software.tar USER/cgts 440 2024-06-12 13:12 post-install.sh USER/cgts 10240 2024-06-12 13:12 metadata.tar USER/cgts 32 2024-06-12 13:12 signature USER/cgts 256 2024-06-12 13:12 signature.v2 USER/cgts 433 2024-06-12 13:12 pre-install.shVerify patch metadata.
$ tar xf /localdisk/loadbuild/USER/stx-master/patch_output/sample_patch.patch -O metadata.tar | tar x -O <?xml version="1.0" ?> <patch> <id>starlingx-1.0.0</id> <sw_version>1.0.0</sw_version> <component>starlingx</component> <summary>sample patch test</summary> <install_instructions>Sample instructions</install_instructions> <warnings>Sample warning</warnings> <status>DEV</status> <unremovable>N</unremovable> <reboot_required>Y</reboot_required> <semantics/> <requires/> <pre_install>pre-install.sh</pre_install> <post_install>post-install.sh</post_install> <packages> <deb>curl_7.74.0-1.3+deb11u11_amd64.deb</deb> <deb>linux-compiler-gcc-10-x86_5.10.205-1.stx.79_amd64.deb</deb> <deb>linux-headers-5.10.0-6-amd64_5.10.205-1.stx.84_amd64.deb</deb> <deb>linux-headers-5.10.0-6-common_5.10.205-1.stx.84_all.deb</deb> <deb>linux-image-5.10.0-6-amd64-unsigned_5.10.205-1.stx.84_amd64.deb</deb> <deb>linux-kbuild-5.10_5.10.205-1.stx.84_amd64.deb</deb> <deb>linux-libc-dev_5.10.205-1.stx.84_amd64.deb</deb> <deb>linux-perf-5.10_5.10.205-1.stx.84_amd64.deb</deb> <deb>linux-rt-headers-5.10.0-6-rt-amd64_5.10.205-1.stx.79_amd64.deb</deb> <deb>linux-rt-headers-5.10.0-6-rt-common_5.10.205-1.stx.79_all.deb</deb> <deb>linux-rt-image-5.10.0-6-rt-amd64-unsigned_5.10.205-1.stx.79_amd64.deb</deb> <deb>linux-rt-kbuild-5.10_5.10.205-1.stx.79_amd64.deb</deb> </packages> </patch>Verify patch content (
software.tar).$ tar xf /localdisk/loadbuild/USER/stx-master/patch_output/sample_patch.patch -O software.tar | tar t linux-kbuild-5.10_5.10.205-1.stx.84_amd64.deb linux-compiler-gcc-10-x86_5.10.205-1.stx.79_amd64.deb linux-rt-image-5.10.0-6-rt-amd64-unsigned_5.10.205-1.stx.79_amd64.deb linux-rt-kbuild-5.10_5.10.205-1.stx.79_amd64.deb linux-perf-5.10_5.10.205-1.stx.84_amd64.deb linux-rt-headers-5.10.0-6-rt-common_5.10.205-1.stx.79_all.deb linux-rt-headers-5.10.0-6-rt-amd64_5.10.205-1.stx.79_amd64.deb linux-headers-5.10.0-6-amd64_5.10.205-1.stx.84_amd64.deb linux-image-5.10.0-6-amd64-unsigned_5.10.205-1.stx.84_amd64.deb linux-libc-dev_5.10.205-1.stx.84_amd64.deb curl_7.74.0-1.3+deb11u11_amd64.deb linux-headers-5.10.0-6-common_5.10.205-1.stx.84_all.deb
Apply a patch¶
Once patches are generated, you can manually apply them to an applicable StarlingX system. Both the StarlingX Horizon GUI and the CLI support the patch applying operation. This example uses the CLI to show more detail.
The life cycle of a patch consists of the following states:
Available: A patch in the Available state means it has been put into the patch storage area, but it has not been put into the software update repository and installed on any host yet.
Partial-Apply: A patch in the Partial-Apply state means the patching process has been triggered by the sw-patch apply command, but the patch has not been installed on all hosts that require it. It may have been installed on some hosts, but not all.
Applied: A patch in the Applied state means it has been installed on all hosts that require it.
Partial-Remove: A patch in the Partial-Remove state means the removing process has been triggered by the sw-patch remove command, but the patch has not been removed from all hosts that installed it. It may have been removed from some hosts, but not all.
Before applying a patch, you must upload it to the file system of the active
controller. Uploading can be performed in many ways. Here is an example using
scp.
scp <patch-id>.patch sysadmin@<oam_ip>:~/
The StarlingX patching system provides the client tool sw-patch, which can
perform all types of patching operations. The many operations supported by
sw-patch are listed below.
controller-0:~$ sw-patch --help
usage: sw-patch [--debug]
<subcommand> ...
Subcommands:
upload: Upload one or more patches to the patching system.
upload-dir: Upload patches from one or more directories to the
patching system.
apply: Apply one or more patches. This adds the specified
patches to the repository, making the update(s)
available to the hosts in the system. Use --all to
apply all available patches.
Patches are specified as a space-separated list of
patch IDs.
remove: Remove one or more patches. This removes the specified
patches from the repository.
Patches are specified as a space-separated list of
patch IDs.
delete: Delete one or more patches from the patching system.
Patches are specified as a space-separated list of
patch IDs.
query: Query system patches. Optionally, specify 'query
applied' to query only those patches that are applied,
or 'query available' to query those that are not.
show: Show details for specified patches.
what-requires: List patches that require the specified patches.
query-hosts: Query patch states for hosts in the system.
host-install: Trigger patch install/remove on specified host. To
force install on unlocked node, use the --force option.
host-install-async: Trigger patch install/remove on specified host. To
force install on unlocked node, use the --force option.
Note: This command returns immediately upon dispatching
installation request.
install-local: Trigger patch install/remove on the local host. This
command can only be used for patch installation prior
to initial configuration.
drop-host: Drop specified host from table.
query-dependencies: List dependencies for specified patch. Use
--recursive for recursive query.
is-applied: Query Applied state for list of patches. Returns True
if all are Applied, False otherwise.
report-app-dependencies: Report application patch dependencies,
specifying application name with --app option, plus a
list of patches. Reported dependencies can be dropped
by specifying app with no patch list.
query-app-dependencies: Display set of reported application patch
dependencies.
commit: Commit patches to free disk space. WARNING: This
action is irreversible!
--os-region-name: Send the request to a specified region
The following example demonstrates how to apply a patch to the system using
sw-patch commands. This example applies an in-service patch to be
installed on all hosts in the system, and the StarlingX system is in a 2+2+2
configuration.
Upload the patch to the patching storage area using the sw-patch upload command.
controller-0:~$ sudo sw-patch upload 001.patch 001 is now available
Check the status of the patch with the sw-patch query command. The “Patch State” value indicates that the patch is available in the system.
controller-0:~$ sudo sw-patch query Patch ID RR Release Patch State ======== == ======= =========== 001 N 19.09 Available
Check the status of all hosts in the cluster with the sw-patch query-hosts command.
controller-0:/$ sudo sw-patch query-hosts Hostname IP Address Patch Current Reboot Required Release State ============ ============== ============= =============== ====== ===== worker-0 192.178.204.7 Yes No 19.09 idle worker-1 192.178.204.9 Yes No 19.09 idle controller-0 192.178.204.3 Yes No 19.09 idle controller-1 192.178.204.4 Yes No 19.09 idle storage-0 192.178.204.12 Yes No 19.09 idle storage-1 192.178.204.11 Yes No 19.09 idle
The “Patch Current” value indicates whether or not there are patches pending for installation or removal on the host. “Yes” means no patch pending, and “No” means there is at least one patch pending.
Once the patch is in the Available state, trigger patch applying using the sw-patch apply command.
controller-0:/$ sudo sw-patch apply 001 001 is now in the repo
Check the status of the patch and the hosts again. As shown below, the patch is in the Partial-Apply state because it has not been installed on any host yet. The “Patch Current” value for all hosts are “No”.
controller-0:~$ sudo sw-patch query Patch ID RR Release Patch State ======== == ======= ============= 001 N 19.09 Partial-Apply
controller-0:~$ sudo sw-patch query-hosts Hostname IP Address Patch Current Reboot Required Release State ============ ============== ============= =============== ====== ===== worker-0 192.178.204.7 No No 19.09 idle worker-1 192.178.204.9 No No 19.09 idle controller-0 192.178.204.3 No No 19.09 idle controller-1 192.178.204.4 No No 19.09 idle storage-0 192.178.204.12 No No 19.09 idle storage-1 192.178.204.11 No No 19.09 idle
Install the patch on each host. In this case, it is an in-service patch, so you don’t need to lock hosts. (If the patch is a reboot-required patch, each node must be locked before the patch can be installed.)
controller-0:~$ sudo sw-patch host-install controller-0 ... Installation was successful.
Check the host status again. The “Patch Current” value of controller-0 has changed to “Yes” and the other “Patch Current” values are still “No”, which is expected.
controller-0:~$ sudo sw-patch query-hosts Hostname IP Address Patch Current Reboot Required Release State ============ ============== ============= =============== ====== ===== worker-0 192.178.204.7 No No 19.09 idle worker-1 192.178.204.9 No No 19.09 idle controller-0 192.178.204.3 Yes No 19.09 idle controller-1 192.178.204.4 No No 19.09 idle storage-0 192.178.204.12 No No 19.09 idle storage-1 192.178.204.11 No No 19.09 idle
To install the patch on all hosts, execute the command against each host.
controller-0:~$ sudo sw-patch host-install controller-1 .... Installation was successful. controller-0:~$ sudo sw-patch host-install worker-0 .... Installation was successful. controller-0:~$ sudo sw-patch host-install worker-1 .... Installation was successful. controller-0:~$ sudo sw-patch host-install storage-0 ... Installation was successful. controller-0:~$ sudo sw-patch host-install storage-1 ... Installation was successful.
Check the status of the patch and the hosts. The patch has been installed on all hosts as shown below. The “Patch Current” value of the hosts changed to “Yes” and the “Patch State” value changed to “Applied”.
controller-0:~$ sudo sw-patch query Patch ID RR Release Patch State ======== == ======= =========== 001 N 19.09 Applied
controller-0:~$ sudo sw-patch query-hosts Hostname IP Address Patch Current Reboot Required Release State ============ ============== ============ =============== ======= ===== worker-0 192.178.204.7 Yes No 19.09 idle worker-1 192.178.204.9 Yes No 19.09 idle controller-0 192.178.204.3 Yes No 19.09 idle controller-1 192.178.204.4 Yes No 19.09 idle storage-0 192.178.204.12 Yes No 19.09 idle storage-1 192.178.204.11 Yes No 19.09 idle
This output confirms that the patch
001.patchhas been applied to the whole system.
StarlingX patching also supports patch removal, using the sw-patch remove and sw-patch host-install commands. The procedure is similar to that of patch applying and is not described here.
Patch orchestration¶
In the example shown above, the hosts in the cluster were updated one by one. For a case where the cluster size is very large, the updating process takes a long time, and the situation takes longer for reboot-required patches. The updating operation becomes very inefficient and can be a burden to the cluster administrator. StarlingX has an advanced feature called patch orchestration. It allows the whole system to be patched with a few operations, which simplifies update tasks for the cluster administrator. The operations can be performed using the CLI, the Horizon GUI, or the VIM RESTful API.
Using the StarlingX CLI client tool
sw-manager, you can create and apply a patch strategy, which then updates the whole system. The supported options are listed below.controller-0:~$ sw-manager patch-strategy -h usage: sw-manager patch-strategy [-h] ... optional arguments: -h, --help show this help message and exit Software Patch Commands: create Create a strategy delete Delete a strategy apply Apply a strategy abort Abort a strategy show Show a strategycontroller-0:~$ sw-manager patch-strategy create -h usage: sw-manager patch-strategy create [-h] [--controller-apply-type {serial,ignore}] [--storage-apply-type {serial,parallel,ignore}] [--worker-apply-type {serial,parallel,ignore}] [--max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10, 11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27, 28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44, 45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61, 62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78, 79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95, 96,97,98,99,100}] [--instance-action {migrate,stop-start}] [--alarm-restrictions {strict,relaxed}] optional arguments: -h, --help show this help message and exit --controller-apply-type {serial,ignore} defaults to serial --storage-apply-type {serial,parallel,ignore} defaults to serial --worker-apply-type {serial,parallel,ignore} defaults to serial --max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10,11,12,13,14,15,16, 17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36, 37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56, 57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76, 77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96, 97,98,99,100} maximum worker hosts to patch in parallel --instance-action {migrate,stop-start} defaults to stop-start --alarm-restrictions {strict,relaxed} defaults to strictUsing the Horizon GUI, click and click the Patch Orchestration tab.
Using the VIM API, http://<oam_ip>:4545
Method
URI
Description
Post
/api/orchestration/sw-update/strategy
Create a patch strategy
Delete
/api/orchestration/sw-update/strategy
Delete current patch strategy
Get
/api/orchestration/sw-update/strategy
Get detailed information of current patch strategy
Post
/api/orchestration/sw-update/strategy/ actions
Apply or abort a patch strategy
The system must be in good condition to use patch orchestration. For example:
All hosts must be in the state of unlocked-enabled-available.
The system is clear of alarms.
Enough spare worker nodes for VM migration.
Current status¶
The whole patching source code is already in StarlingX repositories, across several projects, like “update” and “nfv”.
Patch generation and manual patch application have been roughly verified for both in-service patches and reboot-required patches. They are working.
Patch orchestration has not been verified yet.
