Distributed Software Deploy Orchestration Process using the CLI

Distributed software deploy orchestration can be initiated after the System Controller has been successfully updated (patch) or deployed.

For more information on Prestaging Subcloud Orchestration see, Prestage Subcloud Orchestration.

Note

This section is applicable to users that use the dcmanager software deploy orchestration strategy to manage upgrades across subclouds.

Refer to StarlingX Updates and Upgrades for more details.

About this task

The user first creates a distributed software deploy orchestration strategy, or plan, for the automated software deploy procedure. This customizes the software deploy orchestration, using parameters to specify:

  • whether to stop on failure of a subcloud update/upgrade or continue with the next subcloud

  • whether to update/upgrade hosts serially or in parallel

Based on these parameters, and the state of the subclouds, the software deploy orchestration creates a number of stages for the overall sw-deploy strategy. All the subclouds that are included in the same stage will be updated/upgraded in parallel.

Prerequisites

Distributed software deploy orchestration can only be done on a system that meets the following conditions:

  • All subclouds are clear of management-affecting alarms (with the exception of the alarm upgrade in progress).

  • All hosts of all subclouds must be unlocked, enabled, and available.

  • All subclouds should have been prestaged (--for-sw-deploy).

    ~(keystone_admin)]$ dcmanager prestage-strategy create --for-sw-deploy

  • All the subclouds do not have any VIM strategy different from sw-deploy or in an invalid state.

  • No distributed software deploy orchestration strategy exists, to verify use the command dcmanager sw-deploy-strategy show. An update/upgrade cannot be orchestrated while software deploy orchestration is in progress.

  • The size and format of the persistent filesystem, /opt/platform-backup, of each subcloud must be 30GiB (or larger) and ext4 respectively. From the shell on a subcloud, use the following command to view the details of its persistent file system:

    df -Th /opt/platform-backup

    The type must be ext4 and the size must be 30GiB. For example, on controller-0, run the following command:

    ~(keystone_admin)]$ df -Th /opt/platform-backup
Filesystem  Type    Size    Used     Avail  Use%  Mounted on
/dev/sda1   ext4    29G     6.9G     21G    26%   /opt/platform-backup

  • If a previous upgrade has been done on the subcloud, from the shell on each subcloud, use the following command to remove the previous upgrade data:

    sudo rm /opt/platform-backup/upgrade_data*

You can configure a software deploy Distributed Cloud orchestration strategy using the dcmanager CLI or the Horizon web interface. If you prefer to use Horizon, see Create a Software Deploy Orchestration using Horizon.

Procedure

  1. Review the software status for the subclouds.

    After the System Controller upgrade/deploy is completed, wait for 40 seconds for the software_sync_status of all subclouds to be updated.

    To identify which subclouds are deploy-current (in-sync), use the subcloud list command. For example:

    ~(keystone_admin)]$ dcmanager subcloud list
+----+-----------+--------------+--------------------+-------------+
| id | name      | management   | availability       | sync        |
+----+-----------+--------------+--------------------+-------------+
|  1 | subcloud-1| managed      | online             | out-of-sync |
|  2 | subcloud-2| managed      | online             | out-of-sync |
|  3 | subcloud-3| managed      | online             | out-of-sync |
|  4 | subcloud-4| managed      | online             | out-of-sync |
+----+-----------+--------------+--------------------+-------------+

    Note

    The subclouds are out-of-sync because the software-sync-status is out-of-sync. All of the above subclouds are not deploy-current and, therefore, need to be upgraded/updated.

    To see synchronization details for a subcloud, use the following command:

    ~(keystone_admin)]$ dcmanager subcloud show subcloud1
+-----------------------------+------------------------------+
| Field                       | Value                        |
+-----------------------------+------------------------------+
| id                          | 1                            |
| name                        | subcloud-1                   |
| description                 | None                         |
| location                    | None                         |
| software_version            | nn.nn                        |
| management                  | managed                      |
| availability                | online                       |
| deploy_status               | complete                     |
| management_subnet           | fd01:82::0/64                |
| management_start_ip         | fd01:82::2                   |
| management_end_ip           | fd01:82::11                  |
| management_gateway_ip       | fd01:82::1                   |
| systemcontroller_gateway_ip | fd01:81::1                   |
| group_id                    | 1                            |
| created_at                  | 2021-06-07 21:05:16.224664   |
| updated_at                  | 2021-06-09 20:01:37.525012   |
| dc-cert_sync_status         | in-sync                      |
| firmware_sync_status        | in-sync                      |
| identity_sync_status        | in-sync                      |
| kubernetes_sync_status      | in-sync                      |
| load_sync_status            | not-available                |
| patching_sync_status        | not-available                |
| software_sync_status        | out-of-sync                  |
| platform_sync_status        | in-sync                      |
+-----------------------------+------------------------------+

  2. To create a sw-deploy strategy, use the dcmanager sw-deploy-strategy create <release_id> command.

    The sw-deploy strategy for a Distributed Cloud system controls how updates/upgrades are applied to subclouds.

    ~(keystone_admin)]$  dcmanager sw-deploy-strategy create \
[--subcloud-apply-type <type>] \
[–-max-parallel-subclouds <i>] \
[–-stop-on-failure <level>] \
[--group group] \
[<subcloud>]

    where:

    subcloud-apply-type

    parallel or serial— determines whether the subclouds are upgraded in parallel, or serially.

    If this is not specified using the CLI, the values for subcloud_update_type defined for each subcloud group will be used by default.

    max-parallel-subclouds

    Sets the maximum number of subclouds that can be upgraded in parallel (default 2).

    If this is not specified using the CLI, the values for max_parallel_subclouds defined for each subcloud group will be used by default.

    stop-on-failure

    true(default) or false— determines whether upgrade orchestration failure for a subcloud prevents application to subsequent subclouds.

    group

    Optionally pass the name or ID of a subcloud group to the dcmanager sw-deploy-strategy create command. This results in a strategy that is only applied to all subclouds in the specified group. The subcloud group values are used for subcloud apply type and max parallel subclouds parameters.

    For example:

    ~(keystone_admin)]$ dcmanager sw-deploy-strategy create
+------------------------+----------------------------+
| Field                  | Value                      |
+------------------------+----------------------------+
| strategy type          | sw-deploy                  |
| subcloud apply type    | parallel                   |
| max parallel subclouds | 2                          |
| stop on failure        | False                      |
| state                  | initial                    |
| created_at             | 2024-11-06 12:56:17.111621 |
| updated_at             | None                       |
+------------------------+----------------------------+

  3. To show the settings for the sw-deploy-strategy, use the dcmanager sw-deploy-strategy show command.

    For example:

    ~(keystone_admin)]$ dcmanager sw-deploy-strategy show
+------------------------+----------------------------+
| Field                  | Value                      |
+------------------------+----------------------------+
| strategy type          | sw-deploy                  |
| subcloud apply type    | parallel                   |
| max parallel subclouds | 2                          |
| stop on failure        | False                      |
| state                  | initial                    |
| created_at             | 2020-02-02T14:42:13.822499 |
| updated_at             | None                       |
+------------------------+----------------------------+

    Note

    The values for subcloud apply type and max parallel subclouds will be taken from the subcloud group if specified through the --group parameter.

  4. Review the sw-deploy strategy for the subclouds.

    To show the subclouds that will be upgraded when the sw-deploy strategy is applied, use the dcmanager strategy-step list command. For example:

    ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+--------+---------+---------+------------+-------------+
| cloud            | stage  | state   | details | started_at | finished_at |
+------------------+--------+---------+---------+------------+-------------+
| subcloud-1       | 1      | initial |         | None       | None        |
| subcloud-4       | 1      | initial |         | None       | None        |
| subcloud-5       | 2      | initial |         | None       | None        |
| subcloud-6       | 2      | initial |         | None       | None        |
+------------------+--------+---------+---------+------------+-------------+

    Note

    All the subclouds that are included in the same stage will be deployed in parallel.

  5. To apply the software deploy strategy, use the dcmanager sw-deploy-strategy apply command.

    ~(keystone_admin)]$ dcmanager sw-deploy-strategy apply
+------------------------+----------------------------+
| Field                  | Value                      |
+------------------------+----------------------------+
| subcloud apply type    | parallel                   |
| max parallel subclouds | 2                          |
| stop on failure        | False                      |
| state                  | applying                   |
| created_at             | 2020-02-02T14:42:13.822499 |
| updated_at             | 2020-02-02T14:42:19.376688 |
+------------------------+----------------------------+

  6. To show the step currently being performed on each of the subclouds, use the dcmanager strategy-step list command.

    For example:

    ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+--------+-----------------------+-------------------+----------------------------+----------------------------+
| cloud            | stage  | state                 | details           | started_at                 | finished_at                |
+------------------+--------+-----------------------+-------------------+----------------------------+----------------------------+
| subcloud-1       | 3      | complete              |                   | 2024-11-06 12:57:20.342429 | 2024-11-06 12:57:41.054664 |
| subcloud-4       | 2      | apply VIM sw-deploy   |                   | 2024-11-06 12:57:20.342429 | None                       |
| subcloud-5       | 1      | initial               |                   | None                       | None                       |
| subcloud-6       | 1      | initial               |                   | None                       | None                       |
+------------------+--------+-------------+-----------------------------+----------------------------+----------------------------+

  7. To show the step currently being performed on a subcloud, use the dcmanager strategy-step show <subcloud> command.

    ~(keystone_admin)]$ dcmanager strategy-step show <subcloud>

  8. When all the subclouds within the distributed software deploy orchestration indicate they have entered the complete state, delete the sw-deploy strategy, using the dcmanager sw-deploy-strategy delete command.

    ~(keystone_admin)]$ dcmanager sw-deploy-strategy delete
+------------------------+----------------------------+
| Field                  | Value                      |
+------------------------+----------------------------+
| subcloud apply type    | parallel                   |
| max parallel subclouds | 2                          |
| stop on failure        | False                      |
| state                  | deleting                   |
| created_at             | 2020-03-23T20:04:50.992444 |
| updated_at             | 2020-03-23T20:05:14.157352 |
+------------------------+----------------------------+

Note

Before attempting to log in to the subclouds using the sysadmin account, verify that the subcloud platform_sync_status is synced. This would ensure that the sysadmin password is successfully resynced to the subclouds and that login attempts do not fail.