Manual Rollback Host Software DeploymentΒΆ

For a major release software deployment, you can roll back the Manual Host Software Deployment procedure at any time between software deploy start and software deploy delete. After the software deploy deletion step, aborting and rolling back of the major release deployment is not possible.

Note

This section also covers the abort and rollback of a new patched major release deployment.

Prerequisites

You are in the middle of the Manual Host Software Deployment procedure for a major release between software deploy start and software deploy delete.

Procedure

  1. Abort the current in-progress major release software deployment.

    ~(keystone_admin)]$ software deploy abort
Deployment has been aborted

  2. If the current deploy state is deploy-activate-rollback-pending, then roll back the activate of the aborted deployment, otherwise proceed to 3.

    ~(keystone_admin)]$ software deploy show
+------------------+------------+------+----------------------------------+
| From Release     | To Release |  RR  |    State                         |
+------------------+------------+------+----------------------------------+
| <new-maj-rel-id> | 10.0.0     | True | deploy-activate-rollback-pending |
+------------------+------------+------+----------------------------------+

    ~(keystone_admin)]$ software deploy activate-rollback
Deploy activate-rollback has started

    When running the software deploy activate-rollback command, previous configurations are applied to the controller.

    Alarm 250.001 (Configuration is out-of-date) is raised and cleared as the configurations are applied.

    The software deployment state goes from activate-rollback-done to host-rollback.

    This may take up to 30 mins to complete depending on system configuration and hardware.

    ~(keystone_admin)]$ software deploy show
+------------------+------------+------+---------------+
| From Release     | To Release |  RR  |    State      |
+------------------+------------+------+---------------+
| <new-maj-rel-id> | 10.0.0     | True | host-rollback |
+------------------+------------+------+---------------+

    Note

    If software deploy activate-rollback fails, that is, if the state is activate-rollback-failed, review /var/log/software.log on the active controller for failure details, address the issues, and re-execute the software deploy activate-rollback command.

  3. If the current deploy state is host-rollback, then roll back the deployment of all the hosts.

    ~(keystone_admin)]$ software deploy show
+------------------+------------+------+-------------------+
| From Release     | To Release |  RR  |    State          |
+------------------+------------+------+-------------------+
| <new-maj-rel-id> | 10.0.0     | True | host-rollback     |
+------------------+------------+------+-------------------+

    If the state is host-rollback, then proceed with the rest of this step, otherwise proceed to 4.

    • For an AIO-SX system

      1. Roll back the software release on controller-0.

        1. Lock controller-0.

          ~(keystone_admin)]$ system host-lock controller-0

        2. Roll back the software release on controller-0.

             ~(keystone_admin)]$ software deploy host-rollback controller-0
   Host installation request sent to controller-0.
   Host installation was successful on controller-0

The host is still running the new software release, however boot
parameters have been updated to boot into the previous software
release on the next host reboot, which will occur in the next step
which unlocks the host.

        3. Unlock controller-0.

          ~(keystone_admin)]$ system host-unlock controller-0

          The host will now reboot into the previous software release. Wait for the host to finish rebooting and become available.

          This may take 3-5 mins depending on hardware.

      2. Proceed to step 4 of the main procedure.

    • For an AIO-DX system or standard system

      1. If worker hosts are present, and one or more are in the pending-rollback state, then roll back the software release on all worker hosts in the pending-rollback state one at a time. Otherwise, proceed to step b.

        ~(keystone_admin)]$ software deploy host-list
+--------------+------------------+------------+-------+------------------------------+
| Host         | From Release     | To Release |  RR   |  State                       |
+--------------+------------------+------------+-------+------------------------------+
| controller-0 | <new-maj-rel-id> |  10.0.0    | True  | deploy-host-rollback-pending |
| controller-1 | <new-maj-rel-id> |  10.0.0    | True  | deploy-host-rollback-pending |
| storage-0    | <new-maj-rel-id> |  10.0.0    | True  | deploy-host-rollback-pending |
| storage-1    | <new-maj-rel-id> |  10.0.0    | True  | deploy-host-rollback-pending |
| storage-2    | <new-maj-rel-id> |  10.0.0    | True  | deploy-host-rollback-pending |
| storage-3    | <new-maj-rel-id> |  10.0.0    | True  | deploy-host-rollback-pending |
| worker-0     | <new-maj-rel-id> |  10.0.0    | True  | deploy-host-rollback-pending |
| worker-1     | <new-maj-rel-id> |  10.0.0    | True  | deploy-host-rollback-pending |
| worker-2     | <new-maj-rel-id> |  10.0.0    | True  | deploy-host-rollback-pending |
| worker-3     | <new-maj-rel-id> |  10.0.0    | True  | deploy-host-rollback-deployed|
+--------------+------------------+------------+-------+------------------------------+

        1. Roll back the software release on worker-0.

          1. Lock worker-0.

            ~(keystone_admin)]$ system host-lock worker-0

          2. Roll back the software release on worker-0.

            ~(keystone_admin)]$ software deploy host-rollback worker-0
Host installation request sent to worker-0
Host installation was successful on worker-0.

            The host is still running the new software release, however boot parameters have been updated to boot into the previous software release on the next host reboot, which will occur in the next step which unlocks the host.

          3. Unlock worker-0.

            ~(keystone_admin)]$ system host-unlock worker-0

            The host will now reboot into the previous software release. Wait for the host to finish rebooting and become available. Wait for all the alarms to clear after the unlock before proceeding to the next worker host.

            This may take 3-5 mins depending on hardware.

          4. Display the state of software deployment.

          ~(keystone_admin)]$ software deploy host-list
+--------------+------------------+------------+-------+------------------+
| Host         | From Release     | To Release |  RR   |  State           |
+--------------+------------------+------------+-------+------------------+
| controller-0 | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| controller-1 | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| storage-0    | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| storage-1    | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| storage-2    | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| storage-3    | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| worker-0     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-1     | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| worker-2     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-3     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
+--------------+------------------+------------+-------+------------------+

        2. Repeat the above steps for any remaining worker hosts in the pending-rollback state.

      2. If storage hosts are present, and one or more are in the pending-rollback state, then roll back the software release on all storage hosts in the pending-rollback state, one at a time. Otherwise, proceed to step c.

        ~(keystone_admin)]$ software deploy host-list
+--------------+------------------+------------+-------+------------------+
| Host         | From Release     | To Release |  RR   |  State           |
+--------------+------------------+------------+-------+------------------+
| controller-0 | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| controller-1 | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| storage-0    | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| storage-1    | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| storage-2    | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| storage-3    | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| worker-0     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-1     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-2     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-3     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
+--------------+------------------+------------+-------+------------------+

        1. Roll back the software release on storage-0.

          1. Lock storage-0.

            ~(keystone_admin)]$ system host-lock storage-0

          2. Roll back the software release on storage-0.

            ~(keystone_admin)]$ software deploy host-rollback storage-0
Host installation request sent to storage-0
Host installation was successful on storage-0.

            The host is still running the new software release, however boot parameters have been updated to boot into the previous software release on the next host reboot, which will occur in the next step which unlocks the host.

          3. Unlock storage-0.

            ~(keystone_admin)]$ system host-unlock storage-0

            The host will now reboot into the previous software release. Wait for the host to finish rebooting and become available. Wait for all the alarms to clear after the unlock before proceeding to the next storage host.

            This may take 3-5 mins depending on hardware.

          4. Display the state of software deployment.

            ~(keystone_admin)]$ software deploy host-list
+--------------+------------------+------------+-------+------------------+
| Host         | From Release     | To Release |  RR   |  State           |
+--------------+------------------+------------+-------+------------------+
| controller-0 | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| controller-1 | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| storage-0    | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| storage-1    | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| storage-2    | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| storage-3    | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| worker-0     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-1     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-2     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-3     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
+--------------+------------------+------------+-------+------------------+

        2. Repeat the above steps for any remaining storage hosts in the pending-rollback state.

          Note

          After rolling back the first storage host, you can expect alarm 800.003. The alarm is cleared after all the storage hosts are rolled back.

      3. If both the controllers are in the pending-rollback state, then roll back controller-0 first.

        1. Ensure that controller-1 is active by switching activity from controller-0.

          ~(keystone_admin)]$ system host-swact controller-0

          Wait for the activity to switch to controller-1. This may take up to a minute depending on hardware. Reconnect to the system.

        2. Roll back the software release on controller-0 (the standby controller).

          1. Lock controller-0.

            ~(keystone_admin)]$ system host-lock controller-0

          2. Rollback the software release on controller-0.

            ~(keystone_admin)]$ software deploy host controller-0
Host installation request sent to controller-0.
Host installation was successful on controller-0.

            The host is still running the new software release, however boot parameters have been updated to boot into the previous software release on the next host reboot, which will occur in the next step which unlocks the host.

          3. Unlock controller-0.

            ~(keystone_admin)]$ system host-unlock controller-0

            The host will now reboot into the new software release. Wait for the host to finish rebooting and become available.

            This may take 3-5 mins depending on hardware.

          4. Display the state of software deployment.

            ~(keystone_admin)]$ software deploy host-list
+--------------+------------------+------------+-------+------------------+
| Host         | From Release     | To Release |  RR   |  State           |
+--------------+------------------+------------+-------+------------------+
| controller-0 | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| controller-1 | <new-maj-rel-id> |  10.0.0    | True  | pending-rollback |
| storage-0    | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| storage-1    | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| storage-2    | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| storage-3    | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-0     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-1     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-2     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-3     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
+--------------+------------------+------------+-------+------------------+

      4. If only controller-1 is in the pending-rollback state, then roll back controller-1.

        1. Ensure that controller-0 is active by switching activity from controller-1.

          ~(keystone_admin)]$ system host-swact controller-1

          Wait for the activity to switch to controller-0.

          This may take up to a minute depending on hardware.

          Reconnect to the system.

        2. Roll back the software release on controller-1 (the standby controller).

          1. Lock controller-1

            ~(keystone_admin)]$ system host-lock controller-1

          2. Roll back the software release on controller-1.

            ~(keystone_admin)]$ software deploy host controller-1
Host installation request sent to controller-1.
Host installation was successful on controller-1.

            The host is still running the new software release, however boot parameters have been updated to boot into the previous software release on the next host reboot, which will occur in the next step which unlocks the host.

          3. Unlock controller-1.

            ~(keystone_admin)]$ system host-unlock controller-1

            The host will now reboot into the new software release. Wait for the host to finish rebooting and become available.

            This may take 3-5 mins depending on hardware.

          4. Display the state of software deployment.

            ~(keystone_admin)]$ software deploy host-list
+--------------+------------------+------------+-------+------------------+
| Host         | From Release     | To Release |  RR   |  State           |
+--------------+------------------+------------+-------+------------------+
| controller-0 | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| controller-1 | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| storage-0    | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| storage-1    | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| storage-2    | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| storage-3    | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-0     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-1     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-2     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
| worker-3     | <new-maj-rel-id> |  10.0.0    | True  | rolled back      |
+--------------+------------------+------------+-------+------------------+

  4. Delete the software deployment to complete the rollback.

    ~(keystone_admin)]$ software deploy delete
Deployment has been deleted

    ~(keystone_admin)]$ software deploy show
No deploy in progress

  5. Confirm that the previous software release is now deployed.

    ~(keystone_admin)]$ software list
+--------------------------+-------+-----------+
| Release                  | RR    |   State   |
+--------------------------+-------+-----------+
| starlingx-10.0.0         | True  | deployed  |
| <new-maj-rel-id>         | True  | available |
+--------------------------+-------+-----------+