Handle Kubernetes Version Upgrade Orchestration Failures

The creation or application of a strategy could fail for any of the listed reasons described in this section. Follow the suggested actions in each case to resolve the issue.

Strategy creation failure

  • Reason: Build failed with no reason.

    • Action:

      • Verify that the --worker-apply-type was not set to ignore.

      • Check recent logs added to /var/log/nfv-vim.log.

  • Reason: Alarms from platform are present.

    • Action:

      • Query for management affecting alarms and take actions to clear them.

        ~(keystone_admin)$ fm alarm-list --mgmt_affecting

      • If there are no management affecting alarms present, take actions to clear other reported alarms or try creating the strategy with the relaxed alarms restrictions option --alarm-restrictions relaxed.

  • Reason: No Kubernetes version upgrade required.

    • Action:

      • Verify that the Kubernetes patches have been uploaded and applied. Verify the version of Kubernetes on the hosts by executing “system kube-host-upgrade-list.

        Note

        If the strategy create failed, first you must resolve it. You must delete the failed strategy before you create another strategy.

Strategy Apply Failure

  • Reason: Alarms from platform are present.

    • Action: This suggests that an alarm has been raised since the creation of the strategy. Address the cause of the new alarm, delete the strategy and try creating and applying a new strategy.

  • Reason: Unable to migrate instances.

  • Reason: Kubernetes version upgrade failed. Suggests that the Kubernetes upgrade for the specified host has failed.

  • Reason: Lock host failed.

    • Action:

      • Investigate the /var/log/sysinv.log, and /var/log/nfv-vim.log files.

      • Address the underlying issue.

      • Manually lock and unlock the host.

      • Try recreating and re-applying the Kubernetes version upgrade strategy to automatically finish the upgrade process.

  • Reason: Unlock host failed.

    • Action:

      • Investigate /var/log/mtcAgent.log file for cause logs files.

      • Address the underlying issue.

      • Manually lock and unlock the host to recover.

      • Try recreating and re-applying the Kubernetes version upgrade strategy to automatically finish the upgrade process.

Note

If the strategy apply fails, you must resolve the strategy:command:apply failure, and delete the failed strategy before trying to create and apply another strategy.