Add Worker Nodes at Scale

When scaling a Standard or All-in-Duplex Plus (AIO-DX+) system to a large number of worker nodes, add the nodes using the procedure below so that new nodes reach the inventoried state quickly.

On a large system, each application reapply-evaluation scans every host. As the fleet grows, these evaluations run back-to-back and keep sysinv-conductor busy, so newly added worker nodes take progressively longer to reach the inventoried state and become unlockable. Disabling the reapply-evaluation parameters during the bulk node addition avoids this delay.

Note

This procedure becomes necessary at or above approximately 40 worker nodes. It is not required on smaller systems, where worker nodes can be added using the standard procedure. See Add Compute Nodes to an Existing Duplex System.

Procedure

1. Disable the reapply-evaluation parameters.

   $ system service-parameter-modify platform config evaluate_apps_reapply=disabled
   $ system service-parameter-modify platform config k8s_application_audit=disabled
   $ system service-parameter-modify platform config autoreapply_apps_after_apply_runtime_manifest=disabled
   

2. Add the worker nodes.

   Add one or more host entries to the system inventory, then let the nodes install, come online, and reach the unlocked state. See StarlingX Installation: Adding Hosts Using the host-add Command.

3. Re-enable the reapply-evaluation parameters.

   $ system service-parameter-modify platform config evaluate_apps_reapply=enabled
   $ system service-parameter-modify platform config k8s_application_audit=enabled
   $ system service-parameter-modify platform config autoreapply_apps_after_apply_runtime_manifest=enabled
   

4. Manually re-apply the OpenStack application.

   The application does not auto-reapply after the parameters are re-enabled.

   $ system application-apply wr-openstack
   

5. Verify that the new hypervisors are present.

   $ source /var/opt/openstack/admin-openrc
   $ openstack hypervisor stats show