Install Kubernetes Platform on All-in-One Duplex¶

Overview¶

The AIO-DX deployment option provides a pair of HA servers with each server providing all three cloud functions (controller, worker, and storage).

For more information, see Supported Deployment Models.

For information on getting started quickly using an automated virtual installation, see Automated Virtual Installation.

AIO-DX Installation Workflow¶

The following table shows the high-level installation flow for AIO-DX:

Step	Section	Description
1	Prepare and Plan	Gather prerequisites, prepare servers, and obtain the ISO image.
2	Install Software on Controller-0	Install the platform OS on the first server.
3	Bootstrap System on Controller-0	Run the Ansible bootstrap playbook to initialize services.
4	Configure Controller-0	Set up OAM, management interfaces, and NTP.
4a	Configure PCI-SRIOV Interfaces (Optional)	Enable SR-IOV-capable interfaces on controller-0.
5	Unlock Controller-0	Unlock and bring controller-0 into service.
5a	Ceph Storage Backend Configuration (Optional)	Set up host-based Ceph or Rook-Ceph for persistent volume claims.
6	Install Software on Controller-1 Node	PXE boot and install controller-1.
7	Configure Controller-1	Set up OAM, MGMT interfaces on controller-1.
7a	Configure PCI-SRIOV Interfaces (Optional)	Enable SR-IOV-capable interfaces on controller-1.
7b	Rook Ceph Storage Backend Configuration (Optional)	Configure Ceph storage on controller-1.
8	Unlock Controller-1	Unlock and bring controller-1 into service.
9	Extend Capacity with Worker Nodes (Optional)	Add worker nodes for additional capacity.

Minimum Hardware Requirements¶

This section describes the hardware requirements and server preparation for a StarlingX r13.0 bare metal Duplex deployment configuration.

The recommended minimum hardware requirements for bare metal servers for various host types are:

Minimum Requirements	All-in-one Controller Node	Worker Node
Number of servers	2	2-99
Minimum processor class	Dual-CPU Intel® Xeon® E5 26xx family (SandyBridge) 8 cores/socket or Single-CPU Intel® Xeon® D-15xx family, 8 cores (low-power/low-cost option) Note Platform: 1 physical core with HT enabled or 2 physical cores with HT disabled, (by default, configurable) The use of single physical core for platform function is only suitable for Intel® 4th Generation Xeon® Scalable Processors or above and should not be configured for previous Intel® Xeon® CPU families. For All-In-One systems with older generation processors, two physical cores (or more) must be configured. Application: Remaining cores	(Same as controller node)
Minimum memory	64 GB Platform: Socket 0: 10GB (by default, configurable) Socket 1: 1GB (by default, configurable) Application: Socket 0: Remaining memory Socket 1: Remaining memory	32 GB
Primary disk	500 GB SSD or NVMe (see NVME Configuration)	120 GB (Minimum 10k RPM)
Additional disks	1 or more 500 GB (min. 10K RPM) for Ceph OSD Recommended, but not required: 1 or more SSDs or NVMe drives for Ceph journals (min. 1024 MiB per OSD journal) Recommended, but not required: 1 or more 500G HDs (min. 10K RPM), SSDs or NVMe drives for Container ephemeral disk storage. For StarlingX OpenStack, we recommend 1 or more 500 GB (min. 10K RPM) for VM local ephemeral storage	For StarlingX OpenStack, we recommend 1 or more 500 GB (min. 10K RPM) for VM local ephemeral storage
Minimum network ports	MGMT: 1x1GE ( Recommended: MGMT 2x10GE LAG ). OAM: 1x1GE (Recommended: OAM: 2x1GE LAG) Data: 1 or more x 10GE (Recommended: Data: 2x10GE LAG)	Mgmt/Cluster: 1x10GE (Recommended: MGMT 2x10GE LAG) Data: 1 or more x 10GE (Recommended: Data 2x10GE LAG)
USB	1 (Only required if used for initial installation of controller-0).
Board Management		1 BMC
Power profile		C-states (up to C6) may be configured for some use cases where application workloads can tolerate additional scheduling/timer latency. Note C-states may be enabled depending on application requirements.
Boot order	HD, PXE, USB	HD, PXE
BIOS mode	BIOS or UEFI Note UEFI Secure Boot and UEFI PXE boot over IPv6 are not supported. On systems with an IPv6 management network, you can use a separate IPv4 network for PXE boot. For more information, see PXE Boot Controller-0.	BIOS or UEFI
Bios settings	Hyper-Threading technology enabled Virtualization technology enabled VT for directed I/O enabled CPU power and performance policy set to performance CPU C state control disabled Plug & play BMC detection disabled Note The system will not override the recommended BIOS settings.	(Same as controller node)

The following sections describe system requirements and host setup for a workstation hosting virtual machine(s) where StarlingX will be deployed; i.e., a VM for each StarlingX node (controller, AIO-controller, worker or storage node).

Hardware requirements

The host system should have at least:

Processor: x86_64 is the only supported architecture with BIOS enabled hardware virtualization extensions
Cores: 8
Memory: 32GB RAM
Hard Disk: 500GB HDD
Network: One network adapter with active Internet connection

Software requirements

The host system should have at least:

A workstation computer with Ubuntu 16.04 LTS 64-bit

All other required packages will be installed by scripts in the StarlingX tools repository.

Host setup

Set up the host with the following steps:

Update OS:
```
apt-get update
```

Clone the StarlingX tools repository:

apt-get install -y git
cd $HOME
git clone https://opendev.org/starlingx/virtual-deployment.git

Install required packages:
```
cd $HOME/virtual-deployment/libvirt
bash install_packages.sh
apt install -y apparmor-profiles
apt-get install -y ufw
ufw disable
ufw status
```
Note

On Ubuntu 16.04, if apparmor-profile modules were installed as shown in the example above, you must reboot the server to fully install the apparmor-profile modules.

Prepare and Plan¶

Complete the following prerequisites before starting the StarlingX installation.

Note

By default, StarlingX uses IPv4. To use StarlingX with IPv6:

The entire infrastructure and cluster configuration must be IPv6, with the exception of the PXE boot network.
Not all external servers are reachable via IPv6 addresses (for example Docker registries). Depending on your infrastructure, it may be necessary to deploy a NAT64/DNS64 gateway to translate the IPv4 addresses to IPv6.

Refer to StarlingX IPv6 Deployment for details on how to deploy a NAT64/DNS64 gateway to use StarlingX with IPv6.

Preparing servers is the first step of the StarlingX installation procedure.

Prior to starting the StarlingX installation, ensure that the bare metal servers are in the following state:

Physically installed.
Cabled for power.
Cabled for networking.

Far-end switch ports should be properly configured to realize the networking shown in the diagram earlier in this topic.
All disks are wiped.

This ensures that servers will boot from either the network or USB storage, if present.
BIOS configured with Intel Virtualization (VTD, VTX)
- Disabled for controller-only servers and storage servers.
- Enabled for controller+worker (All-in-one) servers and worker servers.
The servers are powered off.

After the servers are ready, download the ISO and prepare the boot mechanism. For more information, see Prepare the ISO and Boot Method.

Note

The following commands for host, virtual environment setup, and host power-on use KVM/virsh for virtual machine and VM management technology. For an alternative virtualization environment, see: Install StarlingX in VirtualBox.

Prepare virtual environment.

Set up the virtual platform networks for virtual deployment:
```
bash setup_network.sh
```
Prepare virtual servers.

Note

The following commands for host, virtual environment setup, and host power-on use KVM / virsh for virtual machine and VM management technology. For an alternative virtualization environment, see: Install StarlingX in VirtualBox.
1. Prepare virtual environment.
  
  Set up the virtual platform networks for virtual deployment:
```
bash setup_network.sh
```
2. Prepare virtual servers.
  
  Create the XML definitions for the virtual servers required by this configuration option. This will create the XML virtual server definition for:
  - duplex-controller-0
  - duplex-controller-1
  The following command will start/virtually power on:
  - The ‘duplex-controller-0’ virtual server
  - The X-based graphical virt-manager application
```
bash setup_configuration.sh -c duplex -i ./bootimage.iso
```
  If there is no X-server present, errors will occur and the X-based GUI for the virt-manager application will not start. The virt-manager GUI is not absolutely required, and you can safely ignore errors and continue.

Install Software on Controller-0¶

Install the StarlingX software on the first server in the deployment.

This step performs a non-interactive installation of the StarlingX operating system and RPMs on the server’s first disk. After installation, the system reboots into the newly installed OS.

The initial server becomes controller-0 for the deployment.

Note

The disks and disk partitions need to be wiped before the install. Installing a Debian ISO may fail with a message that the system is in emergency mode if the disks and disk partitions are not completely wiped before the install, especially if the server was previously running a CentOS ISO.

Insert the bootable USB into a bootable USB port on the host you are configuring as controller-0. Do one of the following:
- Create a bootable USB drive with the StarlingX host installer ISO. See Create Bootable USB for instructions on how to create a bootable USB with the StarlingX ISO.
- Set up a PXE boot server hosting the installer ISO on the same network as the server that will be used as the initial controller-0. See PXE Boot Controller-0 for details on how to setup a PXEBOOT Server and pxe-boot the StarlingX load on controller-0.
Power on the host.
Attach to a console, ensure the host boots from the USB, and wait for the StarlingX Installer Menus.
Wait for the Install menu to appear. When prompted, make the following selections in the installer:

Note

If you configured the default menu options into the ISO with the update-iso.sh script (with the -d option) as described in Installation Prerequisites, the Install menu will not appear.
1. Select the deployment option.
  
  Choose Debian All-in-one Install (required for an AIO Simplex or Duplex configuration). When prompted during the installation workflow, ensure that you select the Duplex deployment mode.
2. Select console type.
  
  Choose one of the following based on your access method:
  - Graphical Console
  - Serial Console
Wait for the non-interactive software installation to complete and for the server to reboot. This can take 5-10 minutes, depending on the performance of the server.

In the last step of Prepare Host and Environment, the setup_configuration.sh command started the controller-0 virtual server ‘duplex-controller-0’.

On the host, attach to the console of virtual controller-0 and select the appropriate installer menu options to start the non-interactive install of StarlingX software on controller-0.

Note

You can easily miss the first installer menu selection when entering the console. Use ESC to navigate to previous menus to ensure you are at the first installer menu.

virsh console duplex-controller-0

Make the following menu selections in the installer:

First menu: Select ‘All-in-one Controller Configuration’
Second menu: Select ‘Serial Console’

Wait for the non-interactive install of software to complete and for the server to reboot. This can take 5-10 minutes, depending on the performance of the host machine.

Bootstrap System on Controller-0¶

Use the following procedure to initialize, or bootstrap, the StarlingX software on controller-0. Provide a minimal set of system configuration parameters in the Ansible override file to complete the bootstrap process.

After the bootstrap process completes:

The system runs the core StarlingX services on controller-0.
You can access all APIs and CLIs locally.
The services are not yet providing operational workloads.

Note

If a manual or unexpected reboot occurs before the bootstrap phase completes (specifically before you unlock controller-0), you must reinstall the system starting from the software installation step. Back up any updated bootstrap or deployment configuration files to an external location before reinstalling.

Before You Begin

Note

Before running the Ansible bootstrap playbook, it is important that you ensure that controller-0 server time is synchronized correctly. Run the following command:

# check the current server time
$ date

# if the current server time is not correct, update the NTP Servers configuration.
For more information, see :ref:`Configure NTP Servers <configuring-ntp-servers-and-services-using-the-cli>`.

Procedure

Log in to controller-0 as sysadmin / sysadmin and change the password when prompted. This is required only for the first login.

Login: sysadmin
Password:
Changing password for sysadmin.
(current) UNIX Password: sysadmin
New Password:
(repeat) New Password:

Verify and/or configure IP connectivity.
External connectivity is required to run the Ansible bootstrap playbook. The StarlingX boot image will DHCP out all interfaces so the server may have obtained an IP address and have external IP connectivity if a DHCP server is present in your environment. Verify this using the ip addr and ping 8.8.8.8 command.

Otherwise, manually configure an IP address and default IP route. Use the PORT, IP-ADDRESS/SUBNET-LENGTH and GATEWAY-IP-ADDRESS applicable to your deployment environment.
sudo ip address add <IP-ADDRESS>/<SUBNET-LENGTH> dev <PORT> sudo ip link set up dev <PORT> sudo ip route add default via <GATEWAY-IP-ADDRESS> dev <PORT> ping 8.8.8
External connectivity is required to run the Ansible bootstrap playbook.
export CONTROLLER0_OAM_CIDR=10.10.10.3/24 export DEFAULT_OAM_GATEWAY=10.10.10.1 sudo ip address add $CONTROLLER0_OAM_CIDR dev enp7s1 sudo ip link set up dev enp7s1 sudo ip route add default via $DEFAULT_OAM_GATEWAY dev enp7s1
Specify user configuration overrides for the Ansible bootstrap playbook. Create the override file in the system administrator’s home directory (for example, $HOME/localhost.yml). Ansible automatically detects and applies this file during the bootstrap process.

Using a bootstrap override file, you can customize the Ansible bootstrap process during installation. This file lets you override the default settings defined in default.yml by specifying deployment-specific parameters, such as networking and system configuration.

Refer to Ansible Bootstrap Configurations for instructions on creating the override file and for details on advanced Ansible bootstrap configuration scenarios.

Important

Some Ansible bootstrap parameters cannot be changed or are very difficult to change after installation is complete.

Review the set of install-time-only parameters before installation and confirm that your values for these parameters are correct for the desired installation.

Refer to Ansible install-time-only parameters for details.

Specify the user configuration override file for the Ansible bootstrap playbook using one of the following methods:

Note

This Ansible Overrides file for the Bootstrap Playbook ($HOME/localhost.yml) contains security sensitive information, use the ansible-vault create $HOME/localhost.yml command to create it. You will be prompted for a password to protect/encrypt the file. Use the ansible-vault edit $HOME/localhost.yml command if the file needs to be edited after it is created.
1. Use a copy of the default.yml file listed above to provide your overrides.
  
  The default.yml file lists all available parameters for bootstrap configuration with a brief description for each parameter in the file comments.
  
  To use this method, run the ansible-vault create $HOME/localhost.yml command and copy the contents of the default.yml file into the ansible-vault editor, and edit the configurable values as required.
2. Create a minimal user configuration override file.
  
  To use this method, create your override file with the ansible-vault create $HOME/localhost.yml command and provide the minimum required parameters for the deployment configuration as shown in the example below. Use the OAM IP SUBNET and IP addressing applicable to your deployment environment.
  
  Note
  
  During system bootstrap, the platform does not support the use of quotation characters in the keystone user password.
```
cd ~

cat <<EOF > localhost.yml

system_mode: duplex

dns_servers:
  - 8.8.8.8
  - 8.8.4.4

external_oam_subnet: <OAM-IP-SUBNET>/<OAM-IP-SUBNET-LENGTH>
external_oam_gateway_address: <OAM-GATEWAY-IP-ADDRESS>
external_oam_floating_address: <OAM-FLOATING-IP-ADDRESS>
external_oam_node_0_address: <OAM-CONTROLLER-0-IP-ADDRESS>
external_oam_node_1_address: <OAM-CONTROLLER-1-IP-ADDRESS>

admin_username: admin
admin_password: <admin-password>
ansible_become_pass: <sysadmin-password>

EOF
```
  In either of the above options, the bootstrap playbook’s default values will pull all container images required for the StarlingX Platform from Docker hub.
  
  If you have setup a private Docker registry to use for bootstrapping then you will need to add the following lines in $HOME/localhost.yml:
```
docker_registries:
  quay.io:
     url: myprivateregistry.abc.com:9001/quay.io
  docker.elastic.co:
     url: myprivateregistry.abc.com:9001/docker.elastic.co
  gcr.io:
     url: myprivateregistry.abc.com:9001/gcr.io
  ghcr.io:
     url: myprivateregistry.abc.com:9001/ghcr.io
  k8s.gcr.io:
     url: myprivateregistry.abc.com:9001/k8s.gcr.io
  docker.io:
     url: myprivateregistry.abc.com:9001/docker.io
  registry.k8s.io:
     url: myprivateregistry.abc.com:9001/registry.k8s.io
  icr.io:
     url: myprivateregistry.abc.com:9001/icr.io
  defaults:
     type: docker
     username: <your_myprivateregistry.abc.com_username>
     password: <your_myprivateregistry.abc.com_password>

# Add the CA Certificate that signed myprivateregistry.abc.com’s
# certificate as a Trusted CA
ssl_ca_cert: /home/sysadmin/myprivateregistry.abc.com-ca-cert.pem
```
  See Use a Private Docker Registry for more information.
  
  If a firewall is blocking access to Docker hub or your private registry from your StarlingX deployment, you will need to add the following lines in $HOME/localhost.yml (see Docker Proxy Configuration for more details about Docker proxy settings):
```
# Add these lines to configure Docker to use a proxy server
docker_http_proxy: http://my.proxy.com:1080
docker_https_proxy: https://my.proxy.com:1443
docker_no_proxy:
   - 1.2.3.4
```
  Configure system_local_ca_cert, system_local_ca_key and system_root_ca_cert to setup a local intermediate CA (signed by an external Root CA) for managing / signing all of the StarlingX Certificates. See Platform Issuer (system-local-ca) for more details.
  
  Note
  
  The data provided through system_local_ca_key should contain an ECDSA (P-384 or P-521) or RSA (4096 bits or longer) private key, in unencrypted PEM format.
  
  Refer to Ansible Bootstrap Configurations for information on additional Ansible bootstrap configurations for advanced Ansible bootstrap scenarios.

Warning

The cluster-host and cluster-pod subnets cannot be changed after deployment is complete.

If any container workload on this system needs to connect to endpoints outside the cluster, then the cluster-host and cluster-pod networks must be:

Globally routable and reachable from those external endpoints, so that return traffic can reach the workload; and
Assigned a unique subnet that is not used by any other system on the same transport network.

See Cluster Host Network and Network Addressing Requirements. Correcting this later requires either a separate routable platform interface on every node that may host the workload, or a redeployment.

Run the Ansible bootstrap playbook:

Note

Before running the Ansible bootstrap playbook, it is important that you ensure that controller-0 server time is synchronized correctly. Run the following command:

# check the current server time
$ date

# if the current server time is not correct, update the NTP Servers configuration.
For more information, see :ref:`Configure NTP Servers <configuring-ntp-servers-and-services-using-the-cli>`.

ansible-playbook --ask-vault-pass /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml

Wait for Ansible bootstrap playbook to complete. This can take 5-10 minutes, depending on the performance of the host machine.

Configure Controller-0¶

Controller-0 configuration includes setting up administrative access, configuring interfaces, and configuring network time synchronization.

Retrieve admin credentials:
```
source /etc/platform/openrc
```

Configure the OAM interface of controller-0 and specify the attached network as “oam”.

The following example configures the OAM interface on a physical untagged ethernet port, use OAM port name that is applicable to your deployment environment, for example eth0:

~(keystone_admin)$ OAM_IF=<OAM-PORT>
~(keystone_admin)$ system host-if-modify controller-0 $OAM_IF -c platform
~(keystone_admin)$ system interface-network-assign controller-0 $OAM_IF oam

~(keystone_admin)$ OAM_IF=enp7s1
~(keystone_admin)$ system host-if-modify controller-0 $OAM_IF -c platform
~(keystone_admin)$ system interface-network-assign controller-0 $OAM_IF oam

To configure a VLAN or aggregated ethernet interface, see Node Interfaces.

Configure the MGMT interface of controller-0 and specify the attached networks of both “mgmt” and “cluster-host”.

The following example configures the MGMT interface on a physical untagged ethernet port. Use the MGMT port name that is applicable to your deployment environment, for example eth1:

~(keystone_admin)$ MGMT_IF=<MGMT-PORT>
~(keystone_admin)$ system host-if-modify controller-0 lo -c none
~(keystone_admin)$ IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
~(keystone_admin)$ for UUID in $IFNET_UUIDS; do
    system interface-network-remove ${UUID}
done
~(keystone_admin)$ system host-if-modify controller-0 $MGMT_IF -c platform
~(keystone_admin)$ system interface-network-assign controller-0 $MGMT_IF mgmt
~(keystone_admin)$ system interface-network-assign controller-0 $MGMT_IF cluster-host

~(keystone_admin)$ MGMT_IF=enp7s2
~(keystone_admin)$ system host-if-modify controller-0 lo -c none
~(keystone_admin)$ IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
~(keystone_admin)$ for UUID in $IFNET_UUIDS; do
    system interface-network-remove ${UUID}
done
~(keystone_admin)$ system host-if-modify controller-0 $MGMT_IF -c platform
~(keystone_admin)$ system interface-network-assign controller-0 $MGMT_IF mgmt
~(keystone_admin)$ system interface-network-assign controller-0 $MGMT_IF cluster-host

To configure a VLAN or aggregated ethernet interface, see Node Interfaces.

Configure NTP servers for network time synchronization:
~(keystone_admin)$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
To configure PTP instead of NTP, see PTP Server Configuration.
Note

In a virtual environment, this can sometimes cause Ceph clock skew alarms. Also, the virtual instance’s clock is synchronized with the host clock, so it is not absolutely required to configure NTP in this step.
~(keystone_admin)$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org

Configure PCI-SRIOV Interfaces (Optional)¶

The optional SR-IOV configuration step in the installation workflow allows you to enable and prepare SR-IOV-capable interfaces on the system during initial deployment. This step ensures that the platform recognizes SR-IOV hardware and makes those interfaces available for subsequent configuration, but it does not fully provision or configure them for workloads. You can complete the detailed configuration, such as assigning virtual functions (VFs) and enabling SR-IOV for containers or applications, later using the PCI-SRIOV interface support section in Node Management (Kubernetes).

Configure PCI-SR-IOV interfaces for controller-0.

This step applies when using SR-IOV network attachments in Kubernetes-hosted application containers or SR-IOV vNICs in OpenStack-hosted VMs. You can assign the same data networks to PCI-SRIOV interfaces as you would to vSwitch data interfaces.

Configure the PCI-SRIOV interfaces for controller-0.

~(keystone_admin)$ export NODE=controller-0

# List inventoried host’s ports and identify ports to be used as ‘pci-sriov’ interfaces,
# based on displayed linux port name, pci address and device type.
~(keystone_admin)$ system host-port-list ${NODE}

# List host’s auto-configured ‘ethernet’ interfaces,
# find the interfaces corresponding to the ports identified in previous step, and
# take note of their UUID
~(keystone_admin)$ system host-if-list -a ${NODE}

# Modify configuration for these interfaces
# Configuring them as ‘pci-sriov’ class interfaces, MTU of 1500 and named sriov#
~(keystone_admin)$ system host-if-modify -m 1500 -n sriov0 -c pci-sriov ${NODE} <sriov0-if-uuid> -N <num_vfs>
~(keystone_admin)$ system host-if-modify -m 1500 -n sriov1 -c pci-sriov ${NODE} <sriov1-if-uuid> -N <num_vfs>

# If not already created, create Data Networks that the 'pci-sriov'
# interfaces will be connected to
~(keystone_admin)$ DATANET0='datanet0'
~(keystone_admin)$ DATANET1='datanet1'
~(keystone_admin)$ system datanetwork-add ${DATANET0} vlan
~(keystone_admin)$ system datanetwork-add ${DATANET1} vlan

# Assign Data Networks to PCI-SRIOV Interfaces
~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <sriov0-if-uuid> ${DATANET0}
~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <sriov1-if-uuid> ${DATANET1}

To enable SR-IOV network attachments for the above interfaces in Kubernetes-hosted application containers:

Configure the Kubernetes SR-IOV device plugin.

~(keystone_admin)$ system host-label-assign controller-0 sriovdp=enabled

If you are planning on running DPDK in Kubernetes hosted application containers on this host, configure the number of 1G Huge pages required on both NUMA nodes.

# assign 10x 1G huge page on processor/numa-node 0 on controller-0 to applications
~(keystone_admin)$ system host-memory-modify -f application controller-0 0 -1G 10

# assign 10x 1G huge page on processor/numa-node 1 on controller-0 to applications
~(keystone_admin)$ system host-memory-modify -f application controller-0 1 -1G 10

Configure the Kubernetes SR-IOV device plugin.

~(keystone_admin)$ system host-label-assign controller-0 sriovdp=enabled

Unlock Controller-0¶

Additional Optional Configurations¶

If you need to install the StarlingX OpenStack application (stx-openstack), see OpenStack-Specific Host Configuration.

Use this procedure to complete the configuration of controller-0 and unlock it to provide service.

To unlock the controller and bring it into service, use the following command:

~(keystone_admin)$ system host-unlock controller-0

The controller reboots automatically as part of this process. Startup and configuration may take 5-10 minutes, depending on host performance.

Ceph Storage Backend Configuration (Optional)¶

Your application requires a persistent storage backend if it uses PVCs.

Important

The StarlingX OpenStack application requires PVCs.

There are two options for persistent storage backend: the Host-Based Ceph solution and the Rook Container-Based Ceph solution.

Note

Host-based Ceph will be deprecated and removed in a future release. Adopt Rook-Ceph for new deployments.

Add host-based Ceph backend:

~(keystone_admin)$ system storage-backend-add ceph --confirmed

Add an OSD on controller-0 for host-based Ceph:

# List host’s disks and identify disks you want to use for CEPH |OSDs|, taking note of their UUID
# By default, the system uses /dev/sda as the system disk, so you cannot use it for OSD.
~(keystone_admin)$ system host-disk-list controller-0

# Add disk as an OSD storage
~(keystone_admin)$ system host-stor-add controller-0 osd <disk-uuid>

# List OSD storage devices
~(keystone_admin)$ system host-stor-list controller-0

Note

Each deployment model enforces a different structure for the Rook Ceph cluster and its integration with the platform.

For more details see StarlingX Storage Configuration and Management, Install Rook Ceph.

Check if the rook-ceph app is uploaded.

$ source /etc/platform/openrc
$ system application-list

Add Storage-Backend with Deployment Model.

~(keystone_admin)$ system storage-backend-add ceph-rook --deployment controller --confirmed
~(keystone_admin)$ system storage-backend-list
+--------------------------------------+-----------------+-----------+----------------------+----------+------------------+------------------------------------------------------+
| uuid                                 | name            | backend   | state                | task     | services         | capabilities                                         |
+--------------------------------------+-----------------+-----------+----------------------+----------+------------------+------------------------------------------------------+
| 45e3fedf-c386-4b8b-8405-882038dd7d13 | ceph-rook-store | ceph-rook | configuring-with-app | uploaded | block,filesystem | deployment_model: controller replication: 2          |
|                                      |                 |           |                      |          |                  | min_replication: 1                                   |
+--------------------------------------+-----------------+-----------+----------------------+----------+------------------+------------------------------------------------------+

Set up a controllerfs ceph-float filesystem.

~(keystone_admin)$ system controllerfs-add ceph-float=20

Set up a host-fs ceph filesystem on controller-0.

~(keystone_admin)$ system host-fs-add controller-0 ceph=20

List all the disks.

$ system host-disk-list controller-0
+--------------------------------------+-------------+------------+-------------+----------+---------------+--------------+---------------------+--------------------------------------------+
| uuid                                 | device_node | device_num | device_type | size_gib | available_gib | rpm          | serial_id           | device_path                                |
+--------------------------------------+-------------+------------+-------------+----------+---------------+--------------+---------------------+--------------------------------------------+
| 17408af3-e211-4e2b-8cf1-d2b6687476d5 | /dev/sda    | 2048       | HDD         | 292.968  | 0.0           | Undetermined | VBba52ec56-f68a9f2d | /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0 |
| cee99187-dac4-4a7b-8e58-f2d5bd48dcaf | /dev/sdb    | 2064       | HDD         | 9.765    | 0.0           | Undetermined | VBf96fa322-597194da | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0 |
| 0c6435af-805a-4a62-ad8e-403bf916f5cf | /dev/sdc    | 2080       | HDD         | 9.765    | 9.761         | Undetermined | VBeefed5ad-b4815f0d | /dev/disk/by-path/pci-0000:00:0d.0-ata-3.0 |
+--------------------------------------+-------------+------------+-------------+----------+---------------+--------------+---------------------+--------------------------------------------+

Choose empty disks and provide hostname and uuid to finish OSD configuration:

$ system host-stor-add controller-0 osd cee99187-dac4-4a7b-8e58-f2d5bd48dcaf

Wait for OSDs pod to be ready.

$ kubectl get pods -n rook-ceph
NAME                                                     READY   STATUS      RESTARTS      AGE
ceph-mgr-provision-78xjk                                0/1     Completed   0          4m31s
csi-cephfsplugin-572jc                                  2/2     Running     0          5m32s
csi-cephfsplugin-provisioner-5467c6c4f-t8x8d            5/5     Running     0          5m28s
csi-rbdplugin-2npb6                                     2/2     Running     0          5m32s
csi-rbdplugin-provisioner-fd84899c-k8wcw                5/5     Running     0          5m32s
rook-ceph-crashcollector-controller-0-589f5f774-d8sjz   1/1     Running     0          3m24s
rook-ceph-exporter-controller-0-5fd477bb8-c7nxh         1/1     Running     0          3m21s
rook-ceph-mds-kube-cephfs-a-cc647757-6p9j5              2/2     Running     0          3m25s
rook-ceph-mds-kube-cephfs-b-5b5845ff59-xprbb            2/2     Running     0          3m19s
rook-ceph-mgr-a-746fc4dd54-t8bcw                        2/2     Running     0          4m40s
rook-ceph-mon-a-b6c95db97-f5fqq                         2/2     Running     0          4m56s
rook-ceph-operator-69b5674578-27bn4                     1/1     Running     0          6m26s
rook-ceph-osd-0-7f5cd957b8-ppb99                        2/2     Running     0          3m52s
rook-ceph-osd-prepare-controller-0-vzq2d                0/1     Completed   0          4m18s
rook-ceph-provision-zcs89                               0/1     Completed   0          101s
rook-ceph-tools-7dc9678ccb-v2gps                        1/1     Running     0          6m2s
stx-ceph-manager-664f8585d8-wzr4v                       1/1     Running     0          4m31s

Check ceph cluster health.

$ ceph -s
   cluster:
      id:     75c8f017-e7b8-4120-a9c1-06f38e1d1aa3
      health: HEALTH_OK

   services:
      mon: 1 daemons, quorum a (age 32m)
      mgr: a(active, since 30m)
      mds: 1/1 daemons up, 1 hot standby
      osd: 1 osds: 1 up (since 30m), 1 in (since 31m)

   data:
      volumes: 1/1 healthy
      pools:   4 pools, 113 pgs
      objects: 22 objects, 595 KiB
      usage:   27 MiB used, 9.7 GiB / 9.8 GiB avail
      pgs:     113 active+clean

   io:
      client:   852 B/s rd, 1 op/s rd, 0 op/s wr

Due to the additional OpenStack services’ containers running on the controller host, increase the Docker filesystem size from the default 30G to 60G.

# check existing size of docker fs
~(keystone_admin)$ system host-fs-list controller-0
# check available space (Avail Size (GiB)) in cgts-vg LVG where docker fs is located
~(keystone_admin)$ system host-lvg-list controller-0
# if existing docker fs size + cgts-vg available space is less than
# 80G, you will need to add a new disk to cgts-vg.

   # Get device path of BOOT DISK
   ~(keystone_admin)$ system host-show controller-0 | fgrep rootfs

   # Get UUID of ROOT DISK by listing disks
   ~(keystone_admin)$ system host-disk-list controller-0

   # Add new disk to 'cgts-vg' local volume group
   ~(keystone_admin)$ system host-pv-add controller-0 cgts-vg <DISK_UUID>
   ~(keystone_admin)$ sleep 10    # wait for disk to be added

   # Confirm the available space and increased number of physical
   # volumes added to the cgts-vg volume group
   ~(keystone_admin)$ system host-lvg-list controller-0

   # Increase docker filesystem to 60G
   ~(keystone_admin)$ system host-fs-modify controller-0 docker=60

Once controller-0 is operational, use it to:

PXE boot the remaining servers.
Configure and unlock all additional nodes in the deployment.

Install Software on Controller-1 Node¶

Power on the controller-1 server and use your system’s BIOS boot menu to force a network boot. As the system starts, the console prompts you to set the node personality.
Power on the controller-1 server and force it to network boot with the appropriate BIOS boot options for your particular server.
1. On the host, power on the controller-1 virtual server, duplex-controller-1. It will automatically attempt to network boot over the management network:
$ virsh start duplex-controller-1
1. Attach to the console of virtual controller-1:
$ virsh console duplex-controller-1

On the console of controller-0, list hosts to see newly discovered controller-1 host (hostname=None):

~(keystone_admin)$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname     | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1  | controller-0 | controller  | unlocked       | enabled     | available    |
| 2  | None         | None        | locked         | disabled    | offline      |
+----+--------------+-------------+----------------+-------------+--------------+

Use the host id to set the personality of this host to ‘controller’:
```
~(keystone_admin)$ system host-update 2 personality=controller
```

Allow the software installation to complete, controller-1 to reboot, and its status to update to locked/disabled/online in the ‘system host-list’. This process typically takes 5-10 minutes depending on the performance of the host machine.

~(keystone_admin)$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname     | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1  | controller-0 | controller  | unlocked       | enabled     | available    |
| 2  | controller-1 | controller  | locked         | disabled    | online       |
+----+--------------+-------------+----------------+-------------+--------------+

Configure Controller-1¶

Configure the OAM interface of controller-1 and specify the attached network as “oam”.

The following example configures the OAM interface on a physical untagged ethernet port, use the OAM port name that is applicable to your deployment environment, for example eth0:

~(keystone_admin)$ OAM_IF=<OAM-PORT>
~(keystone_admin)$ system host-if-modify controller-1 $OAM_IF -c platform
~(keystone_admin)$ system interface-network-assign controller-1 $OAM_IF oam

~(keystone_admin)$ OAM_IF=enp7s1
~(keystone_admin)$ system host-if-modify controller-1 $OAM_IF -c platform
~(keystone_admin)$ system interface-network-assign controller-1 $OAM_IF oam

To configure a VLAN or aggregated ethernet interface, see Node Interfaces.

Configure MGMT interface on controller-1.

The network install procedure partially configures the MGMT interface, setting the port used for network install as the MGMT port and specifying the attached network of “mgmt”.

To complete the MGMT interface configuration, assign the “cluster-host” network to the MGMT port by running the following command.
```
~(keystone_admin)$ system interface-network-assign controller-1 mgmt0 cluster-host
```

Additional Optional Configurations¶

If you need to install the StarlingX OpenStack application (stx-openstack), see OpenStack-Specific Host Configuration.

Configure PCI-SRIOV Interfaces (Optional)¶

This step ensures that the platform recognizes SR-IOV hardware and makes those interfaces available for subsequent configuration, but it does not fully provision or configure them for workloads. You can complete the detailed configuration, such as assigning virtual functions (VFs) and enabling SR-IOV for containers or applications, later using the PCI-SRIOV interface support section in Node Management (Kubernetes).

Configure PCI-SRIOV interfaces for controller-1.

Configure the PCI-SR-IOV interfaces for controller-1.

~(keystone_admin)$ export NODE=controller-1

# List inventoried host’s ports and identify ports to be used as ‘pci-sriov’ interfaces,
# based on displayed linux port name, pci address and device type.
~(keystone_admin)$ system host-port-list ${NODE}

# List host’s auto-configured 'ethernet' interfaces,
# find the interfaces corresponding to the ports identified in previous step, and
# take note of their UUID
~(keystone_admin)$ system host-if-list -a ${NODE}

# Modify configuration for these interfaces
# Configuring them as 'pci-sriov' class interfaces, MTU of 1500 and named sriov#
~(keystone_admin)$ system host-if-modify -m 1500 -n sriov0 -c pci-sriov ${NODE} <sriov0-if-uuid> -N <num_vfs>
~(keystone_admin)$ system host-if-modify -m 1500 -n sriov1 -c pci-sriov ${NODE} <sriov1-if-uuid> -N <num_vfs>

# If not already created, create Data Networks that the 'pci-sriov' interfaces
# will be connected to
~(keystone_admin)$ DATANET0='datanet0'
~(keystone_admin)$ DATANET1='datanet1'

# Assign Data Networks to PCI-SRIOV Interfaces
~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <sriov0-if-uuid> ${DATANET0}
~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <sriov1-if-uuid> ${DATANET1}

To enable SR-IOV network attachments for the above interfaces in Kubernetes-hosted application containers:

Configure the Kubernetes SR-IOV device plugin.

~(keystone_admin)$ system host-label-assign controller-1 sriovdp=enabled

If planning on running DPDK in Kubernetes hosted application containers on this host, configure the number of 1G Huge pages required on both NUMA nodes.

# assign 10x 1G huge page on processor/numa-node 0 on controller-1 to applications
~(keystone_admin)$ system host-memory-modify -f application controller-1 0 -1G 10

# assign 10x 1G huge page on processor/numa-node 1 on controller-1 to applications
~(keystone_admin)$ system host-memory-modify -f application controller-1 1 -1G 10

Configure the Kubernetes SR-IOV device plugin.

~(keystone_admin)$ system host-label-assign controller-1 sriovdp=enabled

Unlock Controller-1¶

Use this procedure to complete the configuration of controller-1 and unlock it to provide service.

system host-unlock controller-1

Controller-1 reboots to apply the configuration changes and return to service. This process typically takes 5-10 minutes, depending on the performance of the host machine.

Rook Ceph Storage Backend Configuration (Optional)¶

Use this procedure to configure host-specific details when setting up a Ceph-based persistent storage backend.

Add an OSD on controller-1 for host-based Ceph:

# List host’s disks and identify disks you want to use for CEPH OSDs, taking note of their UUID
# By default, /dev/sda is the system disk and cannot be used for OSD.
~(keystone_admin)$ system host-disk-list controller-1

# Add disk as an OSD storage
~(keystone_admin)$ system host-stor-add controller-1 osd <disk-uuid>

# List OSD storage devices
~(keystone_admin)$ system host-stor-list controller-1

Note

Each deployment model enforces a different structure for the Rook Ceph cluster and its integration with the platform.

For more details see StarlingX Storage Configuration and Management, Install Rook Ceph.

Check if the rook-ceph app is uploaded.

~(keystone_admin)$ source /etc/platform/openrc
~(keystone_admin)$ system application-list

Set up a host-fs ceph filesystem on controller-1.

~(keystone_admin)$ system host-fs-add controller-1 ceph=20

List all the disks.

~(keystone_admin)$ system host-disk-list controller-0
+--------------------------------------+-------------+------------+-------------+----------+---------------+--------------+---------------------+--------------------------------------------+
| uuid                                 | device_node | device_num | device_type | size_gib | available_gib | rpm          | serial_id           | device_path                                |
+--------------------------------------+-------------+------------+-------------+----------+---------------+--------------+---------------------+--------------------------------------------+
| 7ce699f0-12dd-4416-ae43-00d3877450f7 | /dev/sda    | 2048       | HDD         | 292.968  | 0.0           | Undetermined | VB0e18230e-6a8780e1 | /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0 |
| bfb83b6f-61e2-4f9f-a87d-ecae938b7e78 | /dev/sdb    | 2064       | HDD         | 9.765    | 9.765         | Undetermined | VB144f1510-14f089fd | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0 |
| 937cfabc-8447-4dbd-8ca3-062a46953023 | /dev/sdc    | 2080       | HDD         | 9.765    | 9.761         | Undetermined | VB95057d1c-4ee605c2 | /dev/disk/by-path/pci-0000:00:0d.0-ata-3.0 |
+--------------------------------------+-------------+------------+-------------+----------+---------------+--------------+---------------------+--------------------------------------------+

~(keystone_admin)$ system host-disk-list controller-1
+--------------------------------------+-------------+------------+-------------+----------+---------------+--------------+---------------------+--------------------------------------------+
| uuid                                 | device_node | device_num | device_type | size_gib | available_gib | rpm          | serial_id           | device_path                                |
+--------------------------------------+-------------+------------+-------------+----------+---------------+--------------+---------------------+--------------------------------------------+
| 52c8e1b5-0551-4748-a7a0-27b9c028cf9d | /dev/sda    | 2048       | HDD         | 292.968  | 0.0           | Undetermined | VB9b565509-a2edaa2e | /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0 |
| 93020ce0-249e-4db3-b8c3-6c7e8f32713b | /dev/sdb    | 2064       | HDD         | 9.765    | 9.765         | Undetermined | VBa08ccbda-90190faa | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0 |
| dc0ec403-67f8-40bf-ada0-6fcae3ed76da | /dev/sdc    | 2080       | HDD         | 9.765    | 9.761         | Undetermined | VB16244caf-ab36d36c | /dev/disk/by-path/pci-0000:00:0d.0-ata-3.0 |
+--------------------------------------+-------------+------------+-------------+----------+---------------+--------------+---------------------+--------------------------------------------+

Choose empty disks and provide hostname and uuid to finish OSD configuration:

~(keystone_admin)$ system host-stor-add controller-0 osd bfb83b6f-61e2-4f9f-a87d-ecae938b7e78
~(keystone_admin)$ system host-stor-add controller-1 osd 93020ce0-249e-4db3-b8c3-6c7e8f32713b

Wait for OSDs pod to be ready.

$ kubectl get pods -n rook-ceph
NAME                                                     READY   STATUS      RESTARTS      AGE
ceph-mgr-provision-w55rh                                 0/1     Completed   0             10m
csi-cephfsplugin-8j7xz                                   2/2     Running     1 (11m ago)   12m
csi-cephfsplugin-lmmg2                                   2/2     Running     0             12m
csi-cephfsplugin-provisioner-5467c6c4f-mktqg             5/5     Running     0             12m
csi-rbdplugin-8m8kd                                      2/2     Running     1 (11m ago)   12m
csi-rbdplugin-provisioner-fd84899c-kpv4q                 5/5     Running     0             12m
csi-rbdplugin-z92sk                                      2/2     Running     0             12m
mon-float-post-install-sw8qb                             0/1     Completed   0             6m5s
mon-float-pre-install-nfj5b                              0/1     Completed   0             6m40s
rook-ceph-crashcollector-controller-0-589f5f774-sp6zf    1/1     Running     0             7m49s
rook-ceph-crashcollector-controller-1-68d66b9bff-zwgp9   1/1     Running     0             7m36s
rook-ceph-exporter-controller-0-5fd477bb8-jgsdk          1/1     Running     0             7m44s
rook-ceph-exporter-controller-1-6f5d8695b9-ndksh         1/1     Running     0             7m32s
rook-ceph-mds-kube-cephfs-a-5f584f4bc-tbk8q              2/2     Running     0             7m49s
rook-ceph-mgr-a-6845774cb5-lgjjd                         3/3     Running     0             9m1s
rook-ceph-mgr-b-7fccfdf64d-4pcmc                         3/3     Running     0             9m1s
rook-ceph-mon-a-69fd4895c7-2lfz4                         2/2     Running     0             11m
rook-ceph-mon-b-7fd8cbb997-f84ng                         2/2     Running     0             11m
rook-ceph-mon-float-85c4cbb7f9-k7xwj                     2/2     Running     0             6m27s
rook-ceph-operator-69b5674578-z456r                      1/1     Running     0             13m
rook-ceph-osd-0-5f59b5bb7b-mkwrg                         2/2     Running     0             8m17s
rook-ceph-osd-prepare-controller-0-rhjgx                 0/1     Completed   0             8m38s
rook-ceph-provision-5glpc                                0/1     Completed   0             6m17s
rook-ceph-tools-7dc9678ccb-nmwwc                         1/1     Running     0             12m
stx-ceph-manager-664f8585d8-5lt8c                        1/1     Running     0             10m

Check ceph cluster health.

$ ceph -s
   cluster:
      id:     c18dfe3a-9b72-46e4-bb6e-6984f131598f
      health: HEALTH_OK

   services:
      mon: 2 daemons, quorum a,b (age 9m)
      mgr: a(active, since 6m), standbys: b
      mds: 1/1 daemons up, 1 hot standby
      osd: 2 osds: 2 up (since 7m), 2 in (since 7m)

   data:
      volumes: 1/1 healthy
      pools:   4 pools, 113 pgs
      objects: 25 objects, 594 KiB
      usage:   72 MiB used, 19 GiB / 20 GiB avail
      pgs:     113 active+clean

   io:
      client:   1.2 KiB/s rd, 2 op/s rd, 0 op/s wr

Note

Controller-0 and controller-1 use IP multicast messaging for synchronization. If loss of synchronization occurs a few minutes after controller-1 becomes available, ensure that the switches and other devices on the management and infrastructure networks are configured with appropriate settings.

In particular, if IGMP snooping is enabled on ToR switches, then a device acting as an IGMP querier is required on the network (on the same VLAN) to prevent nodes from being dropped from the multicast group. The IGMP querier periodically sends IGMP queries to all nodes on the network, and each node sends an IGMP join or report in response. Without an IGMP querier, the nodes do not periodically send IGMP join messages after the initial join sent when the link first goes up, and they are eventually dropped from the multicast group.

Extend Capacity with Worker Nodes (Optional)¶

This section describes the steps to extend capacity with worker nodes on a StarlingX All-in-One Duplex deployment configuration.

Install Software on Worker nodes¶

Power on the worker node servers.
Power on the worker node servers and force them to network boot with the appropriate BIOS boot options for your particular server.
1. On the host, power on the worker-0 and worker-1 virtual servers.
  
  They will automatically attempt to network boot over the management network:
  $ virsh start duplex-worker-0 $ virsh start duplex-worker-1
2. Attach to the consoles of worker-0 and worker-1.
  $ virsh console duplex-worker-0 $ virsh console duplex-worker-1~(keystone_admin)$
As the worker nodes boot, a message appears on their console instructing you to configure the personality of the node.

On the console of controller-0, list hosts to see newly discovered worker node hosts (hostname=None):

~(keystone_admin)$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname     | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1  | controller-0 | controller  | unlocked       | enabled     | available    |
| 2  | controller-1 | controller  | unlocked       | enabled     | available    |
| 3  | None         | None        | locked         | disabled    | offline      |
| 4  | None         | None        | locked         | disabled    | offline      |
+----+--------------+-------------+----------------+-------------+--------------+

Using the host id, set the personality of this host to ‘worker’:
```
~(keystone_admin)$ system host-update 3 personality=worker hostname=worker-0
~(keystone_admin)$ system host-update 4 personality=worker hostname=worker-1
```
This initiates the install of software on worker nodes. This can take 5-10 minutes, depending on the performance of the host machine.

Note

A node with Edgeworker personality is also available. See Deploy Edgeworker Nodes for details.

Wait for the install of software on the worker nodes to complete, for the worker nodes to reboot, and for both to show as locked/disabled/online in ‘system host-list’.

~(keystone_admin)$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname     | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1  | controller-0 | controller  | unlocked       | enabled     | available    |
| 2  | controller-1 | controller  | unlocked       | enabled     | available    |
| 3  | worker-0     | worker      | locked         | disabled    | online       |
| 4  | worker-1     | worker      | locked         | disabled    | online       |
+----+--------------+-------------+----------------+-------------+--------------+

Configure Worker Nodes¶

The MGMT interfaces are partially set up by the network install procedure; configuring the port used for network install as the MGMT port and specifying the attached network of “mgmt”.

Complete the MGMT interface configuration of the worker nodes by specifying the attached network of “cluster-host”.
```
for NODE in worker-0 worker-1; do
   system interface-network-assign $NODE mgmt0 cluster-host
done
```

OpenStack-specific host configuration¶

Important

These steps are required only if the StarlingX OpenStack application (|prefix|-openstack) will be installed.

For OpenStack only: Assign OpenStack host labels to the worker nodes in support of installing the stx-openstack manifest and helm-charts later.
```
for NODE in worker-0 worker-1; do
   system host-label-assign $NODE  openstack-compute-node=enabled
   kubectl taint nodes $NODE openstack-compute-node:NoSchedule
   system host-label-assign $NODE  openvswitch=enabled
   system host-label-assign $NODE  sriov=enabled
   system host-label-assign $NODE  dpdk=enabled
done
```
Note

When using OVS, the DPDK label must not be applied, that is dpdk=enabled is not applicable to OVS. Otherwise, the open vSwitch chart will be disabled.
No additional steps are required.
For OpenStack only: Configure the host settings for the vSwitch.
If using OVS-DPDK vswitch, run the following commands: Default recommendation for worker node is to use two cores on numa-node 0 for OVS-DPDK vSwitch; physical NICs are typically on first numa-node. This should have been automatically configured, if not run the following command.
for NODE in worker-0 worker-1; do # assign 2 cores on processor/numa-node 0 on worker-node to vswitch ~(keystone_admin)$ system host-cpu-modify -f vswitch -p0 2 $NODE done
When using OVS-DPDK, configure 1G of huge pages for vSwitch memory on each NUMA node on the host. It is recommended to configure 1x 1G huge page (-1G 1) for vSwitch memory on each NUMA node on the host.

However, due to a limitation with Kubernetes, only a single huge page size is supported on any one host. If your application VMs require 2M huge pages, then configure 500x 2M huge pages (-2M 500) for vSwitch memory on each NUMA node on the host.
for NODE in worker-0 worker-1; do # assign 1x 1G huge page on processor/numa-node 0 on worker-node to vswitch ~(keystone_admin)$ system host-memory-modify -f vswitch -1G 1 $NODE 0 # assign 1x 1G huge page on processor/numa-node 0 on worker-node to vswitch ~(keystone_admin)$ system host-memory-modify -f vswitch -1G 1 $NODE 1 done
Important

VMs created in an OVS-DPDK environment must be configured to use huge pages to enable networking and must use a flavor with property: hw:mem_page_size=large.

Configure the huge pages for VMs in an OVS-DPDK environment on this host, assuming 1G huge page size is being used on this host, with the following commands:
for NODE in worker-0 worker-1; do # assign 10x 1G huge page on processor/numa-node 0 on worker-node to applications ~(keystone_admin)$ system host-memory-modify -f application -1G 10 $NODE 0 # assign 10x 1G huge page on processor/numa-node 1 on worker-node to applications ~(keystone_admin)$ system host-memory-modify -f application -1G 10 $NODE 1 done
No additional configuration is required for the OVS vswitch in virtual environment.

For OpenStack only: Setup disk partition for nova-local volume group, needed for stx-openstack nova ephemeral disks.

for NODE in worker-0 worker-1; do
   ~(keystone_admin)$ system host-lvg-add ${NODE} nova-local

   # Get UUID of DISK to create PARTITION to be added to ‘nova-local’ local volume group
   # CEPH OSD Disks can NOT be used
   # For best performance, do NOT use system/root disk, use a separate physical disk.

   # List host’s disks and take note of UUID of disk to be used
   ~(keystone_admin)$ system host-disk-list ${NODE}
   # ( if using ROOT DISK, select disk with device_path of
   #   'system host-show ${NODE} | fgrep rootfs'   )

   # Create new PARTITION on selected disk, and take note of new partition’s ‘uuid’ in response
   # The size of the PARTITION needs to be large enough to hold the aggregate size of
   # all nova ephemeral disks of all VMs that you want to be able to host on this host,
   # but is limited by the size and space available on the physical disk you chose above.
   # The following example uses a small PARTITION size such that you can fit it on the
   # root disk, if that is what you chose above.
   # Additional PARTITION(s) from additional disks can be added later if required.
   PARTITION_SIZE=30

   ~(keystone_admin)$ system host-disk-partition-add -t lvm_phys_vol ${NODE} <disk-uuid> ${PARTITION_SIZE}

   # Add new partition to ‘nova-local’ local volume group
   ~(keystone_admin)$ system host-pv-add ${NODE} nova-local <NEW_PARTITION_UUID>
   sleep 2
done

For OpenStack only: Configure data interfaces for worker nodes. Data class interfaces are vswitch interfaces used by vswitch to provide VM virtio vNIC connectivity to OpenStack Neutron Tenant Networks on the underlying assigned Data Network.

Important

A compute-labeled worker host MUST have at least one Data class interface.

Configure the data interfaces for worker nodes.

# Execute the following lines with
~(keystone_admin)$ export NODE=worker-0

# and then repeat with
~(keystone_admin)$ export NODE=worker-1

  # List inventoried host’s ports and identify ports to be used as `data` interfaces,
  # based on displayed linux port name, pci address and device type.
  ~(keystone_admin)$ system host-port-list ${NODE}

  # List host’s auto-configured ‘ethernet’ interfaces,
  # find the interfaces corresponding to the ports identified in previous step, and
  # take note of their UUID
  ~(keystone_admin)$ system host-if-list -a ${NODE}

  # Modify configuration for these interfaces
  # Configuring them as ‘data’ class interfaces, MTU of 1500 and named data#
  ~(keystone_admin)$ system host-if-modify -m 1500 -n data0 -c data ${NODE} <data0-if-uuid>
  ~(keystone_admin)$ system host-if-modify -m 1500 -n data1 -c data ${NODE} <data1-if-uuid>

  # Create Data Networks that vswitch 'data' interfaces will be connected to
  ~(keystone_admin)$ DATANET0='datanet0'
  ~(keystone_admin)$ DATANET1='datanet1'
  ~(keystone_admin)$ system datanetwork-add ${DATANET0} vlan
  ~(keystone_admin)$ system datanetwork-add ${DATANET1} vlan

  # Assign Data Networks to Data Interfaces
  ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <data0-if-uuid> ${DATANET0}
  ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <data1-if-uuid> ${DATANET1}

# Execute the following lines with
 ~(keystone_admin)$ export NODE=worker-0

# and then repeat with
 ~(keystone_admin)$ export NODE=worker-1

 ~(keystone_admin)$ DATA0IF=eth1000
 ~(keystone_admin)$ DATA1IF=eth1001
 ~(keystone_admin)$ PHYSNET0='physnet0'
 ~(keystone_admin)$ PHYSNET1='physnet1'
 ~(keystone_admin)$ SPL=/tmp/tmp-system-port-list
 ~(keystone_admin)$ SPIL=/tmp/tmp-system-host-if-list
 ~(keystone_admin)$ system host-port-list ${NODE} --nowrap > ${SPL}
 ~(keystone_admin)$ system host-if-list -a ${NODE} --nowrap > ${SPIL}
 ~(keystone_admin)$ DATA0PCIADDR=$(cat $SPL | grep $DATA0IF | awk '{print $8}')
 ~(keystone_admin)$ DATA1PCIADDR=$(cat $SPL | grep $DATA1IF | awk '{print $8}')
 ~(keystone_admin)$ DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}')
 ~(keystone_admin)$ DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}')
 ~(keystone_admin)$ DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}')
 ~(keystone_admin)$ DATA1PORTNAME=$(cat  $SPL | grep ${DATA1PCIADDR} | awk '{print $4}')
 ~(keystone_admin)$ DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}')
 ~(keystone_admin)$ DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}')

 ~(keystone_admin)$ system datanetwork-add ${PHYSNET0} vlan
 ~(keystone_admin)$ system datanetwork-add ${PHYSNET1} vlan

 ~(keystone_admin)$ system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID}
 ~(keystone_admin)$ system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID}
 ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0}
 ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1}

Optionally Configure PCI-SRIOV Interfaces¶

Optionally, configure pci-sriov interfaces for worker nodes.

This step is optional for Kubernetes. Do this step if using SR-IOV network attachments in hosted application containers.

This step is optional for OpenStack. Do this step if using SR-IOV vNICs in hosted application VMs. Note that pci-sriov interfaces can have the same Data Networks assigned to them as vswitch data interfaces.

Configure the pci-sriov interfaces for worker nodes.

# Execute the following lines with
~(keystone_admin)$ export NODE=worker-0
# and then repeat with
~(keystone_admin)$ export NODE=worker-1

  # List inventoried host’s ports and identify ports to be used as ‘pci-sriov’ interfaces,
  # based on displayed linux port name, pci address and device type.
  ~(keystone_admin)$ system host-port-list ${NODE}

  # List host’s auto-configured ‘ethernet’ interfaces,
  # find the interfaces corresponding to the ports identified in previous step, and
  # take note of their UUID
  ~(keystone_admin)$ system host-if-list -a ${NODE}

  # Modify configuration for these interfaces
  # Configuring them as ‘pci-sriov’ class interfaces, MTU of 1500 and named sriov#
  ~(keystone_admin)$ system host-if-modify -m 1500 -n sriov0 -c pci-sriov ${NODE} <sriov0-if-uuid> -N <num_vfs>
  ~(keystone_admin)$ system host-if-modify -m 1500 -n sriov1 -c pci-sriov ${NODE} <sriov1-if-uuid> -N <num_vfs>

  # If not already created, create Data Networks that the 'pci-sriov'
  # interfaces will be connected to
  ~(keystone_admin)$ DATANET0='datanet0'
  ~(keystone_admin)$ DATANET1='datanet1'
  ~(keystone_admin)$ system datanetwork-add ${DATANET0} vlan
  ~(keystone_admin)$ system datanetwork-add ${DATANET1} vlan

  # Assign Data Networks to PCI-SRIOV Interfaces
  ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <sriov0-if-uuid> ${DATANET0}
  ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <sriov1-if-uuid> ${DATANET1}

For Kubernetes only To enable using SR-IOV network attachments for the above interfaces in Kubernetes hosted application containers:

Configure the Kubernetes SR-IOV device plugin.

for NODE in worker-0 worker-1; do
   system host-label-assign $NODE sriovdp=enabled
done

If planning on running DPDK in Kubernetes hosted application containers on this host, configure the number of 1G Huge pages required on both NUMA nodes.

for NODE in worker-0 worker-1; do
   # assign 10x 1G huge page on processor/numa-node 0 on worker-node to applications
   ~(keystone_admin)$ system host-memory-modify -f application $NODE 0 -1G 10

   # assign 10x 1G huge page on processor/numa-node 1 on worker-node to applications
   ~(keystone_admin)$ system host-memory-modify -f application $NODE 1 -1G 10
done

Configure the Kubernetes SR-IOV device plugin.

for NODE in worker-0 worker-1; do
   system host-label-assign $NODE sriovdp=enabled
done

Unlock Worker Nodes¶

Unlock worker nodes in order to bring them into service:

for NODE in worker-0 worker-1; do
   system host-unlock $NODE
done

The worker nodes will reboot to apply configuration changes and come into service. This can take 5-10 minutes, depending on the performance of the host machine.

Post-Installation Tasks¶

After completing the installation, perform the following post-installation tasks to fully configure and prepare the system for use. For more information, see Post-Installation Tasks.