OpenStack-Specific Host Configuration¶
:
Important
These steps are required only if the StarlingX OpenStack application (stx-openstack) will be installed.
You can use the same OpenStack procedure for AIO-DX. Repeat the steps for both controller-0 and controller-1. When running the procedure on controller-1, update the commands accordingly by replacing controller-0 with controller-1.
Assign OpenStack host labels to controller-0 in support of installing the stx-openstack manifest and helm-charts later.
~(keystone_admin)$ system host-label-assign controller-0 openstack-control-plane=enabled ~(keystone_admin)$ system host-label-assign controller-0 openstack-compute-node=enabled ~(keystone_admin)$ system host-label-assign controller-0 openvswitch=enabled
Note
If you have a NIC that supports SR-IOV, then you can enable it by using the following:
~(keystone_admin)$ system host-label-assign controller-0 sriov=enabled
Due to the additional OpenStack services running on the AIO controller platform cores, additional platform cores may be required.
A minimum of 4 platform cores are required, 6 platform cores are recommended.
Increase the number of platform cores with the following commands. This example assigns 6 cores on processor/numa-node 0 on controller-0 to platform.
~(keystone_admin)$ system host-cpu-modify -f platform -p0 6 controller-0
A minimum of 4 platform cores are required, 6 platform cores are recommended.
Increase the number of platform cores with the following commands. This example assigns 6 cores on processor/numa-node 0
on controller-0 to platform.
~(keystone_admin)$ system host-cpu-modify -f platform -p0 6 controller-0
The VMs being used for hosts only have 4 cores; 2 for platform and 2 for VMs. There are no additional cores available for platform in this scenario.
Due to the additional OpenStack services’ containers running on the controller host, the size of the Docker filesystem needs to be increased from the default size of 30G to 60G.
# check existing size of docker fs system host-fs-list controller-0 # check available space (Avail Size (GiB)) in cgts-vg LVG where docker fs is located system host-lvg-list controller-0 # if existing docker fs size + cgts-vg available space is less than # 80G, you will need to add a new disk to cgts-vg. # Get device path of BOOT DISK system host-show controller-0 | fgrep rootfs # Get UUID of ROOT DISK by listing disks system host-disk-list controller-0 # Add new disk to 'cgts-vg' local volume group system host-pv-add controller-0 cgts-vg <DISK_UUID> sleep 10 # wait for disk to be added # Confirm the available space and increased number of physical # volumes added to the cgts-vg colume group system host-lvg-list controller-0 # Increase docker filesystem to 60G system host-fs-modify controller-0 docker=60
Configure the host setting for the vSwitch.
StarlingX has OVS (kernel-based) vSwitch configured as default, which:
Runs in a container; defined within the helm charts of stx-openstack manifest.
Shares the core(s) assigned to the platform.
If you require better performance, OVS-DPDK (OVS with the Data Plane Development Kit, which is supported only on bare metal hardware) should be used:
Runs directly on the host (it is not containerized). Requires that at least 1 core be assigned/dedicated to the vSwitch function.
To deploy the default containerized OVS:
~(keystone_admin)$ system modify --vswitch_type none
This does not run any vSwitch directly on the host, instead, it uses the containerized OVS defined in the helm charts of stx-openstack manifest.
To deploy OVS-DPDK, run the following command:
~(keystone_admin)$ system modify --vswitch_type ovs-dpdk
Default recommendation for an AIO-controller is to use a single core for OVS-DPDK vSwitch.
# assign 1 core on processor/numa-node 0 on controller-0 to vswitch ~(keystone_admin)$ system host-cpu-modify -f vswitch -p0 1 controller-0
When using OVS-DPDK, configure 1G of huge pages for vSwitch memory on each NUMA node on the host. It is recommended to configure 1x 1G huge page (-1G 1) for vSwitch memory on each NUMA node on the host.
However, due to a limitation with Kubernetes, only a single huge page size is supported on any one host. If your application VMs require 2M huge pages, then configure 500x 2M huge pages (-2M 500) for vSwitch memory on each NUMA node on the host.
# Assign 1x 1G huge page on processor/numa-node 0 on controller-0 to vswitch ~(keystone_admin)$ system host-memory-modify -f vswitch -1G 1 controller-0 0 # Assign 1x 1G huge page on processor/numa-node 1 on controller-0 to vswitch ~(keystone_admin)$ system host-memory-modify -f vswitch -1G 1 controller-0 1
Important
VMs created in an OVS-DPDK environment must be configured to use huge pages to enable networking and must use a flavor with property: hw:mem_page_size=large Configure the huge pages for VMs in an OVS-DPDK environment on this host, the following commands are an example that assumes that 1G huge page size is being used on this host:
# assign 1x 1G huge page on processor/numa-node 0 on controller-0 to applications ~(keystone_admin)$ system host-memory-modify -f application -1G 10 controller-0 0 # assign 1x 1G huge page on processor/numa-node 1 on controller-0 to applications ~(keystone_admin)$ system host-memory-modify -f application -1G 10 controller-0 1
Note
After controller-0 is unlocked, changing vswitch_type requires locking and unlocking controller-0 to apply the change.
To deploy OVS-DPDK, run the following command:
~(keystone_admin)$ system modify --vswitch_type ovs-dpdk
Default recommendation for an AIO-controller is to use a single core for OVS-DPDK vSwitch.
# assign 1 core on processor/numa-node 0 on controller-0 to vswitch ~(keystone_admin)$ system host-cpu-modify -f vswitch -p0 1 controller-0
Once vswitch_type is set to OVS-DPDK, any subsequent nodes created will default to automatically assigning 1 vSwitch core for AIO controllers and 2 vSwitch cores (both on numa-node 0; physical NICs are typically on first numa-node) for compute-labeled worker nodes.
When using OVS-DPDK, configure 1G of huge pages for vSwitch memory on each NUMA node on the host. It is recommended to configure 1x 1G huge page (-1G 1) for vSwitch memory on each NUMA node on the host.
However, due to a limitation with Kubernetes, only a single huge page size is supported on any one host. If your application VMs require 2M huge pages, then configure 500x 2M huge pages (-2M 500) for vSwitch memory on each NUMA node on the host.
# Assign 1x 1G huge page on processor/numa-node 0 on controller-0 to vswitch ~(keystone_admin)$ system host-memory-modify -f vswitch -1G 1 controller-0 0 # Assign 1x 1G huge page on processor/numa-node 1 on controller-0 to vswitch ~(keystone_admin)$ system host-memory-modify -f vswitch -1G 1 controller-0 1
Important
VMs created in an OVS-DPDK environment must be configured to use huge pages to enable networking and must use a flavor with property:
hw:mem_page_size=large.Configure the huge pages for VMs in an OVS-DPDK environment on this host, the following commands are an example that assumes that 1G huge page size is being used on this host:
# assign 1x 1G huge page on processor/numa-node 0 on controller-0 to applications ~(keystone_admin)$ system host-memory-modify -f application -1G 10 controller-0 0 # assign 1x 1G huge page on processor/numa-node 1 on controller-0 to applications ~(keystone_admin)$ system host-memory-modify -f application -1G 10 controller-0 1
Note
After controller-0 is unlocked, changing vswitch_type requires locking and unlocking controller-0 to apply the change.
The default vSwitch is the containerized OVS that is packaged with the
stx-openstackmanifest/helm-charts. StarlingX provides the option to use OVS-DPDK on the host, however, in the virtual environment OVS-DPDK is not supported, only OVS is supported. Therefore, simply use the default OVS vSwitch here.Add an instances filesystem or set up a disk based nova-local volume group, which is needed for stx-openstack nova ephemeral disks.
Note
Both cannot exist at the same time.
Add an ‘instances’ filesystem
~(keystone_admin)$ export NODE=controller-0 # Create ‘instances’ filesystem ~(keystone_admin)$ system host-fs-add ${NODE} instances=<size>
Or add a ‘nova-local’ volume group:
~(keystone_admin)$ export NODE=controller-0 # Create ‘nova-local’ local volume group ~(keystone_admin)$ system host-lvg-add ${NODE} nova-local # Get UUID of an unused DISK to to be added to the ‘nova-local’ volume # group. CEPH OSD Disks can NOT be used # List host’s disks and take note of UUID of disk to be used ~(keystone_admin)$ system host-disk-list ${NODE} # Add the unused disk to the ‘nova-local’ volume group ~(keystone_admin)$ system host-pv-add ${NODE} nova-local <DISK_UUID>
Note
Both cannot exist at the same time.
Add an ‘instances’ filesystem
~(keystone_admin)$ export NODE=controller-0 # Create ‘instances’ filesystem ~(keystone_admin)$ system host-fs-add ${NODE} instances=<size>
Or add a ‘nova-local’ volume group:
~(keystone_admin)$ export NODE=controller-0 # Create ‘nova-local’ local volume group ~(keystone_admin)$ system host-lvg-add ${NODE} nova-local # Get UUID of an unused DISK to to be added to the ‘nova-local’ volume # group. CEPH OSD Disks can NOT be used # List host’s disks and take note of UUID of disk to be used ~(keystone_admin)$ system host-disk-list ${NODE} # Add the unused disk to the ‘nova-local’ volume group ~(keystone_admin)$ system host-pv-add ${NODE} nova-local <DISK_UUID>
Set up an “instances” filesystem, which is needed for stx-openstack nova ephemeral disks.
~(keystone_admin)$ export NODE=controller-0 ~(keystone_admin)$ system host-fs-add ${NODE} instances=34
Configure data interfaces for controller-0.
Data class interfaces are vSwitch interfaces used by vSwitch to provide VM virtio vNIC connectivity to OpenStack Neutron Tenant Networks on the underlying assigned Data Network.
Important
A compute-labeled AIO-controller host MUST have at least one Data class interface.
Configure the data interfaces for controller-0.
~(keystone_admin)$ NODE=controller-0 # List inventoried host’s ports and identify ports to be used as ‘data’ interfaces, # based on displayed linux port name, pci address and device type. ~(keystone_admin)$ system host-port-list ${NODE} # List host’s auto-configured ‘ethernet’ interfaces, # find the interfaces corresponding to the ports identified in previous step, and # take note of their UUID ~(keystone_admin)$ system host-if-list -a ${NODE} # Modify configuration for these interfaces # Configuring them as ‘data’ class interfaces, MTU of 1500 and named data# ~(keystone_admin)$ system host-if-modify -m 1500 -n data0 -c data ${NODE} <data0-if-uuid> ~(keystone_admin)$ system host-if-modify -m 1500 -n data1 -c data ${NODE} <data1-if-uuid> # Create Data Networks that vswitch 'data' interfaces will be connected to ~(keystone_admin)$ DATANET0='datanet0' ~(keystone_admin)$ DATANET1='datanet1' ~(keystone_admin)$ system datanetwork-add ${DATANET0} vlan ~(keystone_admin)$ system datanetwork-add ${DATANET1} vlan # Assign Data Networks to Data Interfaces ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <data0-if-uuid> ${DATANET0} ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <data1-if-uuid> ${DATANET1}
~(keystone_admin)$ NODE=controller-0 # List inventoried host’s ports and identify ports to be used as ‘data’ interfaces, # based on displayed linux port name, pci address and device type. ~(keystone_admin)$ system host-port-list ${NODE} # List host’s auto-configured ‘ethernet’ interfaces, # find the interfaces corresponding to the ports identified in previous step, and # take note of their UUID ~(keystone_admin)$ system host-if-list -a ${NODE} # Modify configuration for these interfaces # Configuring them as ‘data’ class interfaces, MTU of 1500 and named data# ~(keystone_admin)$ system host-if-modify -m 1500 -n data0 -c data ${NODE} <data0-if-uuid> ~(keystone_admin)$ system host-if-modify -m 1500 -n data1 -c data ${NODE} <data1-if-uuid> # Create Data Networks that vswitch 'data' interfaces will be connected to ~(keystone_admin)$ DATANET0='datanet0' ~(keystone_admin)$ DATANET1='datanet1' # Assign Data Networks to Data Interfaces ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <data0-if-uuid> ${DATANET0} ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <data1-if-uuid> ${DATANET1}
~(keystone_admin)$ DATA0IF=eth1000 ~(keystone_admin)$ DATA1IF=eth1001 ~(keystone_admin)$ export NODE=controller-0 ~(keystone_admin)$ PHYSNET0='physnet0' ~(keystone_admin)$ PHYSNET1='physnet1' ~(keystone_admin)$ SPL=/tmp/tmp-system-port-list ~(keystone_admin)$ SPIL=/tmp/tmp-system-host-if-list ~(keystone_admin)$ system host-port-list ${NODE} --nowrap > ${SPL} ~(keystone_admin)$ system host-if-list -a ${NODE} --nowrap > ${SPIL} ~(keystone_admin)$ DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}') ~(keystone_admin)$ DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}') ~(keystone_admin)$ DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}') ~(keystone_admin)$ DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}') ~(keystone_admin)$ DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}') ~(keystone_admin)$ DATA1PORTNAME=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $4}') ~(keystone_admin)$ DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}') ~(keystone_admin)$ DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}') ~(keystone_admin)$ system datanetwork-add ${PHYSNET0} vlan ~(keystone_admin)$ system datanetwork-add ${PHYSNET1} vlan ~(keystone_admin)$ system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID} ~(keystone_admin)$ system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID} ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0} ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1}
:
Important
These steps are required only if the StarlingX OpenStack application (stx-openstack) will be installed.
Assign OpenStack host labels to controller-1 in support of installing the stx-openstack manifest and helm-charts later.
~(keystone_admin)$ system host-label-assign controller-1 openstack-control-plane=enabled ~(keystone_admin)$ system host-label-assign controller-1 openstack-compute-node=enabled ~(keystone_admin)$ system host-label-assign controller-1 openvswitch=enabledNote
If you have a NIC that supports SR-IOV, then you can enable it by using the following:
~(keystone_admin)$ system host-label-assign controller-1 sriov=enabledDue to the additional openstack services running on the AIO controller platform cores, additional cores may be required.
Increase the number of platform cores with the following commands:
# assign 6 cores on processor/numa-node 0 on controller-1 to platform ~(keystone_admin)$ system host-cpu-modify -f platform -p0 6 controller-1The VMs being used for hosts only have 4 cores; 2 for platform and 2 for VMs; there are not additional cores available for platform in this scenario.
Due to the additional openstack services’ containers running on the controller host, the size of the docker filesystem needs to be increased from the default size of 30G to 60G.
# check existing size of docker fs ~(keystone_admin)$ system host-fs-list controller-1 # check available space (Avail Size (GiB)) in cgts-vg LVG where docker fs is located ~(keystone_admin)$ system host-lvg-list controller-1 # if existing docker fs size + cgts-vg available space is less than # 80G, you will need to add a new disk to cgts-vg. # Get device path of BOOT DISK ~(keystone_admin)$ system host-show controller-1 | fgrep rootfs # Get UUID of ROOT DISK by listing disks ~(keystone_admin)$ system host-disk-list controller-1 # Add new disk to 'cgts-vg' local volume group ~(keystone_admin)$ system host-pv-add controller-1 cgts-vg <DISK_UUID> ~(keystone_admin)$ sleep 10 # wait for disk to be added # Confirm the available space and increased number of physical # volumes added to the cgts-vg colume group ~(keystone_admin)$ system host-lvg-list controller-1 # Increase docker filesystem to 60G ~(keystone_admin)$ system host-fs-modify controller-1 docker=60Configure the host settings for the vSwitch.
If using OVS-DPDK vswitch, run the following commands: Default recommendation for an AIO-controller is to use a single core for OVS-DPDK vSwitch. This should have been automatically configured, if not run the following command.
# assign 1 core on processor/numa-node 0 on controller-1 to vswitch ~(keystone_admin)$ system host-cpu-modify -f vswitch -p0 1 controller-1When using OVS-DPDK, configure 1G of huge pages for vSwitch memory on each NUMA node on the host. It is recommended to configure 1x 1G huge page (-1G 1) for vSwitch memory on each NUMA node on the host.
However, due to a limitation with Kubernetes, only a single huge page size is supported on any one host. If your application VMs require 2M huge pages, then configure 500x 2M huge pages (-2M 500) for vSwitch memory on each NUMA node on the host.
# assign 1x 1G huge page on processor/numa-node 0 on controller-1 to vswitch ~(keystone_admin)$ system host-memory-modify -f vswitch -1G 1 controller-1 0 # Assign 1x 1G huge page on processor/numa-node 1 on controller-0 to vswitch ~(keystone_admin)$ system host-memory-modify -f vswitch -1G 1 controller-1 1Important
VMs created in an OVS-DPDK environment must be configured to use huge pages to enable networking and must use a flavor with property:
hw:mem_page_size=large.Configure the huge pages for VMs in an OVS-DPDK environment on this host, assuming 1G huge page size is being used on this host, with the following commands:
# assign 10x 1G huge page on processor/numa-node 0 on controller-1 to applications ~(keystone_admin)$ system host-memory-modify -f application -1G 10 controller-1 0 # assign 10x 1G huge page on processor/numa-node 1 on controller-1 to applications ~(keystone_admin)$ system host-memory-modify -f application -1G 10 controller-1 1No additional configuration is required for the OVS vswitch in virtual environment.
Add an instances filesystem OR set up a disk based nova-local volume group, which is needed for stx-openstack nova ephemeral disks.
Note
Both cannot exist at the same time.
Add an ‘instances’ filesystem:
~(keystone_admin)$ export NODE=controller-1 # Create ‘instances’ filesystem ~(keystone_admin)$ system host-fs-add ${NODE} instances=<size>Or
Add a ‘nova-local’ volume group:
~(keystone_admin)$ export NODE=controller-1 # Create ‘nova-local’ local volume group ~(keystone_admin)$ system host-lvg-add ${NODE} nova-local # Get UUID of an unused DISK to to be added to the ‘nova-local’ volume # group. CEPH OSD Disks can NOT be used # List host’s disks and take note of UUID of disk to be used ~(keystone_admin)$ system host-disk-list ${NODE} # Add the unused disk to the ‘nova-local’ volume group ~(keystone_admin)$ system host-pv-add ${NODE} nova-local <DISK_UUID>Set up a ‘instances’ filesystem, which is needed for stx-openstack nova ephemeral disks.
~(keystone_admin)$ export NODE=controller-1 ~(keystone_admin)$ system host-fs-add ${NODE} instances=34Configure data interfaces for controller-1. Data class interfaces are vswitch interfaces used by vswitch to provide VM virtio vNIC connectivity to OpenStack Neutron Tenant Networks on the underlying assigned Data Network.
Important
A compute-labeled All-in-one controller host MUST have at least one Data class interface.
Configure the data interfaces for controller-1.
export NODE=controller-1 # List inventoried host's ports and identify ports to be used as 'data' interfaces, # based on displayed linux port name, pci address and device type. system host-port-list ${NODE} # List host’s auto-configured ‘ethernet’ interfaces, # find the interfaces corresponding to the ports identified in previous step, and # take note of their UUID system host-if-list -a ${NODE} # Modify configuration for these interfaces # Configuring them as 'data' class interfaces, MTU of 1500 and named data# ~(keystone_admin)$ system host-if-modify -m 1500 -n data0 -c data ${NODE} <data0-if-uuid> ~(keystone_admin)$ system host-if-modify -m 1500 -n data1 -c data ${NODE} <data1-if-uuid> # Create Data Networks that vswitch 'data' interfaces will be connected to ~(keystone_admin)$ DATANET0='datanet0' ~(keystone_admin)$ DATANET1='datanet1' # Assign Data Networks to Data Interfaces ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <data0-if-uuid> ${DATANET0} ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} <data1-if-uuid> ${DATANET1}~(keystone_admin)$ DATA0IF=eth1000 ~(keystone_admin)$ DATA1IF=eth1001 ~(keystone_admin)$ export NODE=controller-1 ~(keystone_admin)$ PHYSNET0='physnet0' ~(keystone_admin)$ PHYSNET1='physnet1' ~(keystone_admin)$ SPL=/tmp/tmp-system-port-list ~(keystone_admin)$ SPIL=/tmp/tmp-system-host-if-list ~(keystone_admin)$ system host-port-list ${NODE} --nowrap > ${SPL} ~(keystone_admin)$ system host-if-list -a ${NODE} --nowrap > ${SPIL} ~(keystone_admin)$ DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}') ~(keystone_admin)$ DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}') ~(keystone_admin)$ DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}') ~(keystone_admin)$ DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}') ~(keystone_admin)$ DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}') ~(keystone_admin)$ DATA1PORTNAME=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $4}') ~(keystone_admin)$ DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}') ~(keystone_admin)$ DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}') ~(keystone_admin)$ system datanetwork-add ${PHYSNET0} vlan ~(keystone_admin)$ system datanetwork-add ${PHYSNET1} vlan ~(keystone_admin)$ system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID} ~(keystone_admin)$ system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID} ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0} ~(keystone_admin)$ system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1}