Configure PCI Passthrough Ethernet InterfacesΒΆ

A passthrough Ethernet interface is a physical PCI Ethernet NIC on a compute node to which a virtual machine is granted direct access. This minimizes packet processing delays but at the same time demands special operational considerations.

About this task

Configure a PCI Passthrough Ethernet Interface on a host and request it for an instance at boot/create time.

Prerequisites

  • To use PCI passthrough or SR-IOV devices, you must have Intel VT-x and Intel VT-d features enabled in the BIOS.

  • The exercise assumes that the underlying data network group0-data0 exists already, and that VLAN ID 10 is a valid segmentation ID assigned to project1.

Procedure

  1. Log in as the admin user to the StarlingX Platform Horizon Web Interface.

  2. Lock the compute node you want to configure.

  3. Configure the Ethernet interface to be used as a PCI passthrough interface. You can do this using Horizon or the CLI.

    • Using Horizon:

      1. Select Admin > Platform > Host Inventory from the left-hand pane.

      2. Select the Hosts tab.

      3. Click the name of the compute host.

      4. Select the Interfaces tab.

      5. Click the Edit Interface button associated with the interface you want to configure.

        The Edit Interface dialog appears.

        ../../_images/ptj1538163621289.png

      6. Select pci-passthrough, from the Interface Class drop-down, and then select the data network to attach the interface.

      7. (Optional) You may also need to change the MTU.

    • Using the CLI:

      Assign the pci-sriov class to the interface.

      ~(keystone_admin)$ system host-if-modify -c pci-passthrough compute-0 enp0s3
~(keystone_admin)$ system interface-datanetwork-assign compute-0 <enp0s3_interface_uuid> <group0_data0_data_network_uuid>

  4. Check if the Ethernet interface supports SR-IOV.

    1. Check the host port associated with the configured PCI-passthrough interface.

      ~(keystone_admin)$ system host-if-list <host-name> | grep pci-passthrough

    2. Review the value of sriov_totalvfs on the target port.

      If the value is None, the Ethernet interface does not support SR-IOV. Otherwise, it does.

      ~(keystone_admin)$ system host-port-show <host-name> <port-name> | grep sriov_totalvfs

    Note

    For Ethernet interfaces without SR-IOV support, there is a known limitation reported here. This limitation is overcome with a specific step later on this procedure.

  5. Create the net0 project network for Ethernet interfaces that support SR-IOV.

    Warning

    If the Ethernet interface does not support SR-IOV, skip this step.

    Log in as the admin user to the OpenStack Horizon Web Interface.

    Select Admin > Network > Networks, select the Networks tab, and then click Create Network. Fill in the Create Network dialog box as illustrated below. You must ensure that:

    • project1 has access to the project network. Either by assigning it as the owner, as in the illustration (using Project), or by enabling the shared flag.

    • The segmentation ID is set to 10.

    ../../_images/bek1516655307871.png

    Click the Next button to proceed to the Subnet tab.

    Click the Next button to proceed to the Subnet Details tab.

  6. Configure the access switch. Refer to the OEM documentation to configure the access switch.

    Log in as the admin user to the StarlingX Platform Horizon Web Interface.

    Configure the physical port on the access switch used to connect to Ethernet interface enp0s3 to be an access port with the default VLAN ID of 10. Traffic across the connection is therefore untagged, and effectively integrated into the targeted project network.

    You can also use a trunk port on the access switch so that it handles tagged packets as well. However, this opens the possibility for guest applications to join other project networks using tagged packets with different VLAN IDs, which might compromise the security of the system. See OpenStack Introduction: L2 Access Switches for other details regarding the configuration of the access switch.

  7. Unlock the compute node.

  8. Create a neutron port with a VNIC of type direct-physical for Ethernet interfaces that support SR-IOV.

    Warning

    If the Ethernet interface does not support SR-IOV, skip this step.

    First, you must set up the environment and determine the correct network UUID to use with the port.

    ~(keystone_admin)$ source /etc/platform/openrc
~(keystone_admin)$ OS_AUTH_URL=http://keystone.openstack.svc.cluster.local/v3
~(keystone_admin)$ openstack network list | grep net0
~(keystone_admin)$ openstack port create --network <uuid_of_net0> --vnic-type direct-physical <port_name>

    You have now created a port to be used when launching the server in the next step.

  9. Complete the following Nova configuration, for Ethernet interfaces that do not support SR-IOV.

    Warning

    If the Ethernet interface supports SR-IOV, skip this step.

    1. Get the Ethernet interface vendor_id and product_id:

        ~(keystone_admin)$ source /etc/platform/openrc
~(keystone_admin)$ system host-port-show <host-name> <port-name> | grep -E '(pvendor |pdevice )'

    2. Use the retrieved IDs to create a PCI alias with "device_type":"type-PCI", as peer Configure a PCI Alias in Nova.

    3. Configure a flavor with the extra spec key pci_passthrough:alias pointing to the previously created PCI alias, as peer Configure a Flavor to Use a Generic PCI Device

  10. Launch the virtual machine

    Note

    You will need to source to the same project selected in the Create Network net0 step.

    • For Ethernet interfaces with SR-IOV support: specify the port uuid created.

      ~(keystone_admin)$ openstack server create --flavor <flavor_name> --image <image_name> --nic port-id=<port_uuid> <name>

    • For Ethernet interfaces without SR-IOV support: specify the created flavor to use the PCI device.

      ~(keystone_admin)$ openstack server create --flavor <pci_flavor_name> --image <image_name>

    For more information, see the Neutron documentation at: https://docs.openstack.org/neutron/train/admin/config-sriov.html.

Results

The new virtual machine instance is up now. It has a PCI passthrough connection to the net0 project network identified with VLAN ID 10.

Prerequisites

Access switches must be properly configured to ensure that virtual machines using PCI-passthrough or SR-IOV Ethernet interfaces have the expected connectivity. In a common scenario, the virtual machine using these interfaces connects to external end points only, that is, it does not connect to other virtual machines in the same StarlingX OpenStack cluster. In this case:

  • Traffic between the virtual machine and the access switch can be tagged or untagged.

  • The connecting port on the access switch is part of a port-based VLAN.

  • The port-based VLAN provides the required connectivity to external switching and routing equipment needed by guest applications to establish connections to the intended end points.

For connectivity to other virtual machines in the StarlingX OpenStack cluster the following configuration is also required:

  • The VLAN ID used for the project network, 10 in this example, and the default port VLAN ID of the access port on the switch are the same. This ensures that incoming traffic from the virtual machine is tagged internally by the switch as belonging to VLAN ID 10, and switched to the appropriate exit ports.