SR-IOV plugin¶
Create Network Attachment Definitions¶
Network attachment definition specifications must be created in order to reference / request an SR-IOV interface in a container specification.
About this task
The sample network attachments shown in this procedure can be used in a container as shown in Use Network Attachment Definitions in a Container.
Prerequisites
You must have configured at least one SR-IOV interface on a host with the
target datanetwork (datanet-a
or datanet-b
in the example below)
assigned to it before creating a NetworkAttachmentDefinition
referencing
this data network.
Note
The configuration for this SR-IOV interface with either a netdevice
or
a user space network device such as a DPDK poll mode drive.
Note
Mellanox-based interfaces should be bound to the netdevice
vf-driver for
both kernel and user space usage.
After SR-IOV interfaces have been provisioned and the hosts labeled and unlocked, available SR-IOV VF resources are automatically advertised.
They can be referenced in subsequent StarlingX operations using the appropriate
NetworkAttachmentDefinition
name and the following extended resource name:
intel.com/pci_sriov_net_${DATANETWORK_NAME}
For example, with a network called datanet-a
the extended resource name
would be:
intel.com/pci_sriov_net_datanet_a
The extended resource name will convert all dashes (‘-’) in the data network name into underscores (’_’).
SR-IOV enabled interfaces using the netdevice VF driver must be administratively and operationally up to be advertised by the SR-IOV device plugin.
It is not possible to have multiple data networks assigned to the same interface since the VFs resources cannot be shared between pools.
Procedure
Create a simple SR-IOV network attachment definition file called
net1.yaml
associated with the data networkdatanet-a
.~(keystone_admin)]$ cat <<EOF > net1.yaml apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: net1 annotations: k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_datanet_a spec: config: '{ "cniVersion": "0.3.0", "type": "sriov" }' EOF
This
NetworkAttachmentDefinition
is valid for both a kernel-based and a DPDK (vfio) based device.Create an SR-IOV network attachment with a VLAN ID or with IP Address information.
The following example creates an SR-IOV network attachment definition configured for a VLAN with an ID of 2000.
~(keystone_admin)]$ cat <<EOF > net2.yaml apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: net2 annotations: k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_datanet_b spec: config: '{ "cniVersion": "0.3.0", "type": "sriov", "vlan": 2000 }' EOF
The following example creates an SR-IOV network attachment definition configured with IP Address information.
~(keystone_admin)]$ cat <<EOF > net3.yaml apiVersion: crd.projectcalico.org/v1 kind: IPPool metadata: name: mypool spec: cidr: "10.56.219.0/24" ipipMode: "Never" natOutgoing: True --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: net3 annotations: k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_datanet_b spec: config: '{ "cniVersion": "0.3.0", "type": "sriov", "ipam": { "type": "calico-ipam", "assign_ipv4": "true", "ipv4_pools": ["mypool"] }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig" }, "datastore_type": "kubernetes" }' EOF
Use Network Attachment Definitions in a Container¶
Network attachment definitions can be referenced by name when creating a container. The extended resource name of the SR-IOV pool should also be referenced in the resource requests.
About this task
The following examples use network attachment definitions net2
and net3
created in Creating Network Attachment Definitions.
As shown in these examples, it is important that you request the same number of devices as you have network annotations.
Procedure
Create a container with one additional SR-IOV interface using the
net2
network attachment definition.Populate the configuration file
pod1.yaml
with the following contents.apiVersion: v1 kind: Pod metadata: name: pod1 annotations: k8s.v1.cni.cncf.io/networks: '[ { "name": "net2", "interface": "sriov0" } ]' spec: containers: - name: pod1 image: centos/tools imagePullPolicy: IfNotPresent command: [ "/bin/bash", "-c", "--" ] args: [ "while true; do sleep 300000; done;" ] resources: requests: intel.com/pci_sriov_net_datanet_b: '1' limits: intel.com/pci_sriov_net_datanet_b: '1'
Apply the configuration to create the container.
~(keystone_admin)]$ kubectl create -f pod1.yaml pod/pod1 created
After creating the container, an extra network device interface,
net2
, which uses one SR-IOV VF, will appear in the associated container(s).After creating the container, the interface
sriov0
, which uses one SR-IOV VF will appear.At this point you can execute commands and review links on the container. For example:
$ kubectl exec -n default -it pod1 -- ip link show
Create a container with two additional SR-IOV interfaces using the
net2
network attachment definition.Populate the configuration file
pod2.yaml
with the following contents.apiVersion: v1 kind: Pod metadata: name: pod2 annotations: k8s.v1.cni.cncf.io/networks: '[ { "name": "net2", "interface": "sriov0" }, { "name": "net2", "interface": "sriov1" } ]' spec: containers: - name: pod2 image: centos/tools imagePullPolicy: IfNotPresent command: [ "/bin/bash", "-c", "--" ] args: [ "while true; do sleep 300000; done;" ] resources: requests: intel.com/pci_sriov_net_datanet_b: '2' limits: intel.com/pci_sriov_net_datanet_b: '2'
Apply the configuration to create the container.
~(keystone_admin)$ kubectl create -f pod2.yaml pod/pod2 created
After creating the container, two
net2
network device interfaces, which each use one SR-IOV VF, will appear in the associated container(s).After creating the container, the network device interfaces
sriov0
andsriov1
, which uses one SR-IOV VF, will appear in the associated container(s).At this point you can execute commands and review links on the container. For example:
$ kubectl exec -n default -it pod2 -- ip link show
Create a container with two additional SR-IOV interfaces using the
net2
andnet3
network attachment definitions.Populate the configuration file
pod3.yaml
with the following contents.apiVersion: v1 kind: Pod metadata: name: pod3 annotations: k8s.v1.cni.cncf.io/networks: '[ { "name": "net2", "interface": "sriov0" }, { "name": "net3", "interface": "sriov1" } ]' spec: containers: - name: pod3 image: centos/tools imagePullPolicy: IfNotPresent command: [ "/bin/bash", "-c", "--" ] args: [ "while true; do sleep 300000; done;" ] resources: requests: intel.com/pci_sriov_net_datanet_b: '2' limits: intel.com/pci_sriov_net_datanet_b: '2'
Apply the configuration to create the container.
~(keystone_admin)$ kubectl create -f pod3.yaml
net2
andnet3
will each take a device from thepci_sriov_net_datanet_b
pool and be configured on the container/host based on the their respectiveNetworkAttachmentDefinition
specifications (see Creating Network Attachment Definitions).After creating the pod, the network device interface
sriov0
, which uses one SR-IOV VF, will appear in the associated container(s).After creating the container, network device interfaces
net2
andnet3
, which each use one SR-IOV VF, will appear in the associated container(s).At this point you can execute commands and review links on the container. For example:
$ kubectl exec -n default -it pod3 -- ip link show
Note
In the above examples, the PCI addresses allocated to the container are exported via an environment variable. For example:
~(keystone_admin)$ kubectl exec -n default -it pod3 -- printenv PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=pod3 TERM=xterm PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_DATANET_B=0000:81:0e.4,0000:81:0e.0 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 KUBERNETES_SERVICE_HOST=10.96.0.1 KUBERNETES_SERVICE_PORT=443 KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT=tcp://10.96.0.1:443 KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443 container=docker HOME=/root
Create a container with two additional SR-IOV interfaces using the
net1
network attachment definition for a DPDK based application.The configuration of the SR-IOV host interface to which the datanetwork is assigned determines whether the network attachment in the container will be kernel or DPDK-based. The SR-IOV host interface needs to be created with a vfio
vf-driver
for the network attachment in the container to be DPDK-based, otherwise it will be kernel-based.Note
Mellanox based NICs should be bound to a netdevice (default)
vf-driver
.Populate the configuration file
pod4.yaml
with the following contents.apiVersion: v1 kind: Pod metadata: name: testpmd annotations: k8s.v1.cni.cncf.io/networks: '[ { "name": "net1" }, { "name": "net1" } ]' spec: restartPolicy: Never containers: - name: testpmd image: "amirzei/mlnx_docker_dpdk:ubuntu16.04" volumeMounts: - mountPath: /mnt/huge-1048576kB name: hugepage stdin: true tty: true securityContext: privileged: false capabilities: add: ["IPC_LOCK", "NET_ADMIN", "NET_RAW"] resources: requests: memory: 10Gi intel.com/pci_sriov_net_datanet_a: '2' limits: hugepages-1Gi: 4Gi memory: 10Gi intel.com/pci_sriov_net_datanet_a: '2' volumes: - name: hugepage emptyDir: medium: HugePages
Note
You must convert any dashes (-) in the datanetwork name used in the
NetworkAttachmentDefinition
to underscores (_).Apply the configuration to create the container.
~(keystone_admin)$ kubectl create -f pod4.yaml pod/testpmd created