Use Kubernetes CPU Manager Static Policy with application-isolated cores¶
StarlingX supports running the most critical low-latency applications on host CPUs which are completely isolated from the host process scheduler and exclusive (or dedicated/pinned) to the pod.
Prerequisites
You will need to enable the Kubernetes CPU Manager’s Static Policy for the target worker node(s). See StarlingX Administrator Tasks: Kubernetes CPU Manager Policies for details on how to enable this CPU management mechanism.
You will need to configure application-isolated cores for the target worker node(s). See StarlingX Administrator Tasks: Isolate the CPU Cores to Enhance Application Performance for details on how to configure application-isolated cores.
Procedure
Create your pod with <resources:requests:cpu/memory> and <resources:limits:cpu/memory> according to https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#cpu-management-policies in order to select Best Effort or Burstable class; use of application-isolated cores are NOT supported with Guaranteed QoS class. Specifically this requires either:
for Best Effort, do NOT specify <resources:limits:cpu/memory>
or
for Burstable, <resources:limits:cpu/memory> and <resources:limits:cpu/memory> are specified but NOT equal to each other.
Then, to add ‘application-isolated’ CPUs to the pod, configure ‘<resources:requests:windriver.com/isolcpus>’ and ‘<resources:limits:windriver.com/isolcpus>’ equal to each other, in the pod spec. These cores will be exclusive (dedicated/pinned) to the pod. If there are multiple processes within the pod/container, they can be individually affined to separate application-isolated CPUs if the pod/container requests multiple windriver.com/isolcpus resources. This is highly recommended as the Linux kernel does not load balance across application-isolated cores. Start-up code in the container can determine the available CPUs by running sched_getaffinity(), by looking for files of the form /dev/cpu/<X> where <X> is a number, or by parsing /sys/fs/cgroup/cpuset/cpuset.cpus within the container.
For example:
% cat <<EOF > stress-cpu-pinned.yaml apiVersion: v1 kind: Pod metadata: name: stress-ng-cpu spec: containers: - name: stress-ng-app image: alexeiled/stress-ng imagePullPolicy: IfNotPresent command: ["/stress-ng"] args: ["--cpu", "10", "--metrics-brief", "-v"] resources: requests: cpu: 2 memory: "1Gi" windriver.com/isolcpus: 2 limits: cpu: 2 memory: "2Gi" windriver.com/isolcpus: 2 nodeSelector: kubernetes.io/hostname: worker-1 EOF
Note
The nodeSelector is optional and it can be left out entirely. In which case, it will run in any valid node.
You will likely need to adjust some values shown above to reflect your deployment configuration. For example, on an AIO-SX or AIO-DX system. worker-1 would probably become controller-0 or controller-1.
The significant addition to this definition in support of application-isolated CPUs, is the resources section, which sets the windriver.com/isolcpus resource request and limit of 2.
Limitation: If Hyperthreading is enabled in the BIOS and application-isolated CPUs are configured, and these CPUs are allocated to more than one container, the SMT siblings may be allocated to different containers and that could adversely impact the performance of the application.
Workaround: The suggested workaround is to allocate all application-isolated CPUs to a single pod.
Apply the definition.
% kubectl apply -f stress-cpu-pinned.yaml
You can SSH to the worker node and run top, and type ‘1’ to see CPU details per core:
Describe the pod or node to see the CPU Request, CPU Limits and that it is in the “Burstable” QoS Class.
For example:
% kubectl describe <node> Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits windriver/isolcpus Requests windriver/isolcpus Limits AGE --------- ---- ------------ ---------- --------------- ------------- --------------------------- ------------------------- --- default stress-ng-cpu 1 (7%) 2 (15%) 1Gi (3%) 2Gi (7%) 2 (15%) 2 (15%) 9m31s % kubectl describe <pod> stress-ng-cpu ... QoS Class: Burstable
Delete the container.
% kubectl delete -f stress-cpu-pinned.yaml