Reintegrate Kata Containers into StarlingX¶
This spec reintegrates Kata Containers into StarlingX. When the migration to Debian was made, the Kata containers support wasn’t ported. Then, this spec aims to port it.
Problem description¶
Kata Containers is an open source community working to build a secure container runtime with lightweight virtual machines that feel and perform like containers, but provide stronger workload isolation using hardware virtualization technology as a second layer of defense.[0]
Since there is a performance gap between Kata Containers and conventional containers [1], Kata Containers is not used in default unless explicitly requirement is found in the Kubernetes’s pod configuration file. Or in other words, regardless Kata Containers is enabled or not, containers currently we have in StarlingX will not use kata container runtime in default, unless extra change is made.
Use Cases¶
With Kata Containers supported in StarlingX, System developers, testers, operators, administrators could choose what container runtime to use when run a container image, based on their needs. For example, if there is security concern for the pod/container, it could be selected to run with kata container. For the default case, runc will be selected as default low-level container runtime, which is the default low-level runtime for Kubernetes.
Proposed changes¶
The StarlingX has only one container orchestration tool: the Kubernetes. Which is currently already configured to use containerd as container runtime. As said in the previous sessions, runc will be maintained as the default low-level container runtime. Then, to run a pod with Kata, we need to explicitly declare our intention. There are two methods to do so:
Use annotation. Add “io.kubernetes.cri.untrusted-workload” annotation in pod file.[2]
Use Runtime Class.[3] 1st method is simpler, but it is deprecated. 2nd method is recommended. Runtime Class is supported in Kubernetes since v1.14, and current Kubernetes version in StarlingX master is v1.26. So use Runtime Class to select kata container runtime is recommended.
The Kata Containers version that will be integrated in StarlingX 9.0 is the 3.1.3. As were not found precompiled debian packages, then, in build-time, it will be packaged using statically-linked prebuilt binaries from https://github.com/kata-containers/kata-containers/releases.
And, for compatibility reasons, it will be necessary upgrade the virtualizer QEMU from 5.2 to 7.2 version.
Data model impact¶
None. This spec does not change any existing data models.
REST API impact¶
None. This spec does not change any existing REST APIs.
Security impact¶
There is no sensitive data touch for this feature, except containerd need touch certificate/key for local secure registry “registry.local”. Kata Containers could help enhance system’s security by runs in a dedicated kernel, providing isolation of network, I/O and memory and can utilize hardware-enforced isolation with virtualization VT extensions.” [0]
Other end user impact¶
None. This spec does not impact the end user at all.
Performance Impact¶
For containers that doesn’t use kata-runtime, the performance should be the same as previous. For containers that use kata-runtime, some performance drop is expected. Per Kata Containers website:”Delivers consistent performance as standard Linux containers; increased isolation with the performance tax of standard virtual machines.”[0] Also there are two studies regarding the comparative performance between runc and Kata Containers that analyzes and quantifies the aforementioned gap. [1][4]
Other deployer impact¶
None. This spec does not impact deployement at all.
Developer impact¶
None. This spec does not impact the development at all.
Upgrade impact¶
None. This spec does not impact the upgrade process at all.
Implementation¶
Assignee(s)¶
Primary assignee:
Guilherme Batista Leite
Other contributors:
David Liu Davi Frossard Fabio Studyny Higa Marcos Paulo Oliveira Silva Elson Claudio de Oliveira
Repos Impacted¶
stx-integ
stx-virt
stx-root
stx-metal
stx-tools
Work Items¶
Add kata container * Add kata container support
Add the support in StarlingX for Kata Containers. The prebuilt runtime binaries are downloaded and modified to provide customization support.
Upgrade QEMU from version 5.2 to version 7.2 * Upgrade QEMU and port patches
To avoid any kind of incompatibility with Kata Containers 3.1.x version, the QEMU upgrade to version 7.2 is necessary. Currently, the QEMU version on the host has WRO-specific patches which will need to be ported to the new version.
Investigation and Documentation of Kata Containers Limitations * Actualize the existent documentation
Currently, there is a document [5] that describes the known limitations when using Kata Containers, Although, it is out of date. The new version (3.1.3) has already had some of these limitations corrected.
Besides upper work items, there are other work items maybe need added during implementation of this spec. Please check the storyboard link in the beginning of this spec to get the full work item list.
Dependencies¶
This specification depends upon the open source upstream:
https://github.com/kata-containers/kata-containers https://github.com/qemu/qemu
Testing¶
System Configurations¶
The system configurations assumed for testing are:
AIO-SX
Standard
Preconditions¶
A Runtimeclass should be created for Kata-containers.
Test Scenarios¶
Basic Test: Kata Container pod checking its kernel version.
Kata Container support for static CPU policy.
Kata Container support for non-static CPU policy.
Kata Container support for SR-IOV.
Kata Container support for hugepage.
Kata Container support for IPV6.
Documentation Impact¶
The existent end-user Kata Containers documentation will need to be changed. Some outdated info and references will need to be updated.
References¶
[0]: https://katacontainers.io/ [1]: https://www.stackhpc.com/kata-io-1.html [2]: https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#create-an-untrusted-pod-using-kata-containers [3]: https://kubernetes.io/docs/concepts/containers/runtime-class/ [4]: https://gkoslovski.github.io/files/latincom-2021.pdf [5]: https://docs.starlingx.io/usertasks/kubernetes/known-limitations.html
History¶
Release Name |
Description |
|---|---|
stx-9.0 |
Introduced |
