Kernel Module Management Application¶

The app-kernel-module-management application enables dynamic loading of out-of-tree kernel modules at runtime, with modules distributed independently of the platform. Kernel modules can be optionally built on demand into container images or provided as pre-built container images. These images are used to deploy and load the required modules onto target Kubernetes nodes. Custom Kubernetes resources are used to define which kernel modules to load, from which container images, and the specific nodes on which the modules should be deployed.

Install App-kernel-module-management Application¶

Perform the following steps to install the app-kernel-module-management application.

Procedure

Upload the application package by running the following command:

~(keystone_admin)]$ system application-upload /usr/local/share/applications/helm/kernel-module-management-*.tgz

Create and apply the Helm overrides.

The kernel-module-management application requires a Docker registry to manage the module images it builds. The application will use this registry to pull and push the images.

The registry may be an external or a local registry, as shown in the following example:

Set the Docker credentials.

USERNAME="sysinv"
PASSWORD=$(keyring get sysinv services)
DOCKER_CREDENTIALS=$(echo -n "${USERNAME}":"${PASSWORD}" | base64)

Create docker-config.json.

cat >docker-config.json<<EOF
{
  "auths": {
      "https://registry.local:9001": {
         "auth": "$DOCKER_CREDENTIALS"
      }
  }
}
EOF

dconfigjson=$(cat docker-config.json | base64 -w 0)

Create the override file with registry information.

cat >kmm-app-override.yaml<<EOF
dockerRegistrySecretName: "kmm-registry-secret"
dockerConfigJson: "$dconfigjson"
EOF

Apply the overrides.

~(keystone_admin)]$ system helm-override-update kernel-module-management kernel-module-management kernel-module-management --values kmm-app-override.yaml

After uploading the application and applying the overrides, apply the application using the following command.

~(keystone_admin)]$ system application-apply kernel-module-management

After the application is successfully applied, the two pods will be running as shown below:

(keystone_admin)]$ kubectl get pods -n kernel-module-management
NAME                                       READY   STATUS    RESTARTS   AGE
kmm-operator-controller-86dfc6bff8-swvf7   1/1     Running   0          97s
kmm-operator-webhook-788f76bd7c-64t6t      1/1     Running   0          97s

Building and Loading Kernel Modules¶

Create Kernel Module Images¶

To create your own out-of-tree kernel module it is necessary to have an image with the kernel headers to build your own module files against it. For more details, see kernel-module-management docs.

FROM ubuntu as builder
ARG KERNEL_FULL_VERSION
RUN apt-get update && apt-get install -y bc \
bison \
flex \
libelf-dev \
gnupg \
wget \
git \
make \
gcc \
linux-headers-${KERNEL_FULL_VERSION}

The StarlingX project provides a pre-built container image with the tooling and source code to enable you to build your kernel modules inside the StarlingX ecosystem. This pre-built image also includes a hello_world module example that can be used to test the kernel-module-management app. The structure used in the example module can be followed as a basis for creating your own kernel modules.

To create your own module image it is necessary to create a Dockerfile building the wanted modules, as well as the source code for the kernel module itself. The final image only requires the .ko files from the built module, therefore a multi-stage build strategy is advised to create a lighter image, without unnecessary packages and files.

This Dockerfile example uses two stages. The first stage builds the module(s) and the second one builds an image containing the module files and required tools.

FROM docker.io/starlingx/kmm-builder:stx.12.0-v1.0.0 as builder
ARG KERNEL_FULL_VERSION
WORKDIR /usr/src
RUN ["git", "clone", "https://opendev.org/starlingx/app-kernel-module-management.git"]
WORKDIR /usr/src/app-kernel-module-management/tools/builder-image/debian/docker/hello-world-module
RUN make KERNEL_DIR=/lib/modules/${KERNEL_FULL_VERSION}/build && mkdir /out && cp *.ko /out/

FROM ubuntu
ARG KERNEL_FULL_VERSION
RUN apt-get update && apt-get install -y kmod && rm -rf /var/lib/apt/lists/*
COPY --from=builder /out/*.ko /opt/lib/modules/${KERNEL_FULL_VERSION}/
RUN depmod -b /opt ${KERNEL_FULL_VERSION}

With this Dockerfile created, it is possible to build the image passing the current StarlingX kernel version “6.12.0-1-amd64” as a build argument when running docker build.

Warning

The KERNEL_FULL_VERSION variable should always match the current StarlingX kernel version, otherwise the module will not be loaded by the application.

Warning

With new versions of StarlingX, the kernel version could change, creating the necessity to rebuild the module image matching the new kernel and kmm-builder image versions, before reapplying it. Otherwise, the module will experience an outage while the system builds the new version.

Load Module images¶

One way to load the module in the application is pre-building the module image in a lab and then shipping it out to all the sites to avoid the overhead of building separately for each site. For these cases, just push the built image to the registry in use and create a CRD Module.

Note

In addition to using a public container registry, in a Distributed Cloud environment the module image can also be pushed to the Central Cloud registry and pulled from it by the subclouds.

cat << 'EOF' > hello_world_mod.yaml
apiVersion: kmm.sigs.x-k8s.io/v1beta1
kind: Module
metadata:
  name: kmm-hello-world
  namespace: kernel-module-management
spec:
  moduleLoader:
    container:
      modprobe:
        moduleName: hello_world_dmesg
      kernelMappings:
        - literal: "6.12.0-1-amd64"
          containerImage: registry.local:9001/kmm/kmm-hello-world:stx.12.0-6.12.0-1-amd64
          registryTLS:
            insecure: false
            insecureSkipTLSVerify: true

  imageRepoSecret:
    name: "kmm-registry-secret"

  selector:
    kubernetes.io/os: linux
    kubernetes.io/hostname: controller-0

  tolerations:
    - key: "services"
      operator: "Equal"
      value: "disabled"
      effect: "NoExecute"
EOF

Note

As kernel versions change, the kernelMappings also need to be updated to match the version(s) in use.

Note

The example above selects by hostname, another common pattern is selecting by node type using the selector node-role.kubernetes.io/worker: “”. As seen on KMM docs.

To load the module, apply the CRD created by running the following command:

~(keystone_admin)]$ kubectl apply -f hello_world_mod.yaml

Warning

See kernel-module-management official documentation for more details on how to construct a Module custom resource. It is not possible to select documentation for a specific KMM version, so the docs do not take into consideration which version of the application is running. Some configurations may not behave as expected.

Building and loading the modules¶

You can create a CRD Module file and a ConfigMap as an additional step when you want the kernel-module-management application to build the module image. The ConfigMap will carry the Dockerfile content mentioned previously, and the CRD Module will reference the newly created ConfigMap.

cat << 'EOF' > hello_world_cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kmm-hello-world-cm
  namespace: kernel-module-management
data:
  dockerfile: |
    FROM docker.io/starlingx/kmm-builder:stx.12.0-v1.0.0 as builder
    ARG KERNEL_FULL_VERSION
    WORKDIR /usr/src
    RUN ["git", "clone", "https://opendev.org/starlingx/app-kernel-module-management.git"]
    WORKDIR /usr/src/app-kernel-module-management/tools/builder-image/debian/docker/hello-world-module
    RUN make KERNEL_DIR=/lib/modules/${KERNEL_FULL_VERSION}/build && mkdir /out && cp *.ko /out/

    FROM ubuntu
    ARG KERNEL_FULL_VERSION
    RUN apt-get update && apt-get install -y kmod
    COPY --from=builder /out/*.ko /opt/lib/modules/${KERNEL_FULL_VERSION}/
    RUN depmod -b /opt ${KERNEL_FULL_VERSION}
EOF

cat << 'EOF' > hello_world_mod.yaml
apiVersion: kmm.sigs.x-k8s.io/v1beta1
kind: Module
metadata:
  name: kmm-hello-world
  namespace: kernel-module-management
spec:
  moduleLoader:
    container:
      modprobe:
        moduleName: hello_world_dmesg
      kernelMappings:
        - literal: "6.12.0-1-amd64"
          containerImage: registry.local:9001/kmm/kmm-hello-world:stx.12.0-6.12.0-1-amd64
          build:
            buildArgs:
              - name: KERNEL_FULL_VERSION
                value: "6.12.0-1-amd64"
            baseImageRegistryTLS:
              insecure: false
              insecureSkipTLSVerify: true
            dockerfileConfigMap:
              name: kmm-hello-world-cm
          registryTLS:
            insecure: false
            insecureSkipTLSVerify: true

  imageRepoSecret:
    name: "kmm-registry-secret"

  selector:
    kubernetes.io/os: linux
    kubernetes.io/hostname: controller-0

  tolerations:
    - key: "services"
      operator: "Equal"
      value: "disabled"
      effect: "NoExecute"
EOF

To load the module, apply the CRD and ConfigMap created by running the following command:

~(keystone_admin)]$ kubectl apply -f hello_world_cm.yaml -f hello_world_mod.yaml

The kernel module build progress can be tracked using the kubectl logs command.

~(keystone_admin)]$ kubectl logs -n kernel-module-management kmm-hello-world-build-<POD_ID> -f

Note

Successful build pods are garbage-collected immediately, while failed build pods are always preserved and must be deleted manually by the administrator for the build to be restarted.

Regardless of whether the module was loaded with a pre-built image or built using the app, it is possible to confirm that the module is correctly loaded by running the following dmesg:

sudo dmesg

[34352.320647] systemd-sysv-generator[2076803]: Overwriting existing symlink /run/systemd/generator.late/radosgw.service with real service.
[102022.711368] e1000: enp0s3 NIC Link is Down
[102028.836500] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[102422.922297] e1000: enp0s3 NIC Link is Down
[102422.922706] e1000 0000:00:03.0 enp0s3: Reset adapter
[102425.122913] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[102582.800513] e1000: enp0s3 NIC Link is Down
[102586.903818] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[103642.175972] Hello, world!

Unload Kernel Module¶

To unload the kernel module, run the following command in the same path where the previous module files were created:

~(keystone_admin)]$ kubectl delete -f hello_world_cm.yaml -f hello_world_mod.yaml

For the scenario with a pre-built image, run the following command:

~(keystone_admin)]$ kubectl delete -f hello_world_mod.yaml

Confirm that the kernel module was successfully unloaded by running dmesg again and checking for the Goodbye message logged.

sudo dmesg

[102425.122913] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[102582.800513] e1000: enp0s3 NIC Link is Down
[102586.903818] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[103642.175972] Hello, world!
[104469.925530] Goodbye, world!

Uninstall App-kernel-module-management Application¶

To uninstall the app-kernel-module-management application, run the following command:

~(keystone_admin)]$ system application-remove kernel-module-management

Delete the uploaded application by running the following command:

~(keystone_admin)]$ system application-delete kernel-module-management