Kernel Module Management Application

The app-kernel-module-management application enables dynamic loading of out-of-tree kernel modules at runtime, with modules distributed independently of the platform. Kernel modules can be optionally built on demand into container images or provided as pre-built container images. These images are used to deploy and load the required modules onto target Kubernetes nodes. Custom Kubernetes resources are used to define which kernel modules to load, from which container images, and the specific nodes on which the modules should be deployed.

Install App-kernel-module-management Application

Perform the following steps to install the app-kernel-module-management application.

Procedure

1. Upload the application package by running the following command:

   ~(keystone_admin)]$ system application-upload /usr/local/share/applications/helm/kernel-module-management-*.tgz
   

2. Create and apply the Helm overrides.

   The kernel-module-management application requires a Docker registry to manage the module images it builds. The application will use this registry to pull and push the images.

   The registry may be an external or a local registry, as shown in the following example:

   1. Set the Docker credentials.

      USERNAME="sysinv"
      PASSWORD=$(keyring get sysinv services)
      DOCKER_CREDENTIALS=$(echo -n "${USERNAME}":"${PASSWORD}" | base64)
      

   2. Create docker-config.json.

      cat >docker-config.json<<EOF
      {
        "auths": {
            "https://registry.local:9001": {
               "auth": "$DOCKER_CREDENTIALS"
            }
        }
      }
      EOF
      
      dconfigjson=$(cat docker-config.json | base64 -w 0)
      

   3. Create the override file with registry information.

      cat >kmm-app-override.yaml<<EOF
      dockerRegistrySecretName: "kmm-registry-secret"
      dockerConfigJson: "$dconfigjson"
      EOF
      

   4. Apply the overrides.

      ~(keystone_admin)]$ system helm-override-update kernel-module-management kernel-module-management kernel-module-management --values kmm-app-override.yaml
      

3. After uploading the application and applying the overrides, apply the application using the following command.

   ~(keystone_admin)]$ system application-apply kernel-module-management
   

   After the application is successfully applied, the two pods will be running as shown below:

   ~(keystone_admin)]$ kubectl get pods -n kernel-module-management
   NAME                                       READY   STATUS    RESTARTS   AGE
   kmm-operator-controller-86dfc6bff8-swvf7   1/1     Running   0          97s
   kmm-operator-webhook-788f76bd7c-64t6t      1/1     Running   0          97s
   

Build, Load and Sign Kernel Modules

Create Kernel Module Images

To create your own out-of-tree kernel module it is necessary to have an image with the kernel headers to build your own module files against it. For more details, see kernel-module-management docs.

FROM ubuntu as builder
ARG KERNEL_FULL_VERSION
RUN apt-get update && apt-get install -y bc \
bison \
flex \
libelf-dev \
gnupg \
wget \
git \
make \
gcc \
linux-headers-${KERNEL_FULL_VERSION}

The StarlingX project provides a pre-built container image with the tooling and source code to enable you to build your kernel modules inside the StarlingX ecosystem. This pre-built image also includes a hello_world module example that can be used to test the kernel-module-management app. The structure used in the example module can be followed as a basis for creating your own kernel modules.

To create your own module image it is necessary to create a Dockerfile building the wanted modules, as well as the source code for the kernel module itself. The final image only requires the .ko files from the built module, therefore a multi-stage build strategy is advised to create a lighter image, without unnecessary packages and files.

This Dockerfile example uses two stages. The first stage builds the module(s) and the second one builds an image containing the module files and required tools.

FROM docker.io/starlingx/kmm-builder:master-debian-trixie-amd64-stable-latest as builder
ARG KERNEL_FULL_VERSION
WORKDIR /usr/src
RUN ["git", "clone", "https://opendev.org/starlingx/app-kernel-module-management.git"]
WORKDIR /usr/src/app-kernel-module-management/tools/builder-image/debian/docker/hello-world-module
RUN make KERNEL_DIR=/lib/modules/${KERNEL_FULL_VERSION}/build && mkdir /out && cp *.ko /out/

FROM ubuntu
ARG KERNEL_FULL_VERSION
RUN apt-get update && apt-get install -y kmod && rm -rf /var/lib/apt/lists/*
COPY --from=builder /out/*.ko /opt/lib/modules/${KERNEL_FULL_VERSION}/
RUN depmod -b /opt ${KERNEL_FULL_VERSION}

With this Dockerfile created, it is possible to build the image passing the current StarlingX kernel version “6.18.15+deb13-amd64” as a build argument when running docker build.

Note

The kernel version loaded on a StarlingX node depends on the platform release and whether the node was installed with the standard kernel (for example, 6.18.15+deb13-amd64) or the real-time kernel (for example, 6.18.15+deb13-rt-amd64). Before building the image or writing the Module kernelMappings, verify the kernel currently running on each target node:

$ uname -r

Use the value returned as KERNEL_FULL_VERSION in the Dockerfile build arguments and as the literal value in the kernelMappings entry. The examples in the sections that follow are shown separately for the standard and real-time kernels; use the variant that matches the kernel running on the target node. If nodes in the cluster run different kernel types, combine the kernelMappings entries from both variants into a single Module resource, keeping distinct image tags per kernel.

Warning

The KERNEL_FULL_VERSION variable should always match the current StarlingX kernel version, otherwise the module will not be loaded by the application.

Warning

With new versions of StarlingX, the kernel version could change, creating the necessity to rebuild the module image matching the new kernel and kmm-builder image versions, before reapplying it. Otherwise, the module will experience an outage while the system builds the new version.

Load Module images

One way to load the module in the application is pre-building the module image in a lab and then shipping it out to all the sites to avoid the overhead of building separately for each site. For these cases, just push the built image to the registry in use and create a CRD Module.

Note

In addition to using a public container registry, in a Distributed Cloud environment the module image can also be pushed to the Central Cloud registry and pulled from it by the subclouds.

Use the Module example that matches the kernel loaded on the target node.

Example for the standard kernel (6.18.15+deb13-amd64):

cat << 'EOF' > hello_world_mod.yaml
apiVersion: kmm.sigs.x-k8s.io/v1beta1
kind: Module
metadata:
  name: kmm-hello-world
  namespace: kernel-module-management
spec:
  moduleLoader:
    container:
      modprobe:
        moduleName: hello_world_dmesg
      kernelMappings:
        - literal: "6.18.15+deb13-amd64"
          containerImage: registry.local:9001/kmm/kmm-hello-world:stx.13.0-6.18.15-1-amd64
          registryTLS:
            insecure: false
            insecureSkipTLSVerify: true

  imageRepoSecret:
    name: "kmm-registry-secret"

  selector:
    kubernetes.io/os: linux
    kubernetes.io/hostname: controller-0

  tolerations:
    - key: "services"
      operator: "Equal"
      value: "disabled"
      effect: "NoExecute"
EOF

Example for the real-time kernel (6.18.15+deb13-rt-amd64):

cat << 'EOF' > hello_world_mod.yaml
apiVersion: kmm.sigs.x-k8s.io/v1beta1
kind: Module
metadata:
  name: kmm-hello-world
  namespace: kernel-module-management
spec:
  moduleLoader:
    container:
      modprobe:
        moduleName: hello_world_dmesg
      kernelMappings:
        - literal: "6.18.15+deb13-rt-amd64"
          containerImage: registry.local:9001/kmm/kmm-hello-world:stx.13.0-6.18.15-1-rt-amd64
          registryTLS:
            insecure: false
            insecureSkipTLSVerify: true

  imageRepoSecret:
    name: "kmm-registry-secret"

  selector:
    kubernetes.io/os: linux
    kubernetes.io/hostname: controller-0

  tolerations:
    - key: "services"
      operator: "Equal"
      value: "disabled"
      effect: "NoExecute"
EOF

Note

As kernel versions change, the kernelMappings also need to be updated to match the version(s) in use.

Note

The example above selects by hostname, another common pattern is selecting by node type using the selector node-role.kubernetes.io/worker: “”. As seen on KMM docs.

Warning

The tolerations block shown in the examples is required. StarlingX applies the services=disabled:NoExecute taint to nodes during host-lock operations. Without this toleration, the KMM module-loader pods will be evicted from the node and the kernel module will not be loaded. Keep this tolerations block in every Module resource that follows.

To load the module, apply the CRD created by running the following command:

~(keystone_admin)]$ kubectl apply -f hello_world_mod.yaml

Warning

See kernel-module-management official documentation for more details on how to construct a Module custom resource. It is not possible to select documentation for a specific KMM version, so the docs do not take into consideration which version of the application is running. Some configurations may not behave as expected.

Building and loading the modules

You can create a CRD Module file and a ConfigMap as an additional step when you want the kernel-module-management application to build the module image. The ConfigMap will carry the Dockerfile content mentioned previously, and the CRD Module will reference the newly created ConfigMap.

cat << 'EOF' > hello_world_cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kmm-hello-world-cm
  namespace: kernel-module-management
data:
  dockerfile: |
    FROM docker.io/starlingx/kmm-builder:master-debian-trixie-amd64-stable-latest as builder
    ARG KERNEL_FULL_VERSION
    WORKDIR /usr/src
    RUN ["git", "clone", "https://opendev.org/starlingx/app-kernel-module-management.git"]
    WORKDIR /usr/src/app-kernel-module-management/tools/builder-image/debian/all/docker/hello-world-module
    RUN make KERNEL_DIR=/lib/modules/${KERNEL_FULL_VERSION}/build && mkdir /out && cp *.ko /out/

    FROM ubuntu
    ARG KERNEL_FULL_VERSION
    RUN apt-get update && apt-get install -y kmod
    COPY --from=builder /out/*.ko /opt/lib/modules/${KERNEL_FULL_VERSION}/
    RUN depmod -b /opt ${KERNEL_FULL_VERSION}
EOF

Warning

The KERNEL_FULL_VERSION variable should always match the current StarlingX kernel version, otherwise the module will not be loaded by the application.

Warning

With new versions of StarlingX, the kernel version could change, creating the necessity to rebuild the module image matching the new kernel and kmm-builder image versions, before reapplying it. Otherwise, the module will experience an outage while the system builds the new version. As of StarlingX release r12 the kmm-builder image matches kernel v6.12.57 and is Debian Bullseye based. The next release will run kernel 6.18 and be Debian Trixie based, which the kmm-builder image must match.

Use the Module example that matches the kernel loaded on the target node.

Example for the standard kernel (6.18.15+deb13-amd64):

cat << 'EOF' > hello_world_mod.yaml
apiVersion: kmm.sigs.x-k8s.io/v1beta1
kind: Module
metadata:
  name: kmm-hello-world
  namespace: kernel-module-management
spec:
  moduleLoader:
    container:
      modprobe:
        moduleName: hello_world_dmesg
      kernelMappings:
        - literal: "6.18.15+deb13-amd64"
          containerImage: registry.local:9001/kmm/kmm-hello-world:stx.13.0-6.18.15-1-amd64
          build:
            buildArgs:
              - name: KERNEL_FULL_VERSION
                value: "6.18.15+deb13-amd64"
            baseImageRegistryTLS:
              insecure: false
              insecureSkipTLSVerify: true
            dockerfileConfigMap:
              name: kmm-hello-world-cm
          registryTLS:
            insecure: false
            insecureSkipTLSVerify: true

  imageRepoSecret:
    name: "kmm-registry-secret"

  selector:
    kubernetes.io/os: linux
    kubernetes.io/hostname: controller-0

  tolerations:
    - key: "services"
      operator: "Equal"
      value: "disabled"
      effect: "NoExecute"
EOF

Example for the real-time kernel (6.18.15+deb13-rt-amd64):

cat << 'EOF' > hello_world_mod.yaml
apiVersion: kmm.sigs.x-k8s.io/v1beta1
kind: Module
metadata:
  name: kmm-hello-world
  namespace: kernel-module-management
spec:
  moduleLoader:
    container:
      modprobe:
        moduleName: hello_world_dmesg
      kernelMappings:
        - literal: "6.18.15+deb13-rt-amd64"
          containerImage: registry.local:9001/kmm/kmm-hello-world:stx.13.0-6.18.15-1-rt-amd64
          build:
            buildArgs:
              - name: KERNEL_FULL_VERSION
                value: "6.18.15+deb13-rt-amd64"
            baseImageRegistryTLS:
              insecure: false
              insecureSkipTLSVerify: true
            dockerfileConfigMap:
              name: kmm-hello-world-cm
          registryTLS:
            insecure: false
            insecureSkipTLSVerify: true

  imageRepoSecret:
    name: "kmm-registry-secret"

  selector:
    kubernetes.io/os: linux
    kubernetes.io/hostname: controller-0

  tolerations:
    - key: "services"
      operator: "Equal"
      value: "disabled"
      effect: "NoExecute"
EOF

To load the module, apply the CRD and ConfigMap created by running the following command:

~(keystone_admin)]$ kubectl apply -f hello_world_cm.yaml -f hello_world_mod.yaml

The kernel module build progress can be tracked using the “kubectl logs” command.

~(keystone_admin)]$ kubectl logs -n kernel-module-management kmm-hello-world-build-<POD_ID> -f

Note

Successful build pods are garbage-collected immediately, while failed build pods are always preserved and must be deleted manually by the administrator for the build to be restarted.

Regardless of whether the module was loaded with a pre-built image or built using the app, it is possible to confirm that the module is correctly loaded by running the following dmesg:

sudo dmesg

[34352.320647] systemd-sysv-generator[2076803]: Overwriting existing symlink /run/systemd/generator.late/radosgw.service with real service.
[102022.711368] e1000: enp0s3 NIC Link is Down
[102028.836500] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[102422.922297] e1000: enp0s3 NIC Link is Down
[102422.922706] e1000 0000:00:03.0 enp0s3: Reset adapter
[102425.122913] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[102582.800513] e1000: enp0s3 NIC Link is Down
[102586.903818] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[103642.175972] Hello, world!

Sign Kernel Modules

On Secure Boot enabled systems, kernel modules must be signed with a key pair that the kernel trusts (enrolled in the MOK database). When Secure Boot is disabled, signing is optional and modules can be loaded unsigned. When required, KMM can sign the module during the build, using the sign section of the kernel mapping.

Supported Key Format

The sign-file tool used by the KMM sign image (OpenSSL 3) requires:

• Certificate (public): DER format.

• Private key: PKCS#8 PEM format (file starts with “—–BEGIN PRIVATE KEY—–“”).

A PKCS#1 private key (—–BEGIN RSA PRIVATE KEY—–) is not accepted and makes the sign pod fail with:

At main.c:174:
- SSL error:1E08010C:DECODER routines::unsupported
sign-file: /run/secrets/key/key.pem

If you hit this error, convert the private key to PKCS#8 (see the following procedure).

Generate the Signing Key Pair

1. Create the OpenSSL configuration file.

   cat > signing.config <<'EOF'
   [ req ]
   default_bits       = 4096
   distinguished_name = req_distinguished_name
   prompt             = no
   string_mask        = utf8only
   x509_extensions    = myexts
   
   [ req_distinguished_name ]
   CN = KMM Signing Key
   
   [ myexts ]
   basicConstraints   = critical,CA:FALSE
   keyUsage           = digitalSignature
   extendedKeyUsage   = codeSigning
   EOF
   

2. Generate the certificate in DER format and the private key.

   openssl req -x509 -new -nodes -utf8 -sha256 -days 36500 -batch \
       -config signing.config \
       -outform DER -out my_signing_key_pub.der \
       -keyout my_signing_key.priv
   

3. Convert the private key to PKCS#8 PEM (required by sign-file).

   openssl pkcs8 -topk8 -nocrypt -in my_signing_key.priv -out my_signing_key_pkcs8.pem
   

4. Confirm the converted key has the expected header.

   head -1 my_signing_key_pkcs8.pem
   

   The output must be “—–BEGIN PRIVATE KEY—–“.

Create the Secrets

KMM requires two secrets in the “kernel-module-management” namespace: one holding the private key under the data field named key, and another holding the certificate under the data field named cert.

Procedure:

1. Create the two secrets.

   ~(keystone_admin)]$ kubectl create secret generic my-signing-key \
       --from-file=key=my_signing_key_pkcs8.pem -n kernel-module-management
   
   ~(keystone_admin)]$ kubectl create secret generic my-signing-key-pub \
       --from-file=cert=my_signing_key_pub.der -n kernel-module-management
   

2. Validate the certificate secret. The command should print an X.509 certificate.

   ~(keystone_admin)]$ kubectl get secret my-signing-key-pub -n kernel-module-management -o yaml \
       | awk '/cert:/{print $2; exit}' | base64 -d \
       | openssl x509 -inform der -text | head
   

3. Validate the private key secret. The command should print (—–BEGIN PRIVATE KEY—–).

   ~(keystone_admin)]$ kubectl get secret my-signing-key -n kernel-module-management -o yaml \
       | awk '/key:/{print $2; exit}' | base64 -d | head -1
   

Add the “sign” Section to the Module CRD

Add a sign block under the kernel mapping, referencing the two secrets and the module files to sign. Building on the hello_world example shown previously:

Example for the standard kernel (6.18.15+deb13-amd64):

- literal: "6.18.15+deb13-amd64"
  containerImage: registry.local:9001/kmm/kmm-hello-world:stx.13.0-6.18.15-1-amd64
  build:
    buildArgs:
      - name: KERNEL_FULL_VERSION
        value: "6.18.15+deb13-amd64"
    baseImageRegistryTLS:
      insecure: false
      insecureSkipTLSVerify: true
    dockerfileConfigMap:
      name: kmm-hello-world-cm
  registryTLS:
    insecure: false
    insecureSkipTLSVerify: true
  sign:
    keySecret:
      name: my-signing-key
    certSecret:
      name: my-signing-key-pub
    filesToSign:
      - /opt/lib/modules/6.18.15+deb13-amd64/hello_world_dmesg.ko

Example for the real-time kernel (6.18.15+deb13-rt-amd64):

- literal: "6.18.15+deb13-rt-amd64"
  containerImage: registry.local:9001/kmm/kmm-hello-world:stx.13.0-6.18.15-1-rt-amd64
  build:
    buildArgs:
      - name: KERNEL_FULL_VERSION
        value: "6.18.15+deb13-rt-amd64"
    baseImageRegistryTLS:
      insecure: false
      insecureSkipTLSVerify: true
    dockerfileConfigMap:
      name: kmm-hello-world-cm
  registryTLS:
    insecure: false
    insecureSkipTLSVerify: true
  sign:
    keySecret:
      name: my-signing-key
    certSecret:
      name: my-signing-key-pub
    filesToSign:
      - /opt/lib/modules/6.18.15+deb13-rt-amd64/hello_world_dmesg.ko

When applied, KMM runs a build pod (unsigned image) followed by a sign pod, which signs the listed .ko files and pushes the signed image to containerImage.

Verify the Signature

After the sign pod completes, the .ko file inside the signed image carries the signature. Confirm the signer by running the following commands:

modinfo -F signer    <path-to>/hello_world_dmesg.ko
modinfo -F signature <path-to>/hello_world_dmesg.ko

The signer field should match the CN used in the certificate (for example, KMM Signing Key).

Enroll the Public Key in the MOK

For the kernel to trust the signature on a Secure Boot enabled node, enroll the public certificate in the MOK database and reboot the node.

Procedure:

1. Import the public certificate and set an enrollment password when prompted.

   $ sudo mokutil --import my_signing_key_pub.der
   

2. Reboot the node and confirm the enrollment in the MOK Manager screen.

   Note

   Confirming the enrollment requires console access to the node.

3. After the reboot, verify that the key is trusted by the kernel.

   $ sudo keyctl list %:.machine
   

   The output should list the enrolled key.

See also

For full Secure Boot and MOK details, see the upstream kernel-module-management Secure Boot documentation.

Unload Kernel Module

To unload the kernel module, run the following command in the same path where the previous module files were created:

~(keystone_admin)]$ kubectl delete -f hello_world_cm.yaml -f hello_world_mod.yaml

For the scenario with a pre-built image, run the following command:

~(keystone_admin)]$ kubectl delete -f hello_world_mod.yaml

Confirm that the kernel module was successfully unloaded by running dmesg again and checking for the Goodbye message logged.

sudo dmesg

[102425.122913] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[102582.800513] e1000: enp0s3 NIC Link is Down
[102586.903818] e1000: enp0s3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[103642.175972] Hello, world!
[104469.925530] Goodbye, world!

Uninstall App-kernel-module-management Application

To uninstall the app-kernel-module-management application, run the following command:

~(keystone_admin)]$ system application-remove kernel-module-management

Delete the uploaded application by running the following command:

~(keystone_admin)]$ system application-delete kernel-module-management