Kubernetes Pod Core Dump Handler

Introduction

The default coredump handler (systemd-coredump) is not capable of limiting the namespace access and has a global configuration for any application on the system. To solve these issues a new Kubernetes aware core dump handler was created, allowing the configuration of the coredump on a per pod basis and limiting the namespace access. The Pod Core Dump Handler allows you to set the core_pattern on a per Pod basis. When applications running on pods generate core dumps, they are sent to Kubernetes Coredump Handler for handling according to the pod annotations configurations.

Individual Pods can control the core dump handling by specifying Kubernetes Pod annotations that instruct the core dump handler for specific applications. In this way, the core dump will have a specific size, will be saved at a specific folder with a specific name, and you can allocate disk space on a per-pod basis.

See below the Kubernetes Pod Annotations.

Kubernetes Pod Annotations

The following are application annotations available for core dump configuration:

--starlingx.io/core_pattern: <pattern> (default “core.PID”)

This annotation is used to determine the path in the pod namespace where the core dump file is saved and the name that the core dump should have. In the example below, the pattern is saving inside the pod on the /coredump-log path (folder available in this specific pod). It is also possible to format the filename using arguments described in the core documentation. For more information see https://man7.org/linux/man-pages/man5/core.5.html. This annotation is required if you want to use the Kubernetes core dump handler. If this annotation is not used, the default core dump handler (systemd-coredump) will be used.

--starlingx.io/core_compression: none|lz4 (default “none”)

This annotation is used to determine whether the file will be compressed. Using the “lz4” option, the file will be compressed, decreasing the file size. Use “none” or don’t configure this annotation to not compress the file.

--starlingx.io/core_max_size: <size> (defaults to 0 - unlimited).

This annotation sets the maximum core dump file size. It can be set from bytes to gigabytes or as a percentage of total amount of disk space. If the file is larger than the defined size, the file will be truncated.

--starlingx.io/core_max_used: (defaults to 0 - unlimited)

This annotation sets the maximum amount of disk space to be used when saving the core dump. If the file is larger than the remaining space, the file will be truncated.

--starlingx.io/core_min_free: <size> (defaults to 0 - unlimited)

This annotation sets the minimum amount of disk space to keep free when saving the core dump. If the file is larger than the remaining space, the file will be truncated.

where:

--pattern

supports the core_pattern defined by the core documentation. For more information see https://man7.org/linux/man-pages/man5/core.5.html.

--size

supports standard size suffixes B, K, M, G or percentage (0 to disable)

YAML Example with all kubernetes-coredump-handler configuration annotations:

kind: Pod
apiVersion: v1
metadata:
  name: dummy-pod-with-annotation
  annotations: {
    starlingx.io/core_pattern: "/coredump-log/core.%P.%u.%g.%s.%t.%e",
    starlingx.io/core_compression: lz4,
    starlingx.io/core_max_size: 200k,
    starlingx.io/core_max_used: 50%,
    starlingx.io/core_min_free: 20%
    }
 spec:
   containers:
     - name: dummy-pod-with-annotation
     image: ubuntu
     command: ["/bin/bash", "-ec", "while :; do echo '.'; sleep 5 ; done"]
     volumeMounts:
       - name: hostpath-volume
         mountPath: /coredump-log
 volumes:
   - name: hostpath-volume
     hostPath:
       path: /var/lib/systemd/coredump/
 restartPolicy: Never