Kubernetes Custom Configuration¶

Introduction¶

Kubernetes configuration can be customized during deployment by specifying bootstrap overrides in the localhost.yml file during the Ansible bootstrap process.

Note

Kubernetes custom configuration is only applicable to the bootstrap phase and runtime support is not available.

Custom configuration includes:

Configuring options on kube-apiserver such as feature gates and admission controllers,
Configuring options on kube-controller-manager such as node-monitor-period and pod-eviction-timeout,
Configuring options on kube-scheduler such as feature gates
Configuring options on kubelet such as maximum pods and enabling unsafe sysctls.

kube-apiserver configuration¶

The Kubernetes API server validates and configures data for the api objects which include pods, services, replicationcontrollers, and others. The API Server services REST operations and provides the frontend to the cluster’s shared state through which all other components interact.

For a list of all configurable options of kube-apiserver, see https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/ .

To set or override a kube-apiserver option, add the desired parameters to an apiserver_extra_args section in the localhost.yml.

Example usage:

apiserver_extra_args:
  admission-control-config-file: "/etc/kubernetes/admission-control-config-file.yml"
  audit-policy-file: "/etc/kubernetes/audit-policy-file.yml"
  default-not-ready-toleration-seconds: "35"
  default-unreachable-toleration-seconds: "35"
  feature-gates: "SCTPSupport=true,TTLAfterFinished=true,HugePageStorageMediumSize=true,RemoveSelfLink=false,MemoryManager=true"
  enable-admission-plugins: "NodeRestriction,PodNodeSelector"
  event-ttl: "20h"

kube-controller-manager configuration¶

The Kubernetes controller manager is a daemon that embeds the core control loops shipped with Kubernetes. A controller is a control loop that watches the shared state of the cluster through the apiserver and makes changes attempting to move the current state towards the desired state.

For a list of all configurable options of kube-controller-manager, see https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/ .

To set or override a kube-controller-manager option, add the desired parameters to an controllermanager_extra_args section in the localhost.yml .

Example usage:

controllermanager_extra_args:
  node-monitor-period: "4s"
  node-monitor-grace-period: "25s"
  pod-eviction-timeout: '35s'
  feature-gates: "TTLAfterFinished=true,MemoryManager=true"

kube-scheduler configuration¶

The Kubernetes scheduler is a control plane process which assigns Pods to Nodes. The scheduler determines which Nodes are valid placements for each Pod in the scheduling queue according to constraints and available resources. The scheduler then ranks each valid Node and binds the Pod to a suitable Node. Multiple different schedulers may be used within a cluster; kube-scheduler is the reference implementation.

For a list of all configurable options of kube-scheduler, see https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/> .

To set or override a kube-scheduler option, add the desired parameters to an scheduler_extra_args section in the localhost.yml .

Example usage:

scheduler_extra_args:
  feature-gates: "TTLAfterFinished=false"

kubelet configurations¶

The kubelet is the primary “node agent” that runs on each node.

For a list of all configurable options, see https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/ .

To set or override a kubelet option, add the desired parameters to an kubelet_configurations section in the localhost.yml .

Example usage:

kubelet_configurations:
  featureGates:
    MemoryManager: true
    HugePageStorageMediumSize: true

apiserver_extra_volumes, controllermanager_extra_volumes, scheduler_extra_volumes¶

Some options/parameters specified in apiserver_extra_args, controllermanager_extra_args and scheduler_extra_volumes refer to a configuration file. The contents of these files are configured in the bootstrap overrides (localhost.yml) with the apiserver_extra_volumes, controllermanager_extra_volumes and scheduler_extra_volumes definitions.

For instance, if admission plugins are configured and need additional configuration, that configuration should be set in a specific file referenced by the admission-control-config-file parameter.

See the example below where the ‘admission-control-config-file’ option and the ‘PodNodeSelector’ admission plugin is specified for kube-apiserver. Both of these options require the specification of a yaml file.

Example usage:

apiserver_extra_args:
  admission-control-config-file: "/etc/kubernetes/admission-control-config-file.yaml"
  enable-admission-plugins: "PodNodeSelector"

apiserver_extra_volumes:
  - name: admission-control-config-file
    mountPath: "/etc/kubernetes/admission-control-config-file.yaml"
    readOnly: true
    pathType: "File"
    content: |
      apiVersion: apiserver.config.k8s.io/v1
      kind: AdmissionConfiguration
      plugins:
      - name: PodSecurity
        configuration:
          apiVersion: pod-security.admission.config.k8s.io/v1beta1
          kind: PodSecurityConfiguration
          # Defaults applied when a mode label is not set.
          #
          # Level label values must be one of:
          # - "privileged" (default)
          # - "baseline"
          # - "restricted"
          #
          # Version label values must be one of:
          # - "latest" (default)
          # - specific version like "v1.24"
          defaults:
            enforce: "privileged"
            enforce-version: "latest"
            audit: "privileged"
            audit-version: "latest"
            warn: "privileged"
            warn-version: "latest"
          exemptions:
            # Array of authenticated usernames to exempt.
            usernames: []
            # Array of runtime class names to exempt.
            runtimeClasses: []
            # Array of namespaces to exempt.
            namespaces: []
      - name: pod-node-selector
        mountPath: "/etc/kubernetes/pod-node-selector.yaml"
        readOnly: true
        pathType: "File"
        content: |
          podNodeSelectorPluginConfig:
          clusterDefaultNodeSelector: name-of-node-selector
          namespace1: name-of-node-selector
          namespace2: name-of-node-selector

The example below enables kubernetes auditing which requires an audit-policy-file.yaml file to specify the details of what events should be audited.

Example usage:

apiserver_extra_args:
  audit-policy-file: /etc/kubernetes/audit-policy-file.yaml
  audit-log-path: /var/log/kubernetes/audit/audit.log

apiserver_extra_volumes:
  - name: audit-policy-file
    mountPath: "/etc/kubernetes/audit-policy-file.yaml"
    readOnly: true
    pathType: "File"
    content: |
      # Log all requests at the Metadata level.
      apiVersion: audit.k8s.io/v1
      kind: Policy
      rules:
      - level: Metadata
  - name: audit-log-path
    mountPath: "/var/log/kubernetes/audit/"
    readOnly: false
    pathType: "DirectoryOrCreate"

Complex Example configuration¶

apiserver_extra_args:
  admission-control-config-file: "/etc/kubernetes/admission-control-config-file.yml"
  audit-policy-file: "/etc/kubernetes/audit-policy-file.yml"
  default-not-ready-toleration-seconds: "35"
  default-unreachable-toleration-seconds: "35"
  feature-gates: "SCTPSupport=true,TTLAfterFinished=true,HugePageStorageMediumSize=true,RemoveSelfLink=false,MemoryManager=true"
  enable-admission-plugins: "NodeRestriction,PodNodeSelector"
  event-ttl: "20h"
  audit-log-path: "/var/log/kubernetes/audit/audit.log"
  audit-log-maxage: "1"
  audit-log-maxbackup: "2"
  audit-log-maxsize: "1"

scheduler_extra_args:
  feature-gates: "TTLAfterFinished=false"

controllermanager_extra_args:
  node-monitor-period: "4s"
  node-monitor-grace-period: "25s"
  pod-eviction-timeout: '35s'
  feature-gates: "TTLAfterFinished=true,MemoryManager=true"

kubelet_configurations:
  featureGates:
    MemoryManager: true
    HugePageStorageMediumSize: true

apiserver_extra_volumes:
  - name: admission-control-config-file
    mountPath: "/etc/kubernetes/admission-control-config-file.yml"
    pathType: "File"
    readOnly: true
    content: |
      apiVersion: apiserver.config.k8s.io/v1
      kind: AdmissionConfiguration
      plugins:
      - name: PodNodeSelector
        path: /etc/kubernetes/podnodeselector.yaml
  - name: pod-nodes-selector-plugin-config
    mountPath: "/etc/kubernetes/podnodeselector.yaml"
    pathType: "File"
    readOnly: true
    content: |
      podNodeSelecto+rPluginConfig:
      clusterDefaultNodeSelector: name-of-node-selector
      namespace1: name-of-node-selector
      namespace2: name-of-node-selector
  - name: audit-policy-file
    mountPath: "/etc/kubernetes/audit-policy-file.yml"
    pathType: "File"
    readOnly: true
    content: |
      # Log all requests at the Metadata level.
      apiVersion: audit.k8s.io/v1
      kind: Policy
      rules:
      - level: Metadata
  - name: audit-log-path
    mountPath: "/var/log/kubernetes/audit/"
    readOnly: false
    pathType: 'DirectoryOrCreate'

scheduler_extra_volumes:
  - name: sch-admission-control-config-file
    mountPath: "/etc/kubernetes/admission-control-config-file.yml"
    pathType: "File"
    readOnly: true
    content: |
      apiVersion:
      kind: AdmissionConfiguration
      plugins:
      - name: PodNodeSelector
        path: /etc/kubernetes/podnodeselector.yaml
  - name: sch-pod-nodes-selector-plugin-config
    mountPath: "/etc/kubernetes/podnodeselector.yaml"
    pathType: "File"
    readOnly: true
    content: |
      podNodeSelectorPluginConfig:
      clusterDefaultNodeSelector: name-of-node-selector
      namespace1: name-of-node-selector
      namespace2: name-of-node-selector
  - name: sch-audit-policy-file
    mountPath: "/etc/kubernetes/audit-policy-file.yml"
    pathType: "File"
    readOnly: true
    content: |
      # Log all requests at the Metadata level.
      apiVersion: audit.k8s.io/v1
      kind: Policy
      rules:
      - level: Metadata

controllermanager_extra_volumes:
   - name: cm-admission-control-config-file
     mountPath: "/etc/kubernetes/admission-control-config-file.yml"
     pathType: "File"
     readOnly: true
     content: |
       apiVersion: apiserver.config.k8s.io/v1
       kind: AdmissionConfiguration
       plugins:
       - name: PodNodeSelector
         path: /etc/kubernetes/podnodeselector.yaml
   - name: cm-pod-nodes-selector-plugin-config
     mountPath: "/etc/kubernetes/podnodeselector.yaml"
     pathType: "File"
     readOnly: true
     content: |
       podNodeSelectorPluginConfig:
       clusterDefaultNodeSelector: name-of-node-selector
       namespace1: name-of-node-selector
       namespace2: name-of-node-selector
   - name: cm-audit-policy-file
     mountPath: "/etc/kubernetes/audit-policy-file.yml"
     pathType: "File"
     readOnly: true
     content: |
       # Log all requests at the Metadata level.
       apiVersion: audit.k8s.io/v1
       kind: Policy
       rules:
       - level: Metadata