StarlingX AppArmor Support

Storyboard: https://storyboard.openstack.org/#!/story/2010310

This specification describes the introduction of AppArmor into the StarlingX solution, initially for the purposes of running end user’s hosted containers under AppArmor profiles.

AppArmor is a Mandatory Access Control (MAC) system built on Linux’s LSM (Linux Security Modules) interface. In practice, the kernel queries AppArmor before each system call to know whether the process is authorized to do the given operation. Through this mechanism, AppArmor confines programs to a limited set of resources.

AppArmor applies a set of rules (known as a “profile”) on each program. The profile applied by the kernel depends on the installation path of the program being executed, the rules applied do not depend on the user. All users face the same set of rules when they are executing the same program but traditional user permissions still apply and might result in different behavior.

AppArmor profiles contain a list of access control rules on resources that each program can make use of. Each profile can be loaded either in enforcing or complaining mode. The former enforces the policy and reports violation attempts, while the latter does not enforce the policy but still logs the system calls that would have been denied.

AppArmor helps administrators in running a more secure kubernetes deployment by restricting what containers/pods are allowed to do, and/or provide better auditing through system logs.The access needed by a container/pod is configured through profiles tuned to allow the access such as Linux capabilities, network access, file permissions, etc.

Problem description

Need a mechanism for restricting application pods run by the end users. AppArmor does this by allowing the system administrator to restrict programs’ capabilities with per-program profiles.Then Kubernetes’ container.apparmor.security.beta.kubernetes.io annotation can be used in a pod spec to assign an AppArmor profile to a pod.

Management of AppArmor profiles across all the kubernetes hosts of a StarlingX kubernetes cluster should be simple for end user.

There are concerns over the system performance degradation when enabling the AppArmor. However, some end users want it enabled for security reasons, and are willing to accept the system performance degradation.

Use Cases

AppArmor can be optionally enabled on a per host basis:

  • Default recommendation will be that AppArmor is enabled on all hosts

  • However, if required (e.g. to avoid performance impact on some cards), user could choose which host to enable AppArmor profiles (e.g., Enable AppArmor only on controllers in a Standard System) NOTE In this case, a user would need to use labels and node-selector in pod spec to ensure the pod is only scheduled on nodes with AppArmor enabled.

If AppArmor is enabled on host(s) of a StarlingX system, a user will be able to run selected user’s hosted application pods under a selected AppArmor profile by specifying the AppArmor profile in the pod’s spec.

NOTE Support for AppArmor profiles on the StarlingX Platform entities (both host processes and containers) themselves, is left to a future release.

Proposed change

This change provides a way to enable/disable on a particular host.

  • Default recommendation will be that AppArmor is enabled on all hosts

  • However, if required (e.g. to avoid performance impact on some hosts), user could choose which host to enable AppArmor profiles (e.g., Enable AppArmor only on controllers in a Standard System) NOTE In this case, a user would need to use labels and node-selector in pod spec to ensure the pod is only scheduled on nodes with AppArmor enabled.

In order apply the profiles to a particular pod, the profiles need to be available to the host machine where pod is launched. Security Profile Operator (SPO, https://github.com/kubernetes-sigs/security-profiles-operator) provides AppArmor profile management (i.e. loading/unloading) across k8s nodes. SPO defines an AppArmor Profile CRD, such that end users’ can define AppArmor profiles for SPO to manage.

SPO will be integrated as a system application using FluxCD manifest which utilizes helm chart that deploys a controller/operator for centrally managing the AppArmor Profile resources, a webhook for validation of AppArmor Profile resources and a daemonset for loading/unloading the AppArmor Profiles. The DaemonSet configuration creates an SPO pod/container on each host in the StarlingX cluster. Creating a DaemonSet configuration for AppArmor container ensures that SPO will run on all kubernetes nodes, such as controllers, all-in-one controllers and worker nodes.

Once an AppArmor profile is loaded to the K8S nodes, it can be applied to a particular pod using annotations. | example: container.apparmor.security.beta.kubernetes.io/<pod_name>:localhost/<profile_ref>. Please refer https://kubernetes.io/docs/tutorials/security/apparmor/#example

Alternatives

In order apply the profiles to a particular pod, the profiles need to be available to the host machine where pod is launched. There are several solutions available and below alternatives were evaluated:

  • security-profiles-operator

  • apparmor-loader

  • kube_apparmor_Manager

  • Manually copying and loading profiles on each host (DEV Options)

Below is a comparision table

Alternatives to load AppArmor profiles in K8s cluster

AppArmor Loader

Security Profile Operator(SPO)

AppArmor_Kube_Manager

Manually Loading Profiles (DEV options)

Community Type

K8S

K8S

Private, sysdiglab

NA

Approach

Daemon Set

Daemon Set

SSH based copy

Manual copy (SSH based)

Developement Type

POC

Controlled as Alpha, beta release

Not sure

NA

Active

Last updated 4 years back

Very Much Active

Last updated 2 years back

NA

Release Frequency

NA

Quarterly basis

NA

NA

Latest Version

No version control

V0.4.3

V0.0.5

NA

Last Release Data

Jun 07, 2022

Apr 04, 2020

NA

Roadmap

No

Yes, KEP and Roadmap in place

No

NA

Github Link

https://github.com/kubernetes/kubernetes/tree/master/test/images/apparmor-loader

https://github.com/kubernetes-sigs/security-profiles-operator

https://github.com/sysdiglabs/kube-apparmor-manager

NA

Functionality

Only Load AppArmor profiles on all worker nodes in k8s cluster

Out-of-tree Kubernetes enhancement | Supports SELinux, seccomp, AppArmor | AppArmor profiles load and unload | Feature enabling and disabling using feature gate | Supplementary features – Metrics , service monitoring, log enricher, | Debugging features- CPU and Memory profile endpoints

Load AppArmor profiles on all worker nodes in k8s cluster | Sync feature to cater profile change synchronize across worker nodes

Copy the profiles on each host using SSH | Load the profile on each host using aa-enforce, aa-complain, or apparmor_parser | Unload the profiles on each host using aa-disable or apparmer_parser

Security Profile Operator (SPO) is the best option available and will continue our effort in incorporating it in StarlingX based on below advantages.

  • Better community support

  • Feature rich

  • Roadmap in place

There are concerns about the stability of AppArmor support in SPO and the basic functionality of deploying AppArmor profiles to various nodes is not working at this point. We will continue our effort to track SPO fixes to apparmor functionality.

  • Retest updates to SPO and provide feedback to contributor quickly

  • Offer any assistance we can to SPO contributor

  • Identify any missing/desired functionality (e.g. complain vs enforce)

For now, we will proceed in parallel with

Data model impact

A host level attribute “apparmor” would be added to the host table. This can contain two possible values “enabled” or “disabled”

REST API impact

AppArmor can be enabled or disabled at runtime on a per-host basis.

  • Below cli command(s) would be used to enable or disable AppArmor in kernel for a particular host * system host-lock <hostname> * system host-update <hostname> apparmor=enabled/disabled Example: system host-update controller-0 apparmor=enabled * system host-unlock <hostname>

    The above command(s) would enable the AppArmor in kernel with NO services (on host or containerized) in the WRCP/StarlingX Solution running under an apparmor profile NOTE will explore whether default profiles enabled in Debian Bullseye can be used as default when AppArmor is enabled

  • Changes in the below API to reflect AppArmor. | GET /v1/ihosts/​{host_id}​ | Description: Shows detailed information about a specific host | INPUT: | host_id : csapi:UUID: part of user-info | Output: | “apparmor”: “enabled” or “apparmor”: “disabled” would be added to the output json |

  • Addition of a new attribute ‘apparmor’ to the below API:

PATCH /v1/ihosts/​{host_id}​
Apparmor attribute can be modified. Support would be added for apparmor (xsd:string)
INPUT:
host_id : csapi:UUID: part of user-info
Example: [
{
“path”:”/apparmor”,
“value”:”enabled”,
“op”:”replace”
}
]
Output:
“apparmor”: “enabled” or “apparmor”: “disabled” would be added to the output json
NOTE: Changing the ‘apparmor’ attribute, will ONLY be allowed when host is locked.

Security impact

The AppArmor helps in running a more secure kubernetes deployment by restricting what containers/pods are allowed to do. User should be able to run selected user’s hosted application pods under a selected AppArmor profile. AppArmor enforces the policy defined in the profile as well as reports policy violation attempts.

Other end user impact

  • If apparmor is NOT enabled (disabled by default), then there is NO impact on end user

  • If apparmor is used, * there is admin user impact to enable apparmor on hosts and load apparmor profiles on host and * there is general user impact that they ‘can’ run hosted containers under apparmor profiles for enhanced security.

Performance Impact

AppArmor is known to have a performance impact when running. There will be a performance impact characterization (StarlingX Infrastructure Operations and Datapaths) of enabling AppArmor in the StarlingX which involved getting data with:

  • apparmor disabled

  • apparmor enabled but NO apparmor profile applied to datapath app pod

  • apparmor enabled and apparmor profile applied to datapath app pod

Other deployer impact

Developer impact

None.

Upgrade impact

A new host level attribute “apparmor” would be added to the host table. There should not be any impact on backup and restore, upgrade, and rollback.

Implementation

Assignee(s)

Primary assignee: * Jagatguru Prasad Mishra (jmishra)

Other contributors: * Sirin Shaikh (sshaikh) * Kirti Singh (ksingh) * Rahul Roshan Kachchap (rkachchap)

Repos Impacted

  • starlingx/config and starlingx/stx-puppet repo would be modified to add API/CLI support for AppArmor.

  • A new repo would be created for security-profiles-operator.

Work Items

  • Integration of SPO as a system application

  • CLI, RESTAPI, DB update and config update backend

  • Performance impact testing

  • End-to-end integration testing

Dependencies

Testing

Testing will involve:

  • Unit test cases for the new code introduced

  • Installing/enabling the SPO system app

  • verifying the enabling/disabling of Apparmor at host level using API and CLI

  • Configuring Apparmor profiles across hosts

  • Configuring Kubernetes Pods with Apparmor profiles

  • Verifying that a policy violation by a container is logged in file “/var/log/syslog”.

  • Backup and Restore testing

  • Upgrades testing

  • DC testing

Documentation Impact

Some information that is useful to be documented will be:

  • enabling Apparmor per host

  • Configuring Apparmor profiles across hosts

  • Configuring Kubernetes Pods with Apparmor profiles with example to do simple test

  • Guidelines on how to write/develop apparmor profiles for your containers using aa-genprof

References

History

Revisions

Release Name

Description

STX-8.0

Introduced