StarlingX AppArmor Support¶
Storyboard: https://storyboard.openstack.org/#!/story/2010310
This specification describes the introduction of AppArmor into the StarlingX solution, initially for the purposes of running end user’s hosted containers under AppArmor profiles.
AppArmor is a Mandatory Access Control (MAC) system built on Linux’s LSM (Linux Security Modules) interface. In practice, the kernel queries AppArmor before each system call to know whether the process is authorized to do the given operation. Through this mechanism, AppArmor confines programs to a limited set of resources.
AppArmor applies a set of rules (known as a “profile”) on each program. The profile applied by the kernel depends on the installation path of the program being executed, the rules applied do not depend on the user. All users face the same set of rules when they are executing the same program but traditional user permissions still apply and might result in different behavior.
AppArmor profiles contain a list of access control rules on resources that each program can make use of. Each profile can be loaded either in enforcing or complaining mode. The former enforces the policy and reports violation attempts, while the latter does not enforce the policy but still logs the system calls that would have been denied.
AppArmor helps administrators in running a more secure kubernetes deployment by restricting what containers/pods are allowed to do, and/or provide better auditing through system logs.The access needed by a container/pod is configured through profiles tuned to allow the access such as Linux capabilities, network access, file permissions, etc.
Problem description¶
Need a mechanism for restricting application pods run by the end users. AppArmor does this by allowing the system administrator to restrict programs’ capabilities with per-program profiles.Then Kubernetes’ container.apparmor.security.beta.kubernetes.io annotation can be used in a pod spec to assign an AppArmor profile to a pod.
Management of AppArmor profiles across all the kubernetes hosts of a StarlingX kubernetes cluster should be simple for end user.
There are concerns over the system performance degradation when enabling the AppArmor. However, some end users want it enabled for security reasons, and are willing to accept the system performance degradation.
Use Cases¶
AppArmor can be optionally enabled on a per host basis:
Default recommendation will be that AppArmor is enabled on all hosts
However, if required (e.g. to avoid performance impact on some cards), user could choose which host to enable AppArmor profiles (e.g., Enable AppArmor only on controllers in a Standard System) NOTE In this case, a user would need to use labels and node-selector in pod spec to ensure the pod is only scheduled on nodes with AppArmor enabled.
If AppArmor is enabled on host(s) of a StarlingX system, a user will be able to run selected user’s hosted application pods under a selected AppArmor profile by specifying the AppArmor profile in the pod’s spec.
NOTE Support for AppArmor profiles on the StarlingX Platform entities (both host processes and containers) themselves, is left to a future release.
Proposed change¶
This change provides a way to enable/disable on a particular host.
Default recommendation will be that AppArmor is enabled on all hosts
However, if required (e.g. to avoid performance impact on some hosts), user could choose which host to enable AppArmor profiles (e.g., Enable AppArmor only on controllers in a Standard System) NOTE In this case, a user would need to use labels and node-selector in pod spec to ensure the pod is only scheduled on nodes with AppArmor enabled.
In order apply the profiles to a particular pod, the profiles need to be available to the host machine where pod is launched. Security Profile Operator (SPO, https://github.com/kubernetes-sigs/security-profiles-operator) provides AppArmor profile management (i.e. loading/unloading) across k8s nodes. SPO defines an AppArmor Profile CRD, such that end users’ can define AppArmor profiles for SPO to manage.
SPO will be integrated as a system application using FluxCD manifest which utilizes helm chart that deploys a controller/operator for centrally managing the AppArmor Profile resources, a webhook for validation of AppArmor Profile resources and a daemonset for loading/unloading the AppArmor Profiles. The DaemonSet configuration creates an SPO pod/container on each host in the StarlingX cluster. Creating a DaemonSet configuration for AppArmor container ensures that SPO will run on all kubernetes nodes, such as controllers, all-in-one controllers and worker nodes.
Once an AppArmor profile is loaded to the K8S nodes, it can be applied to a particular pod using annotations. | example: container.apparmor.security.beta.kubernetes.io/<pod_name>:localhost/<profile_ref>. Please refer https://kubernetes.io/docs/tutorials/security/apparmor/#example
Alternatives¶
In order apply the profiles to a particular pod, the profiles need to be available to the host machine where pod is launched. There are several solutions available and below alternatives were evaluated:
security-profiles-operator
apparmor-loader
kube_apparmor_Manager
Manually copying and loading profiles on each host (DEV Options)
Below is a comparision table
AppArmor Loader |
Security Profile Operator(SPO) |
AppArmor_Kube_Manager |
Manually Loading Profiles (DEV options) |
|
---|---|---|---|---|
Community Type |
K8S |
K8S |
Private, sysdiglab |
NA |
Approach |
Daemon Set |
Daemon Set |
SSH based copy |
Manual copy (SSH based) |
Developement Type |
POC |
Controlled as Alpha, beta release |
Not sure |
NA |
Active |
Last updated 4 years back |
Very Much Active |
Last updated 2 years back |
NA |
Release Frequency |
NA |
Quarterly basis |
NA |
NA |
Latest Version |
No version control |
V0.4.3 |
V0.0.5 |
NA |
Last Release Data |
Jun 07, 2022 |
Apr 04, 2020 |
NA |
|
Roadmap |
No |
Yes, KEP and Roadmap in place |
No |
NA |
Github Link |
https://github.com/kubernetes/kubernetes/tree/master/test/images/apparmor-loader |
https://github.com/kubernetes-sigs/security-profiles-operator |
NA |
|
Functionality |
Only Load AppArmor profiles on all worker nodes in k8s cluster |
Out-of-tree Kubernetes enhancement | Supports SELinux, seccomp, AppArmor | AppArmor profiles load and unload | Feature enabling and disabling using feature gate | Supplementary features – Metrics , service monitoring, log enricher, | Debugging features- CPU and Memory profile endpoints |
Load AppArmor profiles on all worker nodes in k8s cluster | Sync feature to cater profile change synchronize across worker nodes |
Copy the profiles on each host using SSH | Load the profile on each host using aa-enforce, aa-complain, or apparmor_parser | Unload the profiles on each host using aa-disable or apparmer_parser |
Security Profile Operator (SPO) is the best option available and will continue our effort in incorporating it in StarlingX based on below advantages.
Better community support
Feature rich
Roadmap in place
There are concerns about the stability of AppArmor support in SPO and the basic functionality of deploying AppArmor profiles to various nodes is not working at this point. We will continue our effort to track SPO fixes to apparmor functionality.
Retest updates to SPO and provide feedback to contributor quickly
Offer any assistance we can to SPO contributor
Identify any missing/desired functionality (e.g. complain vs enforce)
For now, we will proceed in parallel with
using DEV option of linux AppArmor tools (aa commands, https://gitlab.com/apparmor/apparmor/-/wikis/ManPages) to load apparmor profiles on hosts AND
work closely in testing and providing feedback (and possibly changes) to SPO project
Data model impact¶
A host level attribute “apparmor” would be added to the host table. This can contain two possible values “enabled” or “disabled”
REST API impact¶
AppArmor can be enabled or disabled at runtime on a per-host basis.
Below cli command(s) would be used to enable or disable AppArmor in kernel for a particular host * system host-lock <hostname> * system host-update <hostname> apparmor=enabled/disabled Example: system host-update controller-0 apparmor=enabled * system host-unlock <hostname>
The above command(s) would enable the AppArmor in kernel with NO services (on host or containerized) in the WRCP/StarlingX Solution running under an apparmor profile NOTE will explore whether default profiles enabled in Debian Bullseye can be used as default when AppArmor is enabled
Changes in the below API to reflect AppArmor. | GET /v1/ihosts/{host_id} | Description: Shows detailed information about a specific host | INPUT: | host_id : csapi:UUID: part of user-info | Output: | “apparmor”: “enabled” or “apparmor”: “disabled” would be added to the output json |
Addition of a new attribute ‘apparmor’ to the below API:
Security impact¶
The AppArmor helps in running a more secure kubernetes deployment by restricting what containers/pods are allowed to do. User should be able to run selected user’s hosted application pods under a selected AppArmor profile. AppArmor enforces the policy defined in the profile as well as reports policy violation attempts.
Other end user impact¶
If apparmor is NOT enabled (disabled by default), then there is NO impact on end user
If apparmor is used, * there is admin user impact to enable apparmor on hosts and load apparmor profiles on host and * there is general user impact that they ‘can’ run hosted containers under apparmor profiles for enhanced security.
Performance Impact¶
AppArmor is known to have a performance impact when running. There will be a performance impact characterization (StarlingX Infrastructure Operations and Datapaths) of enabling AppArmor in the StarlingX which involved getting data with:
apparmor disabled
apparmor enabled but NO apparmor profile applied to datapath app pod
apparmor enabled and apparmor profile applied to datapath app pod
Other deployer impact¶
Developer impact¶
None.
Upgrade impact¶
A new host level attribute “apparmor” would be added to the host table. There should not be any impact on backup and restore, upgrade, and rollback.
Implementation¶
Assignee(s)¶
Primary assignee: * Jagatguru Prasad Mishra (jmishra)
Other contributors: * Sirin Shaikh (sshaikh) * Kirti Singh (ksingh) * Rahul Roshan Kachchap (rkachchap)
Repos Impacted¶
starlingx/config and starlingx/stx-puppet repo would be modified to add API/CLI support for AppArmor.
A new repo would be created for security-profiles-operator.
Work Items¶
Integration of SPO as a system application
CLI, RESTAPI, DB update and config update backend
Performance impact testing
End-to-end integration testing
Dependencies¶
Security Profile Operator (SPO, https://github.com/kubernetes-sigs/security-profiles-operator)
Testing¶
Testing will involve:
Unit test cases for the new code introduced
Installing/enabling the SPO system app
verifying the enabling/disabling of Apparmor at host level using API and CLI
Configuring Apparmor profiles across hosts
Configuring Kubernetes Pods with Apparmor profiles
Verifying that a policy violation by a container is logged in file “/var/log/syslog”.
Backup and Restore testing
Upgrades testing
DC testing
Documentation Impact¶
Some information that is useful to be documented will be:
enabling Apparmor per host
Configuring Apparmor profiles across hosts
Configuring Kubernetes Pods with Apparmor profiles with example to do simple test
Guidelines on how to write/develop apparmor profiles for your containers using aa-genprof
References¶
https://www.appvia.io/blog/how-to-use-the-security-profiles-operator
https://medium.com/information-and-technology/so-what-is-apparmor-64d7ae211ed
https://github.com/kubernetes-sigs/security-profiles-operator
https://github.com/kubernetes/kubernetes/tree/master/test/images/apparmor-loader
https://kubernetes.io/docs/tutorials/security/apparmor/#example
History¶
Release Name |
Description |
---|---|
STX-8.0 |
Introduced |