Linux Auditing System¶
This section describes the Linux Auditing System containerized solution for StarlingX. The container-based solution aligns with the modular architecture approach of the StarlingX Platform product.
The Linux Auditing System helps system administrators track security violation events based on preconfigured audit rules. The events are recorded in a log file and the information in the log entries helps to detect misuse or unauthorized activities. Some examples of auditable events are:
file or directory access (Such as files/directories that were accessed, modified, executed, or attributes changed)
system calls (For example, useradd, time-related system calls)
commands run by a user (For example, a rule can be defined for every executable in the /bin directory and tracked per user.)
security events, such as failed login attempts
network access (The iptables and ebtables utilities can be configured to trigger audit events.)
The Linux Audit daemon, auditd, is the main component of the Linux Auditing System, and is responsible for writing the audit logs. For more information on auditd daemon configuration, see https://man7.org/linux/man-pages/man8/auditd.8.html.
To run auditd on StarlingX Platform, you must enable auditd in the kernel of StarlingX Platform hosts and then upload and apply the auditd system application.
Enable Auditd in the Kernel¶
The Linux Auditing System is disabled in the StarlingX kernel by default.
To enable auditd in the kernel of all hosts in the system, set the system service parameter audit to ‘1’ and apply the service-parameter change, using the following commands, executed on the active controller.
~(keystone_admin)]$ system service-parameter-modify platform kernel audit=1
~(keystone_admin)]$ system service-parameter-apply platform
To persist the service parameter change, all hosts need to be locked and unlocked, using the following commands for each host depending on the deployed configuration:
For AIO-SX deployments:
~(keystone_admin)$ system host-lock controller-0
~(keystone_admin)$ system host-unlock controller-0
For AIO-DX and Standards deployments, after controller-1 is locked/unlocked swact controller-0 to make controller-1 the active node. The next set of commands are executed on controller-0 node:
~(keystone_admin)$ system host-lock controller-1
~(keystone_admin)$ system host-unlock controller-1
~(keystone_admin)$ system host-swact controller-0
On controller-1, after controller-0 is locked/unlocked swact controller-1 to go back to controller-0 as the active node. The next set of commands are executed on controller-1 node:
~(keystone_admin)$ system host-lock controller-0
~(keystone_admin)$ system host-unlock controller-0
~(keystone_admin)$ system host-swact controller-1
For each worker node in the configuration execute the commands from controller-0:
~(keystone_admin)$ system host-lock worker-0
~(keystone_admin)$ system host-unlock worker-0
To verify if the grub kernel parameter audit was updated to ‘1’, for a particular host, ssh to the host, and check the cmdline parameters, for example:
~(keystone_admin)]$ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-5.10.57-200.185.tis.el7.x86_64 root=UUID=e11d78a2-7e1c-4613-84c7-002647b1cf8d ro security_profile=standard module_blacklist=integrity,ima tboot=false crashkernel=512M biosdevname=0 console=ttyS0,115200 iommu=pt usbcore.autosuspend=-1 selinux=0 enforcing=0 nmi_watchdog=panic,1 softlockup_panic=1 softdog.soft_panic=1 intel_iommu=on user_namespace.enable=1 nopti nospectre_v2 nospectre_v1 hugepagesz=2M hugepages=0 default_hugepagesz=2M irqaffinity=2-3 rcu_nocbs=2-3 kthread_cpus=0-1 audit=1 audit_backlog_limit=8192
Note
Enabling auditd should only be done if the purpose is to start
auditd in the container/pod using the process described in
Start Auditd System Application.
Otherwise, there will be unnecessary performance impact and the backlog
events queue limit will eventually exceed, causing audit: kauditd hold
queue overflow messages to be displayed.
Start Auditd System Application¶
Prerequisites
Set the audit grub kernel parameter to ‘1’.
The auditd container that runs the auditd daemon must be started by uploading and applying the audit-armada-app.
The auditd system application is installed as part of the software install or upgrade.
The auditd system application tarball can be found after installation
in the /usr/local/share/applications/helm directory. The name of the
tarball is auditd-<version>.tgz, for example, auditd-1.0-2.tgz.
Use the following commands to upload and apply the auditd system application:
~(keystone_admin)]$ system application-upload /usr/local/share/applications/helm/auditd-1.0-2.tgz
# check the app was uploaded
~(keystone_admin)]$ system application-show auditd
# if status is "uploaded" proceed with app apply
~(keystone_admin)]$ system application-apply auditd
# check the app was applied
~(keystone_admin)]$ system application-show auditd
# if successful, status will be "applied"
To check that auditd container/pod is created and running on each master and worker node, use the following command:
~(keystone_admin)]$ kubectl get pods -n kube-system -o wide | grep auditd
ns-auditd-9hgq5 1/1 Running 0 2m46s face::e95d:7b0:368d:55f8 compute-0 <none> <none>
ns-auditd-btww5 1/1 Running 1 2m46s face::2d8f:b75d:d511:81ef compute-1 <none> <none>
ns-auditd-czsdf 1/1 Running 1 2m46s face::977:4894:111d:5bf0 compute-2 <none> <none>
ns-auditd-hs62t 1/1 Running 0 2m46s face::3 controller-1 <none> <none>
ns-auditd-nn8jw 1/1 Running 0 2m46s face::2 controller-0 <none> <none>
Auditd Configuration Overrides¶
The Auditd daemon specific configuration is available in the /etc/audit/auditd.conf
file. For more information, see, https://man7.org/linux/man-pages/man5/auditd.conf.5.html.
Besides the auditd main configuration file auditd.conf, auditd uses
audit rules configuration that is available in the /etc/audit/audit.rules
file which defines what audit events are logged. For more information on how
audit rules are configured, see https://linux.die.net/man/7/audit.rules.
In the StarlingX Platform containerized auditd solution, both configuration files have default settings that can be overwritten using Helm chart overrides.
The Helm chart overrides are applied using the following command:
~(keystone_admin)]$ system helm-override-update auditd auditd kube-system --reuse-values --values /home/sysadmin/<user_specific_config>.yaml
The <user_specific_config>.yaml defines the overrides that will apply either
to the auditd.conf and/or to the audit.rules files.
Note
The default values for auditd.conf should be sufficient and you do not
need to update them.
In rare cases, the following example describes how to update the default value with the desired value.
Example of user defined overrides file for auditd.conf:
auditdconf: |-
##########################################################################
#
# auditd.conf
#
##########################################################################
local_events = yes
write_logs = yes
log_file = /var/log/audit/audit.log
log_group = root
log_format = RAW
flush = INCREMENTAL_ASYNC
freq = 50
max_log_file = 8
num_logs = **8**
priority_boost = 4
disp_qos = lossy
dispatcher = /sbin/audispd
name_format = NONE
##name = mydomain
max_log_file_action = IGNORE
space_left = 75
space_left_action = SYSLOG
##verify_email = yes
##action_mail_acct = root
admin_space_left = 50
admin_space_left_action = SYSLOG
disk_full_action = SYSLOG
disk_error_action = SYSLOG
use_libwrap = yes
##tcp_listen_port = 60
##tcp_listen_queue = 5
##tcp_max_per_addr = 1
##tcp_client_ports = 1024-65535
##tcp_client_max_idle = 0
enable_krb5 = no
krb5_principal = auditd
distribute_network = no
Example of user defined overrides file for audit.rules:
auditdrules: |-
## First rule - delete all
-D
## Increase the buffers to survive stress events.
## Make this bigger for busy systems
-b 8192
## Set failure mode to syslog
-f 1
-a always,exit -F arch=b32 -S init_module,finit_module,delete_module -F key=modules
-a always,exit -F arch=b64 -S init_module,finit_module,delete_module -F key=modules
Note
The log rotation configuration in auditd.conf file must not be updated,
and must use the default value, max_log_file_action = IGNORE, since
the logrotate linux utility is used to manage auditd log rotation.
Apply the audit rules overrides using the following command:
~(keystone_admin)]$ system application-apply auditd
Check that application apply has completed successfully:
~(keystone_admin)]$ system application-show auditd
The Helm chart overrides system helm-override-update command,
automatically applies the additional rules from the user provided yaml file to
the audit.rules in the auditd container.
Similarly, configuration overrides can be applied to update the default
configuration of auditd.conf using the system helm-override-update
command.
Auditd logs¶
auditd logs can be viewed on the host in the /var/log/audit directory.
Logs are generated by the auditd daemon running in the container and the
logs record auditable events configured using the audit.rules file. Log
rotation is automatically configured by the system.
Disable Auditd¶
You may decide to disable auditd for performance reasons. First, you must remove the auditd application. Then, you must set the kernel service parameter audit to ‘0’. These steps removes the auditd containers on all hosts and the auditd application.
Use the following system commands to disable auditd:
To remove the auditd application:
~(keystone_admin)]$ system application-remove auditd
~(keystone_admin)]$ system application-delete auditd
To verify that the application does not exist in the system:
~(keystone_admin)]$ system application-list |grep auditd
To set the kernel service parameter audit to ‘0’:
~(keystone_admin)]$ system service-parameter-modify platform kernel audit=0
~(keystone_admin)]$ system service-parameter-apply platform
To persist the kernel parameter change, all hosts need to be locked and unlocked:
~(keystone_admin)]$ system host-lock controller-0
~(keystone_admin)]$ system host-unlock controller-0
