Manage Alarms Using Aodh¶

StarlingX OpenStack uses Aodh to create and manage alarms for OpenStack services and resources. Aodh alarms are OpenStack Telemetry resources and operate independently of the platform fault management system.

StarlingX OpenStack supports the following alarm types:

Metric-based alarms (--type prometheus) monitor telemetry metrics from OpenStack services. The alarm evaluates a PromQL expression against a defined threshold and transitions between the ok and alarm states as metric values change. Metric-based alarms require the prometheus platform application and are used to monitor resource utilization and service health.
Event-based alarms (--type event) monitor OpenStack service events generated from service notifications processed by the Ceilometer notification agent. These alarms transition to the alarm state when a specified event_type occurs. Event-based alarms do not require the prometheus platform application and are used to monitor instance, volume, and image lifecycle events. After being triggered, an event-based alarm remains in the alarm state until it is reset or deleted.
Composite alarms (--type composite) combine multiple metric-based alarm conditions using logical operators such as and and or. Composite alarms are not covered in this document.

You can configure and manage alarms using the openstack alarm CLI, which is included with the StarlingX OpenStack.

The openstack alarm CLI communicates with the aodh-api service to create, view, update, and delete alarms. The aodh-api service stores alarm definitions and maintains their current state.

Metric-based alarms are evaluated by aodh-evaluator using metrics stored in the Prometheus TSDB, while event-based alarms are evaluated by aodh-listener using Ceilometer events generated from OpenStack service notifications. When an alarm changes state, aodh-notifier executes any configured alarm actions, typically by sending HTTP POST notifications to the specified action URLs.

Aodh alarms are separate from StarlingX fault management alarms and do not appear in the fm alarm-list output.

Create a Metric-Based Alarm ¶

In this release, Prometheus collects metrics from the Ceilometer exporters only. Therefore, alarm rules must use metrics provided by the following exporters:

Ceilometer Central Exporter
Ceilometer Compute Exporter

Metrics exposed by the OpenStack Exporter are not available for metric-based alarms. If an alarm rule references a metric that is not available in Prometheus, the alarm is created successfully but remains in the insufficient data state because Aodh has no metric data to evaluate.

Prerequisites

Verify that the prometheus platform application is applied:
```
system application-show prometheus
```
If the application is not applied, deploy it before continuing. For deployment instructions, see Configure Prometheus for Metrics Storage.
Verify that the required Aodh and Ceilometer services are running:
```
$ kubectl -n openstack get pod -l application=aodh
$ kubectl -n openstack get pod -l application=ceilometer
```
Confirm that all listed pods show a Running status in the output.

Procedure

Verify that the PromQL expression you plan to use returns metric data. Access the Prometheus web UI, run the expression from the Graph page, and verify that it returns data.

For more information, see Access the Prometheus Web UI in the StarlingX Admin Tasks > Kubernetes > Metrics Server > Configure Prometheus for Metrics Storage.

An alarm whose query returns no results cannot evaluate its rule and remains in the insufficient data state.
Create a metric-based alarm.

Specify the PromQL expression with --query and the threshold value with --threshold. These parameters are required for this alarm type. Use --comparison-operator to define how Aodh compares the query result to the threshold. Valid values are lt, le, eq, ne, ge, and gt. If you do not specify a comparison operator, Aodh uses eq by default.

The following example uses gt (greater than). Replace this with the operator appropriate for your condition.
```
openstack alarm create --type prometheus --name <alarm-name> \
  --query '<promql-expression>' \
  --comparison-operator gt \
  --threshold <value> \
  --alarm-action <action-url>
```

Verify that the alarm was created successfully.

openstack alarm list
openstack alarm show <alarm-name> -c state -c state_reason -c state_timestamp

Results

The aodh-evaluator service continuously reevaluates the alarm rule, allowing the alarm state to track the current metric value. Because the alarm can return to the ok state, you can configure an --ok-action to trigger a notification when the condition clears. If the alarm state never changes, verify that the metric returns data and that the PromQL expression matches the intended metric behavior.

Note

Aodh uses the Ceilometer polling interval, Prometheus scrape interval, and Aodh evaluation interval to determine how quickly it detects and reports a condition. Before you tune any of these settings, review the Ceilometer configuration documentation. It explains how the intervals interact and helps you identify the most effective settings to adjust for faster alarm response times.

Create an Event-Based Alarm ¶

Event-based alarms monitor OpenStack service events and transition to the alarm state when aodh-listener receives a matching event, as specified by the --event-type option.

Prerequisites

Verify that the required Aodh and Ceilometer services are running. The prometheus platform application is not required for event-based alarms.

$ kubectl -n openstack get pod -l application=aodh
$ kubectl -n openstack get pod -l application=ceilometer,component=notification

Verify that all listed pods show a Running status in the output.

Procedure

Create the alarm.

Specify the event to monitor with --event-type. Set --repeat-actions to true to trigger an alarm action for every matching event. If you do not set this option, only the first matching event triggers an alarm action.

openstack alarm create --type event --name <alarm-name> \
  --event-type <event-type> \
  --repeat-actions true \
  --alarm-action <action-url>

For example, to create an alarm for image deletion events:

openstack alarm create --type event --name image-delete-alarm \
  --event-type image.delete \
  --repeat-actions true \
  --alarm-action <action-url>

Verify the alarm configuration.

Newly created event-based alarms start in the insufficient data state with a state reason of Not evaluated yet.
```
openstack alarm show <alarm-name> -c type -c event_type -c repeat_actions -c state -c state_reason
```
Generate a matching event and verify the alarm state.

When aodh-listener receives a matching event, the alarm state changes to alarm and the state_reason field identifies the event that triggered the alarm.
```
openstack alarm show <alarm-name> -c state -c state_reason -c state_timestamp
```

Results

When Aodh receives a matching event, it changes the alarm state from insufficient data to alarm and executes the configured alarm actions. Unlike metric-based alarms, event-based alarms do not return to the ok state. After the first matching event, the alarm remains in the alarm state; subsequent matching events update state_reason but do not change the alarm state.

Filter Event-Based Alarms¶

In addition to --event-type, you can use --query to filter events based on event traits.

Traits are event attributes that the Ceilometer notification agent extracts from service notifications.

Specify each filter using the following format:

traits.<trait-name><operator><type>::<value>

For example: traits.status=string::error

Where:

<operator> is one of =, !=, <, <=, >, or >=.
<type> is one of string, integer, float, or boolean.

You can combine multiple filter expressions by separating them with a semicolon (;).

For example, the following alarm monitors image.delete events only when the image size exceeds 1 GiB:

openstack alarm create --type event --name large-image-delete-alarm \
     --event-type image.delete \
     --query 'traits.size>integer::1073741824' \
     --repeat-actions true \
     --alarm-action <action-url>

Note

Aodh evaluates a trait filter only when the specified trait is present in the event payload. If the trait is absent, the event does not match the alarm condition and Aodh does not trigger any alarm actions. Traits are service-specific and can vary between events of the same type. For example, some image.delete events include a size trait, while others do not. Before creating a trait-based filter, verify that the required trait is present in the events you want to monitor.

Update Event Type and Trait Definitions¶

StarlingX OpenStack provides a predefined set of event types and traits in the ceilometer-notification container. These definitions are stored in Ceilometer containers at /etc/ceilometer/event_definitions.yaml and follow the Ceilometer Event Definitions Format.

Procedure

View the current event type and trait definitions:

$ kubectl -n openstack exec -t pod/ceilometer-notification-<pod_id> -- \
 cat /etc/ceilometer/event_definitions.yaml

Create a Helm override file to add or modify event type and trait definitions.

For example:

cat <<EOF > ceilometer-events.yaml
conf:
  event_definitions:
    - event_type: ['volume.exists', 'volume.create.*', 'volume.delete.*',
                   'volume.resize.*', 'volume.attach.*',
                   'volume.detach.*', 'volume.update.*',
                   'snapshot.exists', 'snapshot.create.*',
                   'snapshot.delete.*', 'snapshot.update.*']
      traits: &cinder_traits
        user_id:
          fields: payload.user_id
        project_id:
          fields: payload.tenant_id
        availability_zone:
          fields: payload.availability_zone
        display_name:
          fields: payload.display_name
        replication_status:
          fields: payload.replication_status
        status:
          fields: payload.status
        created_at:
          fields: payload.created_at

    - event_type: ['volume.exists', 'volume.create.*', 'volume.delete.*',
                   'volume.resize.*', 'volume.attach.*',
                   'volume.detach.*', 'volume.update.*']
      traits:
        <<: *cinder_traits
        resource_id:
          fields: payload.volume_id
        host:
          fields: payload.host
        size:
          fields: payload.size
        type:
          fields: payload.volume_type
        replication_status:
          fields: payload.replication_status
EOF

Apply the Helm override:

system helm-override-update stx-openstack ceilometer openstack   --values ceilometer-events.yaml

Apply the updated OpenStack application:
```
system application-apply stx-openstack
```

Warning

The Ceilometer conf.event_definitions parameter is defined as a list. Any override replaces the entire existing configuration rather than merging with it. When you override event definitions, include the complete set of event types and traits that Ceilometer must support. Any event type or trait omitted from the override is removed and is no longer available to Ceilometer or to Aodh event-based alarms.

Configure HTTP Notification Endpoints ¶

An alarm action defines a URL that aodh-notifier calls with an HTTP POST request when an alarm changes state. Use one or more of the following options with openstack alarm create to configure notification URLs:

--alarm-action: Runs when the alarm enters the alarm state.
--ok-action: Runs when the alarm enters the ok state.
--insufficient-data-action: Runs when the alarm enters the insufficient data state.

You can configure multiple URLs for each action type.

Alarm actions can use either an IP address or a DNS name. The notification endpoint can be located inside or outside the cluster.

Note

Use a Kubernetes Service DNS name rather than a pod IP address. Pod IP addresses can change when pods are restarted or recreated, which can cause alarm notifications to fail. Service DNS names provide a stable endpoint for notification delivery.

Procedure

Verify that the notification endpoint is reachable.

curl -s -o /dev/null -w '%{http_code}\n' -X POST \
  http://<receiver-service>.<namespace>.svc.cluster.local:<port> \
  -H 'Content-Type: application/json' -d '{"probe":"pre-check"}'

A successful response returns HTTP status code 200.

Create an alarm and configure the notification endpoint.

openstack alarm create --type event --name image-delete-alarm \
  --event-type image.delete \
  --repeat-actions true \
  --alarm-action http://<receiver-service>.<namespace>.svc.cluster.local:<port>

Trigger the alarm condition and verify that the endpoint receives the notification.

The notification payload contains alarm information, including alarm_name, alarm_id, severity, previous, current, reason, and reason_data.

For event-based alarms, reason_data contains the matching event, including its event_type and associated traits.

Results

When the alarm changes state, the configured endpoint receives an HTTP POST notification containing the alarm details.

Configure HTTPS Notification Endpoints with a Private CA ¶

By default, aodh-notifier validates HTTPS action URLs using the system CA bundle. If the notification endpoint uses a certificate signed by a private CA, configure Aodh to trust that CA before using the endpoint as an alarm action.

The private CA certificate location is specified using the Helm parameter aodh_notifier.tls.host_cert. The default path is:

/var/opt/openstack/certs/aodh-rest-notifier.pem

When you apply the StarlingX OpenStack application, the certificate is loaded into the aodh-notifier deployment and used to validate HTTPS notification endpoints.

Procedure

Copy the private CA certificate to /var/opt/openstack/certs/aodh-rest-notifier.pem on the active controller.

Update the Helm override if the certificate is not at the default path.

~(keystone_admin)]$ system helm-override-update stx-openstack aodh openstack --set aodh_notifier.tls.host_cert=/path/to/private-ca.pem

Apply the application.

~(keystone_admin)]$ system application-apply stx-openstack

Verify that the certificate was loaded successfully.

$ kubectl -n openstack get secret aodh-rest-notifier-ca-cert
$ kubectl -n openstack get deployment aodh-notifier -o jsonpath='{.spec.template.spec.volumes[*].name}{"\n"}'

Create or update the alarm to use an https:// action URL as described in Configure HTTP Notification Endpoints.

Results

Alarm notifications are delivered to HTTPS endpoints that use certificates signed by the configured private CA.

Note

To rotate the certificate, replace the file on the host, reapply the application, and restart the notifier deployment.

~(keystone_admin)]$ system application-apply stx-openstack
~(keystone_admin)]$ kubectl -n openstack rollout restart deployment aodh-notifier

Note

A successful application apply does not guarantee that the private CA certificate was configured. If the certificate file is missing, the application applies successfully, but the aodh-rest-notifier-ca-cert secret is not created and HTTPS notifications signed by the private CA continue to fail. Verify that the secret exists and is mounted correctly by running the verification commands in the preceding step.

Alarm Behaviour Without the Prometheus Application ¶

Metric-based alarms (--type prometheus) depend on metrics stored in the Prometheus TSDB. When the prometheus platform application is not applied or has been removed, the aodh-evaluator service cannot query metrics and stops evaluating metric-based alarms.

Event-based alarms (--type event) are not affected. Their evaluation path runs from OpenStack service notifications through RabbitMQ and the Ceilometer Notification Agent to aodh-listener, and does not involve the Prometheus server at any point. Event-based alarms continue to operate normally regardless of whether the prometheus application is applied.

Note

If the prometheus application is applied after |prefix|-openstack is already running, you must reapply |prefix|-openstack so that the Prometheus application is detected and metric-based alarm evaluation is enabled.

~(keystone_admin)]$ system application-apply stx-openstack

Troubleshoot OpenStack Alarm Issues ¶

The following table lists Aodh alarm issues and recommended actions.

Issue	Resolution
A metric-based alarm (`--type prometheus`) remains in the `insufficient data` state.	Verify that the `prometheus` platform application is applied. If it is not applied, see Alarm Behaviour Without the Prometheus Application. Alternatively, use an event-based alarm.
A metric-based alarm that references an OpenStack Exporter metric never evaluates.	OpenStack Exporter metrics are not collected into the Prometheus TSDB used by Aodh. Create metric-based alarms using Ceilometer exporter metrics instead.
An event-based alarm triggers a notification only for the first matching event.	Check the `repeat_actions` setting. If `repeat_actions` is `false`, update the alarm with `openstack alarm update <alarm-id> --repeat-actions true`.
An event-based alarm never returns to the `ok` state.	This behaviour is expected. Event-based alarms remain in the `alarm` state after the first matching event. Subsequent matching events update the alarm details but do not return the alarm to `ok`.
A trait filter specified with `--query` never matches an event.	Verify that the specified trait exists in the event payload. Filters that reference absent traits do not match and do not generate an error.
`openstack alarm-history show <alarm-name>` returns a 404 error.	Alarm history queries require an alarm ID rather than an alarm name. Retrieve the ID using `openstack alarm list` and rerun the command.
An `https://` alarm action fails while the same endpoint works over HTTP.	Verify that the private CA certificate is available in the `aodh-rest-notifier-ca-cert` secret. If the secret is missing, correct the certificate path, reapply the StarlingX OpenStack application, and restart the Aodh notifier deployment.