Configure Ceilometer for Telemetry Data Collection¶

Ceilometer is the data collection component of the StarlingX OpenStack Telemetry project. Its role is to efficiently poll metering data from StarlingX OpenStack services, collect event data by monitoring notifications on the message bus, and publish the normalized results to various targets for storage, analysis, and alarming.

For more details see OpenStack Ceilometer documentation.

Ceilometer Agents ¶

Ceilometer runs as three main components, which have different purpose and are independent from each other:

The Ceilometer Central Exporter is the central agent, a Deployment with a single replica deployed in one of the StarlingX OpenStack control nodes.

It polls the StarlingX OpenStack service APIs (Nova, Cinder, Glance, Neutron, etc.) on a fixed polling interval and publishes the results as metrics on its Prometheus metrics endpoint. Because it reaches its sources over the APIs, it’s not tied to any StarlingX OpenStack node in particular.
The Ceilometer Compute Exporter is the compute agent, a DaemonSet with one pod deployed per StarlingX OpenStack compute node.

It polls the local hypervisor for per-instance data (CPU, disk, network, memory meters, etc) and publishes those as metrics on its own Prometheus metrics endpoint. Because each Compute Agent pod only knows about the instances on its own node, the complete picture is the union of all of them, and this locality is reflected in how its metrics Service behaves (Metrics Endpoints and Metrics Services).
The Ceilometer Notification Agent is a Deployment with one replica per StarlingX OpenStack control node.

It is not a Prometheus Exporter, which means that it exposes no metrics endpoint at all. Instead, it consumes the notifications that StarlingX OpenStack services emit on AMQP bus (StarlingX OpenStack RabbitMQ), converts them into Ceilometer events, and republishes them for Aodh to evaluate in its events-based alarm pipeline. For more details see Configure Prometheus for Metrics Storage.

Metrics Endpoints and Metrics Services ¶

Each of the two Ceilometer exporters serves their metrics in a Prometheus format metrics endpoint at /metrics on port 9101. These endpoints are exposed by Metrics Services, which are Kubernetes Services deployed in the openstack namespace:

Exporter	Metrics Service (namespace `openstack`)	Service type	Port
Ceilometer Central Exporter	`osh-openstack-ceilometer-central`	headless (`clusterIP: None`)	`9101/TCP`, named `metrics`
Ceilometer Compute Exporter	`osh-openstack-ceilometer-compute`	ClusterIP with internalTrafficPolicy: Local	`9101/TCP`, named `metrics`

Why the central Service is headless. There is exactly one central agent pod, and nothing needs a stable virtual IP in front of it.

A headless Service has no clusterIP; DNS resolution returns the pod’s own address rather than a load-balanced VIP.

The practical consequence is that there is no virtual IP to query. The Service is reached by its DNS name, which resolves to the central agent pod itself (Verify each Exporter).

Why the compute Service is node-local. The compute agent’s metrics describe only the instances on its own node, so a request that landed on an arbitrary compute node’s pod would return an answer that is correct but not the one the operator asked for. internalTrafficPolicy: Local prevents that ambiguity structurally: traffic sent to the Service is served only by the agent on the same node as the caller, and a request from a node that runs no compute agent fails rather than being routed elsewhere.

Verify each Exporter ¶

Query each Ceilometer Exporter through its metrics Service DNS name. Both Service DNS names resolve from the platform node and from inside any pod in the cluster:

curl -s http://osh-openstack-ceilometer-central.openstack.svc.cluster.local:9101/metrics
curl -s http://osh-openstack-ceilometer-compute.openstack.svc.cluster.local:9101/metrics

What differs between the two exporters is not the command but where it may be run:

Ceilometer Central Exporter. There is one central agent, and its metrics Service is headless (Metrics Endpoints and Metrics Services), so the name resolves to that agent’s own pod address. The request answers from anywhere in the cluster.
Ceilometer Compute Exporter. The name resolves to a node-local Service, so the request must be issued from the compute node that runs the agent instance you want to read, or from a pod scheduled on that node. Identify the compute nodes with system host-list, and repeat the request from each of them to cover the whole cluster.

Note

When a request to the Ceilometer Compute metrics service is sent from a node that hosts no compute agent, the request will fail by design. The internalTrafficPolicy: Local refuses to route it to another node’s agent (Metrics Endpoints and Metrics Services).

Checking that an exporter’s metrics Service exists at all. Once the Ceilometer Exporter Services can be disabled by the operator (Enable and Disable the Metrics Services and ServiceMonitors), before diagnosing an endpoint, confirm they are deployed:

$ kubectl -n openstack get svc osh-openstack-ceilometer-central osh-openstack-ceilometer-compute

A missing Service means the corresponding metrics Service is disabled, not that the exporter has failed.

How the Prometheus Server Discovers the Ceilometer Exporters ¶

When the prometheus platform application is applied, its Prometheus server scrapes both Ceilometer exporters without the operator configuring anything. Two facts explain that, and both matter when something is missing:

The scrape path discovery is driven by Ceilometer ServiceMonitor objects. The ServiceMonitors configuration tells the Prometheus Operator which Service’s endpoints to scrape.

The Prometheus server then scrapes the pod endpoints behind that Service directly, bypassing the Service’s virtual IP entirely.

This is why “is the Prometheus server scraping it?” and “can I curl it?” are different questions. A scrape can be healthy while the curl in Verify each Exporter fails because it was run from the wrong node, and a curl can succeed while the ServiceMonitor that would cause a scrape does not exist.

Both Ceilometer ServiceMonitors are configured with a 30-second scrape interval and 10-second timeout.
Discovery works across namespaces because the platform application does not restrict it. The Ceilometer ServiceMonitors live in namespace openstack and the Prometheus server lives in monitoring.

The platform application ships with a default Prometheus Operator’s *SelectorNilUsesHelmValues settings that make the Prometheus Operator watch for ServiceMonitor objects cluster-wide.

The operator does no label or namespace wiring, and so a ServiceMonitor created anywhere in the cluster is picked up on the same basis.

kubectl get servicemonitor -n openstack
NAME                                                AGE
osh-openstack-ceilometer-servicemonitor-central     ...
osh-openstack-ceilometer-servicemonitor-compute     ...

Apply `Prometheus` before StarlingX OpenStack ¶

Apply the prometheus application before you apply StarlingX OpenStack if you want to collect Ceilometer metrics with Prometheus.

The prometheus application creates the monitoring resources that Prometheus uses to discover and collect Ceilometer metrics. If you apply StarlingX OpenStack first, Prometheus cannot automatically collect Ceilometer metrics.

Important

Apply prometheus before applying StarlingX OpenStack if you plan to use telemetry metrics, PromQL queries, dashboards, or metric-based alarms.

If you apply StarlingX OpenStack first, the application deployment completes successfully and reports no warning or alarm; however, Prometheus does not collect Ceilometer metrics.

To restore metric collection, apply prometheus and then re-apply StarlingX OpenStack.

Apply the applications in the following order:

system application-apply prometheus
system application-apply stx-openstack

If you have already applied StarlingX OpenStack without prometheus, recover by running:

system application-apply prometheus
system application-apply stx-openstack

Verify Ceilometer ServiceMonitors and Metrics Collection ¶

After you apply both prometheus and StarlingX OpenStack, verify that the Ceilometer monitoring resources exist and that Prometheus is collecting Ceilometer metrics.

Verify that both applications are applied:

system application-list | grep -E "prometheus||prefix|-openstack"

Verify the CRDs exist (created by applying prometheus).
```
kubectl get crd | grep monitoring.coreos.com
```

Verify that the Ceilometer ServiceMonitors exist:

kubectl get servicemonitor -n openstack

Example output:

NAME                                                  Age
osh-openstack-ceilometer-servicemonitor-central       14h
osh-openstack-ceilometer-servicemonitor-compute       14h

If the ServiceMonitors do not exist, apply prometheus and then re-apply StarlingX OpenStack.

Verify that Prometheus is scraping the Ceilometer targets:

kubectl -n monitoring get svc kube-prometheus-stack-prometheus

curl -s http://[prometheus-svc-ip]:9090/api/v1/targets \
  | jq '.data.activeTargets[] | select(.labels.job | test("ceilometer")) | {job: .labels.job, health}'

Example output:

{ "job": "osh-openstack-ceilometer-central", "health": "up" }
{ "job": "osh-openstack-ceilometer-compute", "health": "up" }

Verify that Prometheus has collected Ceilometer metrics:
```
curl -s http://[prometheus-svc-ip]:9090/api/v1/label/__name__/values \
  | jq -r '.data[]' | grep ceilometer
```
If the command returns Ceilometer metrics, Prometheus is successfully collecting telemetry data and making it available for queries, dashboards, and alarms.

Poll Configuration ¶

By default, Ceilometer is configured to poll a subset of meters documented at Telemetry Measurements. The list of meter classes officially supported by Ceilometer and the identification of the ones enabled by the StarlingX OpenStack is presented in the following table:

Meters	Enabled (Y/N)	Reason
OpenStack Compute	Y	-
IPMI	N	Ceilometer IPMI agent is not supported
OpenStack Image	Y	-
OpenStack Block Storage	Y	-
OpenStack File Share	N	OpenStack Manila service is not supported
OpenStack Object Storage	N	OpenStack Swift service is not supported
Ceph Object Storage	N	Ceph Object Gateway (radosgw) is not supported
OpenStack Identity	Y	-
OpenStack Networking	Y	-
VPN-as-a-Service (VPNaaS)	Y	-
Firewall-as-a-Service (FWaaS)	Y	-
Octavia Load Balancer	N	OpenStack Octavia service is not supported
Designate DNS	N	OpenStack Designate service is not supported
OpenStack alarming	N	OpenStack metrics endpoint (Gnocchi) is not supported

By default, all the meters are polled with an interval of 600 seconds. To change the subset of meters polled by StarlingX OpenStack or their polling interval, use a values file to override the full list of Ceilometer pollsters and meters.

The polling sources are a YAML list, and so any configuration change will be considered as a complete replacement of its current content rather than a merge into it. Therefore, the override file must include a complete source definition, including name, interval and meters.

The following example changes the default config by enabling all the pollsters and meters supported by Ceilometer and reducing the polling interval from 600s (10 minutes - default) to 60s (1 minute).

cat <<EOF > ceilometer-polling.yaml
conf:
  polling:
    sources:
      - name: all_pollsters
        interval: 60
        meters:
          - "*"
EOF

system helm-override-update |prefix|-openstack ceilometer openstack --values ceilometer-polling.yaml
system application-apply |prefix|-openstack

Three intervals compound into alarm latency, and lowering the wrong one achieves nothing. The delay between a condition arising and a metric-based alarm firing is the sum of the polling interval (how often Ceilometer collects the value, which is 600 seconds by default), the scrape interval (how often the Prometheus server pulls the metrics endpoint, which is 30 seconds by default), and Aodh’s own evaluation interval for the alarm.

The polling interval dominates by a wide margin at its default, so an operator who needs faster metrics-based alarms must reduce it.

Reducing only the scrape interval changes almost nothing, because the value being scraped is the same one until Ceilometer next polls.

Faster polling costs API load on the polled services and storage in the metric store, so treat it as a trade-off rather than a free improvement.

Enable and Disable the Metrics Services and ServiceMonitors ¶

Four Ceilometer Helm values govern whether the metrics plumbing is created: manifests.service_central_metrics and manifests.service_compute_metrics for the two metrics Services, and manifests.servicemonitor_central and manifests.servicemonitor_compute for the two ServiceMonitors. All four are true by default.

To turn one off, e.g., the compute ServiceMonitor:

cat <<EOF > ceilometer-monitoring.yaml
manifests:
  servicemonitor_compute: false
EOF

system helm-override-update |prefix|-openstack ceilometer openstack --values ceilometer-monitoring.yaml
system application-apply |prefix|-openstack

Setting the value back to true and applying again reverses it. As always, the override alone changes nothing until the apply runs.

Turning off a metrics Service also removes its ServiceMonitor. A ServiceMonitor is created only when three conditions hold together: (i) the monitoring.coreos.com/v1 API is present in the cluster (aka prometheus application has been applied); (ii) its own ServiceMonitor manifest toggle is true and the corresponding metrics Service is enabled. Therefore, disabling manifests.service_compute_metrics removes both the Service and the compute ServiceMonitor.

The reverse is not true, and it is deliberate. Disabling a ServiceMonitor leaves its metrics Service in place, because the Prometheus server is not necessarily the only consumer for the Ceilometer metrics. An external metrics consumer can scrape these metrics services directly (See Manage Alarms Using Aodh). Disabling the ServiceMonitor is therefore the correct way to stop the platform Prometheus application from collecting Ceilometer metrics while keeping them available to anything else.

Also note that the two ServiceMonitor toggles have no effect on a system where the monitoring.coreos.com/v1 API is absent. That is, where the prometheus application has never been applied. Nothing is broken by this; there is simply no Prometheus Operator to act on the objects.

Troubleshoot Ceilometer ¶

Symptom	Check	Action
A `curl` of the Ceilometer Compute Exporter’s metrics Service returns nothing	Which node the command ran on, and whether that node runs a compute agent: `kubectl -n openstack get pod -l application=ceilometer,component=compute -o wide`	Re-run from a node that hosts the agent (Verify each Exporter).
A `curl` of the Ceilometer Central Exporter’s metrics Service is written against an IP address, and there is no address to use	The Service is headless, it has no `clusterIP`: `kubectl -n openstack get svc osh-openstack-ceilometer-central`	Query the Service DNS name instead; it resolves to the central agent pod (Verify each Exporter)
The Prometheus server shows no Ceilometer targets, and no error anywhere	Whether the Ceilometer Compute and Central ServiceMonitor objects exist: `kubectl get servicemonitor -n openstack`	If one or both are missing, check its toggle and the corresponding metrics Service toggle; a disabled metrics Service silently removes its ServiceMonitor, so the target simply never appears (Enable and Disable the Metrics Services and ServiceMonitors)
Both ServiceMonitors are missing, StarlingX OpenStack reports `applied` with healthy pods, and no warning or alarm was raised	Whether the `monitoring.coreos.com` CRDs existed when StarlingX OpenStack was applied: `kubectl get crd \| grep monitoring.coreos.com`. The CRDs are created only when `prometheus` is applied	StarlingX OpenStack was applied before `prometheus`, so the ServiceMonitors were never rendered. Apply `prometheus`, then re-apply StarlingX OpenStack (Apply Prometheus before StarlingX OpenStack)
A metrics endpoint answers, but the Prometheus server still has no target for it	The two are independent paths. The endpoint is reachable through the Service, the scrape is driven by the ServiceMonitor against pod endpoints	Confirm the ServiceMonitor exists (above) and that the `prometheus` application is applied (Section 2.2). A reachable endpoint proves nothing about scraping (How the Prometheus Server Discovers the Ceilometer Exporters)
Metrics appear, but a changed polling interval has no effect	Whether the override set the correct `interval` in the source definition `system helm-override-show <appname> ceilometer openstack`	Re-apply the interval through the `conf.polling.sources` values file (Poll Configuration)
Alarms based on Ceilometer metrics fire much later than expected	The polling interval first, then the scrape and evaluation intervals	The default 600-second polling interval dominates the total latency; reduce it rather than the scrape interval (Poll Configuration)
`memory.*` metrics are missing for some instances but present for others	When the affected instances were created	Instances created before the Ceilometer support has been introduced keep their original domain definition and need a hard reboot to gain the memory balloon device that these meters depend on.
No Ceilometer metrics at all, from either exporter	Whether the agents are running: `kubectl -n openstack get pod -l application=ceilometer`	If the pods are absent, the corresponding Helm chart is disabled. Re-enable it with a Helm chart attribute followed by an apply (Section 1.3)

Configure Ceilometer for Telemetry Data Collection

Configure Ceilometer for Telemetry Data Collection¶

StarlingX

Page Contents