Configure Prometheus for Metrics Storage

Overview

The prometheus platform application provides metric storage and query capabilities for the observability stack. It collects metrics from endpoints exposed by other applications, stores them in a time-series database (TSDB), and makes them available for PromQL queries and Aodh alarms.

For more information about the observability stack, see Observability Overview.

The prometheus application is an optional platform application based on the kube-prometheus-stack Helm chart. Its resources are deployed in the monitoring namespace.

The prometheus application includes the following components:

• A Prometheus Operator that discovers metrics endpoints through ServiceMonitor resources created by other applications and configures the Prometheus server to collect metrics from those endpoints.

• A Prometheus server that collects and stores metrics in a time-series database (TSDB).

A newly applied prometheus application does not collect any metrics by default. All built-in scrape targets provided by the kube-prometheus-stack Helm chart are disabled, and Grafana and Alertmanager are not deployed.

The following table lists the built-in components that are disabled by default and what their absence means for the deployed application:

Disabled Built-in Components

Built-in component	Default state	What its absence means
nodeExporter	Disabled	No host metrics — no CPU, memory, filesystem, or network series for the nodes themselves.
kubeStateMetrics	Disabled	No Kubernetes object state metrics — nothing about pods, deployments, DaemonSets, or nodes as objects.
kubelet	Disabled	No kubelet metrics and no cAdvisor container metrics.
kubeApiServer	Disabled	No Kubernetes API server metrics.
kubeControllerManager	Disabled	No controller-manager metrics.
kubeScheduler	Disabled	No scheduler metrics.
kubeEtcd	Disabled	No etcd metrics.
kubeProxy	Disabled	No kube-proxy metrics.
coreDns	Disabled	No cluster DNS metrics.
grafana	Disabled	No dashboard ships. The only graphical interface is the Prometheus server’s own web UI, reached by port forwarding (see Access the Prometheus Web UI).
alertmanager	Disabled	No alerting mechanism ships — no Alertmanager, no notification routing, no silences, and no Prometheus alerting rules.

The Prometheus server collects metrics only from targets exposed through ServiceMonitor resources created by other applications. For example, when the StarlingX OpenStack application is applied, it deploys Ceilometer ServiceMonitor resources that are used for service discovery by the Prometheus server.

Within the cluster, the Prometheus server is reachable at kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090.

Apply the Prometheus Application

By default, the prometheus application is uploaded but not applied on a newly installed system. Apply prometheus before applying stx-openstack if your deployment requires telemetry metric collection, metric-based alarms, or PromQL queries. If stx-openstack is applied first, telemetry monitoring does not function correctly even though the application reports a successful deployment. To recover, apply prometheus and then re-apply stx-openstack.

For information on configuring Ceilometer for telemetry data collection, see Configure Ceilometer for Telemetry Data Collection.

The application can be configured using the system application-* and system helm-override-* commands. When working with overrides, use the following values:

• Application name: prometheus

• Helm chart: kube-prometheus-stack

• Namespace: monitoring

Before the application is applied, several validation checks are performed:

• Bootstrap validation prevents automatic application deployment while platform bootstrap is still in progress.

• StorageClass availability validation ensures that the configured Kubernetes StorageClass exists in the cluster. For more information, see Configure Storage for the Prometheus Application.

• StorageClass immutability validation prevents changing the configured StorageClass after the application has been applied.

The system rejects the apply operation and returns an appropriate error message if any of these validations fail.

Procedure

1. Verify that the prometheus application is present and in the uploaded state:

   system application-list
   

2. Apply the application:

   system application-apply prometheus
   

3. Confirm the application is running:

   system application-show prometheus
   # Wait until the status is applied
   
   kubectl -n monitoring get pods
   # Verify that the Prometheus components are running
   

Configure Storage for the Prometheus Application

The prometheus application stores collected metrics in a Prometheus time-series database (TSDB). The TSDB uses a Kubernetes PVC for persistent storage. You select the storage backend through a Kubernetes StorageClass.

The application can use any storage backend as long as the associated Kubernetes StorageClass is available in the cluster.

The following storage configurations were tested with StarlingX:

• Ceph RBD (general storage class)

• NetApp NAS/NFS

• NetApp SAN iSCSI

• NetApp SAN FC

This list reflects the storage backends that were tested. Backends not listed may also be supported, provided they have a working StorageClass.

Review the default values in the table below before applying the application. You must configure the StorageClass before applying the application; it cannot be changed afterwards. The PVC size should also be set before the first apply, because a later override does not resize an already-bound claim unless the StorageClass permits volume expansion. To change the StorageClass, you must remove and reapply the application, which permanently deletes the PVC and all stored metric data (see Remove the Prometheus Application).

The Prometheus server retains metrics based on both a time limit and a storage size limit. Metrics are removed when the first of these limits is reached. On busy systems with high metric volumes, the storage size limit may be reached before 15 days, resulting in a shorter retention period. To retain metrics for a longer period, configure a larger PVC size before deploying the Prometheus application.

Setting	Default Value	Description
PVC size [1]	20Gi	Specifies the storage capacity available for Prometheus TSDB metrics.
StorageClass [1]	general	Identifies the Kubernetes storage class that provisions the PVC.
Time retention	15d	Removes metrics older than 15 days.
Size retention	18Gi	Limits TSDB storage to approximately 90% of the default 20Gi PVC size.
walCompression	true	Compresses the write-ahead log (WAL) to reduce disk usage with minimal CPU overhead.
enableAdminAPI	false	Disables the Prometheus administrative API. As a result, you cannot delete series data or create TSDB snapshots through the API.
Prometheus server replicas	1 or 2	In multi-controller configurations, Prometheus deploys two replicas with a required anti-affinity rule, ensuring that each replica runs on a different controller. Each replica gets its own PVC of the configured size; they share the StorageClass but do not share the volume. Plan for twice the PVC size when sizing storage for a two-controller system.

Footnotes

[1] (1,2)

The StorageClass is immutable after the first apply, and changing it requires removing and redeploying the application, which deletes the PVC and all stored metrics. Although PVC size can sometimes be expanded later, the initial size is applied when the claim is created and is not automatically adjusted by subsequent overrides. Therefore, plan both the storage class and capacity requirements before applying the application. Set these values while the application is still being uploaded, then apply.

Procedure

1. List the StorageClasses available in the cluster.

   ~(keystone_admin)$ kubectl get storageclass
   

2. Set the StorageClass to use.

   ~(keystone_admin)$ system helm-override-update prometheus kube-prometheus-stack monitoring \
     --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName=<class>
   

3. Apply the application.

   ~(keystone_admin)$ system application-apply prometheus
   

Access the Prometheus Web UI

The Prometheus web UI is not accessible directly from outside the Kubernetes cluster. External access through Ingress with TLS is not provided in this release. To access the UI, create a temporary port-forwarding session from your workstation to the Prometheus service.

Procedure

1. Forward a local port to the Prometheus service:

   ~(keystone_admin)$ kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090
   

Results

While the port-forward session is running, access the Prometheus web UI at: http://localhost:9090.

The Status > Targets page displays the targets currently being scraped by the Prometheus server and serves as the authoritative source for scrape status information.

The Graph page allows you to enter and test PromQL expressions and verify metric data collected by Prometheus. You can use the page to validate queries before using them in metric-based Aodh alarms or by external applications that consume Prometheus metrics.

Apply Prometheus in a Distributed Cloud

In a Distributed Cloud configuration, you must apply the prometheus application on the Central Cloud before applying it on any subcloud. The Central Cloud application makes the required application images available to the subclouds.

Each subcloud instance is independent. It has its own Prometheus server, TSDB, PVC, and retention window. Instances do not federate, replicate, or share data. A query against one subcloud returns only that subcloud’s metrics. Removing the application on one subcloud has no effect on any other.

To apply the prometheus application in a Distributed Cloud configuration:

Procedure

1. Verify that the prometheus application is in the uploaded state on the Central Cloud and on each subcloud. The Distributed Cloud installation uploads the application automatically.

   system application-list
   

2. On the Central Cloud, apply the prometheus application.

   system application-apply prometheus
   system application-show prometheus
   

   Wait until the status changes to applied before proceeding to the next system. This step populates the registry with the application images required by the subclouds.

3. On each subcloud that requires local metric collection and storage, apply the prometheus application.

   system application-apply prometheus
   system application-show prometheus
   

   Wait until the application status changes to applied. Note that this step succeeds only after the prometheus application has been successfully applied on the Central Cloud.

Remove the Prometheus Application

Removing the prometheus application permanently deletes the PVC, the monitoring namespace, and all stored metric data. The system does not create a backup before removal. In addition, the TSDB snapshot API is not available because enableAdminAPI is disabled by default.

Prerequisites

If you need to retain metric data, extract the required information through the PromQL query API before removing the prometheus application.

Procedure

1. To remove the application:

   system application-remove --force prometheus
   

Note

The --force option is required to remove the application. This requirement ensures that the removal is intentional, because removing the application also deletes the Prometheus Operator, the PVC, and all stored metric data.

When you re-apply the application after removal, it creates a new, empty TSDB on a new PVC.

Prometheus Application: Common Issues and Resolutions

The following table lists common issues and recommended actions for the prometheus platform application. Rows marked as by-design require no corrective action.

Prometheus Troubleshooting

Symptom	Check	Resolution
system application-apply prometheus is rejected immediately, naming storage.	Check whether any StorageClass exists: kubectl get storageclass	Configure a storage backend that provides a StorageClass, then apply again (see Configure Storage for the Prometheus Application).
The apply is rejected and the message names a StorageClass that does not exist.	Compare the class named in the override against the classes that exist: system helm-override-show prometheus kube-prometheus-stack monitoring and kubectl get storageclass	The rejection message lists the available classes and prints the corrective system helm-override-update command. Run it with a class from that list, then apply.
The Prometheus web UI cannot be reached from a browser.	Check that the port-forward command is running and that the local port is not in use.	There is no Ingress and the Service is a ClusterIP. Start a port-forward and access the UI at http://localhost:9090 (see Access the Prometheus Web UI).
The Prometheus server has only Ceilometer scrape targets and no host or Kubernetes platform metrics are present.	Check which ServiceMonitor resources are registered: kubectl get servicemonitor -A This application scrapes only the targets that other applications register through ServiceMonitor resources. It does not scrape host or Kubernetes platform metrics. The nodeExporter, kubeStateMetrics, kubelet, API server, controller manager, scheduler, etcd, kube-proxy, and CoreDNS targets are all disabled by design. With the StarlingX OpenStack application applied, only Ceilometer targets are expected.	This is by design. No action is required.
There is no dashboard and no way to define an alerting rule.	Grafana and Alertmanager are both disabled by design. The Prometheus web UI is the only graphical interface deployed.	Use the Graph page in the web UI to run PromQL queries (see Access the Prometheus Web UI).
An override that changes the StorageClass is rejected on an applied application.	Confirm the application is applied: system application-show prometheus	The StorageClass can only be changed while the application is uploaded, not applied. Remove the application and apply it again with the new class (see Configure Storage for the Prometheus Application).
system application-remove prometheus is refused.	Check whether the command was issued without --force.	Repeat the command with --force. Before proceeding, note that removal deletes the PVC and all stored metrics (see Remove the Prometheus Application).
On a subcloud, system application-apply prometheus fails to pull images.	Check whether the application has been applied on the Central Cloud: system application-show prometheus	Apply it on the Central Cloud first. That apply populates the central registry (see Apply Prometheus in a Distributed Cloud).
The Prometheus server has no Ceilometer targets even though both prometheus and StarlingX OpenStack report applied.	Check whether the Ceilometer ServiceMonitors exist: kubectl get servicemonitor -n openstack If they are missing, StarlingX OpenStack was applied before prometheus created the monitoring.coreos.com CRDs, so the ServiceMonitors were never rendered.	Apply prometheus first, then re-apply StarlingX OpenStack so the ServiceMonitors render. For information on configuring Ceilometer for telemetry data collection, see Configure Ceilometer for Telemetry Data Collection.