StarlingX OpenStack 13.0¶
ISO image¶
The pre-built ISO (Debian) for StarlingX 13.0 is located at the
StarlingX mirror repo:
OpenStack Helm Charts:
Source Code for StarlingX 13.0¶
The source code for StarlingX 13.0 is available on the r/stx.13.0 branch in the StarlingX repositories.
Deployment¶
To deploy StarlingX 13.0, see Consuming StarlingX.
For detailed installation instructions, see StarlingX 13.0 Installation Guides.
New Features / Enhancements / Limitations¶
This release introduces several new features designed to improve usability and efficiency, along with targeted enhancements to existing functionality. Performance, stability, and overall user experience have been enhanced. Some features may have known limitations or constraints in this release; these are documented to help set expectations and guide usage.
Update from StarlingX OpenStack Epoxy to StarlingX OpenStack Gazpacho (SLURP)¶
StarlingX OpenStack Release 13.0 has been updated from OpenStack Epoxy to OpenStack Gazpacho (Skip-Level Upgrade Release Process - SLURP).
Gazpacho is designated as a SLURP release, which introduces a new upgrade version in OpenStack.
Gazpacho is the OpenStack release codename for StarlingX OpenStack Release 2026.1.
Upversion of OpenStack Services to Gazpacho (2026.1)¶
StarlingX OpenStack Release 13.0 updated all core OpenStack services from OpenStack 2025.1 (Epoxy) to OpenStack 2026.1 (Gazpacho). These include:
OpenStack Gazpacho for new installations of StarlingX OpenStack 13.0, delivered as part of the SLURP (Skip Level Upgrade Release Process) release.
Gazpacho on K8s v1.35 (default).
Functionality and services provided in the Gazpacho initial release of April 1, 2026.
This update is delivered as an in-service application update, with the following updates:
OpenStack configuration preserved across the application.
Minimal elapsed time for the application.
Minimal OpenStack control plane service disruption.
Hosted virtual machines remain in service throughout the update.
Note
Nova services run in Eventlet concurrency mode by default, in accordance
with the StarlingX OpenStack Concurrency Policy. This is the default and validated
configuration, and it is recommended for all deployments. Although the
concurrency mode can be changed through Helm overrides, this is not
recommended.
See: https://releases.openstack.org/gazpacho/index.html.
See: The Deprecation and Removal Notices section below.
Debian 13 (Trixie) OS Alignment for StarlingX OpenStack in StarlingX 13.0¶
StarlingX OpenStack Release 13.0 aligns with the StarlingX Release 13.0 base operating system Debian 13 (Trixie), requiring updates across OpenStack maintained host packages and containerized services.
This feature delivers updates required to ensure OpenStack components are aligned with the Trixie-based platform on StarlingX Release 13.10.
OpenStack Release 13.10 is supported on StarlingX Release 13.0 and includes the following updates:
Kubernetes versions supported in StarlingX OpenStack Release 13.0:
1.32.2
1.32.13
1.33.0
1.34.1
1.34.8
1.35.2
Supported default version in StarlingX OpenStack 13.0 for new installs is:
1.35.2
Extended Storage Backend (ESB)¶
This feature introduces ESB in StarlingX OpenStack, describes its capabilities and supported storage protocols, and outlines how to determine whether your type of deployment can be supported by ESB.
Earlier, StarlingX OpenStack supported only a predefined set of platform-managed storage backends (Ceph, NetApp NFS, NetApp iSCSI, and NetApp FC), which were automatically detected and configured. ESB extends this capability to additional storage backends by enabling the configuration of any upstream OpenStack Cinder driver included with the platform as a custom backend.
An extended backend is a storage backend that you define yourself. You declare it once in the Cinder Helm chart overrides, and StarlingX OpenStack handles the pod-level settings for the specified protocol, injects credentials from a Kubernetes Secret, and passes your driver configuration to Cinder unchanged. StarlingX OpenStack does not validate the driver configuration, so you are responsible for ensuring that the configuration is valid.
See:
OpenStack Ceilometer Support for Resource Usage and Telemetry Collection¶
This feature enables resource-owner Telemetry in StarlingX OpenStack through the deployment of OpenStack Ceilometer and its native Prometheus exporter. Ceilometer polling agents running on the control and compute planes collect resource and workload metrics from OpenStack services and hypervisors, then expose the data through a standard Prometheus-compatible endpoint for integration with Analytics, Prometheus, and Aodh.
The collected Telemetry provides visibility into tenant-level resource utilization and virtual machine workload behavior, including per-tenant storage utilization for volumes, images, and snapshots, as well as per-VM CPU utilization, memory usage, disk I/O, network I/O, and other performance metrics. The Ceilometer central agent gathers metrics from OpenStack services such as Cinder, Glance, and Neutron, while the compute agent collects virtualization metrics locally from libvirt on each compute node.
This capability enables operators and resource owners to monitor infrastructure and workload usage, support capacity planning, and improve operational observability across the OpenStack cloud.
Configure Ceilometer for Telemetry Data Collection¶
Ceilometer is the data collection component of the StarlingX OpenStack Telemetry project. Its role is to efficiently poll metering data from StarlingX OpenStack services, collect event data by monitoring notifications on the message bus, and publish the normalized results to various targets for storage, analysis, and alarming.
See:
OpenStack Installation Overview
Ceilometer Service Monitor for Prometheus¶
StarlingX OpenStack exposes two integration interfaces for external observability platforms:
Metrics endpoints exposed by the Ceilometer Central Exporter, Ceilometer Compute Exporter, and OpenStack Exporter, which can be scraped directly.
An HTTP endpoint to receive alarm actions generated by StarlingX OpenStack.
An external platform can use either or both interfaces independently.
OpenStack Aodh (Alarming)¶
StarlingX OpenStack uses Aodh to create and manage alarms for OpenStack services and resources. Aodh alarms are OpenStack Telemetry resources and operate independently of the platform fault management system.
OpenStack Metrics Export for Prometheus¶
The StarlingX OpenStack application provides observability capabilities that give you
visibility into the health, performance, and operational state of your cloud.
With components such as the Ceilometer exporters, OpenStack Exporter, Aodh
services, and the optional prometheus platform application, you can collect
and store metrics, process service events, evaluate alarms, and expose
Telemetry data to monitoring and Analytics systems.
This feature enhances infrastructure-level observability of the OpenStack cloud by deploying a Prometheus OpenStack exporter within OpenStack. The exporter queries OpenStack services, including Nova, Cinder, Neutron, Glance, Keystone, Heat, and Placement, and exposes infrastructure metrics in Prometheus format for consumption by external analytics and monitoring systems.
The exported metrics provide operators with visibility into the health, utilization, and capacity of the OpenStack infrastructure. Key metrics include service and agent availability by host, hypervisor resource capacity and utilization (vCPUs, memory, and storage), tenant quota consumption and limits, network resource inventory (networks, subnets, ports, routers, and floating IPs), Placement allocation ratios, and Heat stack status. This capability enables proactive monitoring, capacity planning, and faster identification of infrastructure issues across the OpenStack environment.
See:
OVS-DPDK Support for Bonded (AE) Interfaces¶
StarlingX OpenStack now supports bonded (AE) data interfaces with OVS-DPDK for a validated configuration consisting of a two-member, data-class AE interface operating in active-standby mode on hosts configured with OVS-DPDK. The bonded interface can be used with a single data network of either VLAN or flat type.
Configure Horizon Apache Worker Recycling¶
In deployments with sustained Horizon usage, the Apache WSGI worker processes that serve the dashboard may accumulate memory over time. This is expected behavior for long-running Python/Django processes. Starting with StarlingX OpenStack Release 13.0, worker recycling is enabled by default to prevent unbounded memory growth.
Dex SSO Setup and Verification¶
This enhancement provides details for configuring and validating Dex-based Single Sign-On (SSO) integration for Horizon and Keystone in the OpenStack environment. Users are guided through the complete deployment and verification workflow, enabling validation of each component after the OpenStack application is applied.
This applies to all supported OpenStack deployment models, including AIO-SX, AIO-DX, Standard, and Distributed Cloud subcloud deployments. Common configuration and verification steps are consolidated into a single workflow, with deployment-specific guidance provided only where required. For Distributed Cloud deployments, details for additional RedirectURI registration performed through an Ansible playbook are provided.
Known Limitations and Procedural Changes for StarlingX 13.0 OpenStack¶
Neutron Helm Chart Apply Fails with Overrides Using the Yaml List Syntax¶
The Neutron chart supported both YAML list syntax and comma-separated syntax from StarlingX OpenStack Release 12.0.
yaml list syntax
conf:
neutron:
DEFAULT:
service_plugins:
- router
- network_segment_range
- qos
comma-separated syntax
conf:
neutron:
DEFAULT:
service_plugins: router,network_segment_range,qos
However, for StarlingX OpenStack Release 13.0, Neutron supports only the comma-separated syntax. The previously supported YAML list syntax is no longer accepted and can cause the Release 13.0 application apply operation, as well as the OpenStack application from Release 12.0 to Release 13.0, to fail.
Procedural Changes: Use the system helm-override-update CLI to replace any Neutron user overrides or apply new ones following the comma-separated syntax accepted by OpenStack 13.0. For application updates, update the overrides before triggering the system application-update, otherwise the update operation is expected to fail.
Nova Running-Deleted Reaper May Delete Active VMs and Attached Volumes¶
The Nova running_deleted_instance periodic task uses the default reap action.
If the Nova database and a compute host become out of sync, such that the
database marks an instance as deleted while the instance continues to run on
the host, the task can destroy the running instance.
For boot-from-volume instances configured with delete-on-termination, the associated volume can also be deleted when the instance is reaped. As a result, a temporary inconsistency between the Nova database and compute host state, such as one that may occur during an application reapply operation, can lead to the permanent loss of a tenant VM and its backing volume.
Procedural Changes: Configure the Nova running_deleted_instance_action
parameter to log in the Nova static overrides. This prevents the reaper task
from deleting instances and volumes when a mismatch exists between the Nova
database and compute host state. Instead, the discrepancy is logged, allowing
an administrator to investigate and resolve the issue without risking data loss.
Apply the override and reapply stx-openstack to ensure that all nova-compute services are running with the updated setting.
conf:
nova:
DEFAULT:
running_deleted_instance_action: log
Note
The poll interval must stay greater than 0 (default 60). The setting takes effect per nova-compute when its pod restarts.
Note
Setting the Nova running_deleted_instance_action parameter to log prevents
the reaper task from deleting instances and volumes when a database and
host state mismatch occurs. However, a separate orphan cleanup
process runs whenever a nova-compute pod starts, during compute
unlock, and application reapply. If a deleted-but-running
instance is detected, this process can remove the instance’s libvirt domain
on the next nova-compute restart. While the instance is removed, any
associated volume is preserved because the cleanup process does not invoke
the volume service.
This behavior is planned to be addressed in a future release. Until then, administrators should investigate and reconcile any deleted-but-running instances and avoid unnecessary nova-compute restarts while such instances exist.
Public HTTP Endpoint Unreachable Over IPv6 on IPv6-Only OAM Networks¶
In deployments that use an IPv6-only OAM network and expose public endpoints over HTTP (port 80), the public HTTP endpoint may be unreachable over IPv6. The network policy that opens port 80 for NGINX traffic is rendered for IPv4 only, resulting in the rule being applied to the IPv4 firewall chain but not the IPv6 chain.
As a result, clients on the OAM network, as well as stx-openstack pods running on a controller that does not host the OAM floating IP, cannot access public HTTP endpoints over IPv6. This may cause operations that depend on those endpoints to time out.
This limitation affects only HTTP (port 80) access on IPv6-only OAM deployments. Deployments that use HTTPS (port 443) or IPv4-based OAM networks are not affected.
Procedural Changes: Configure the nginx-ports-control Helm chart to use IPv6 and reapply stx-openstack.
Procedure
Create an override file (e.g. /home/sysadmin/nginx-ports-control-override.yaml).
global_network_policy: ingress: ipVersion: 6
Apply the override and reapply the application:
$ system helm-override-update stx-openstack nginx-ports-control openstack --values <override-file> $ system application-apply stx-openstack
Verify that the controller does not hold the OAM floating IP and verify that the
gnp-for-nginx-portspolicy is rendered with ipVersion: 6 and that connections to the OAM floating IP on port 80 succeed over IPv6.
Note
This workaround configures the policy for IPv6 only. While appropriate for IPv6-only OAM deployments, it removes IPv4 support for port 80 and should not be used on dual-stack OAM networks, which require both IPv4 and IPv6 connectivity.
This update is planned to be addressed in a future release.
libvirt Pod Crash-Loop on a Wrong-Type Path and nova-compute Identity Loss¶
On StarlingX OpenStack Release 12.0, certain libvirt failures can persist after a controller reboot or service restart and may prevent OpenStack compute services from operating correctly.
Three related failure scenarios that have been observed are:
A directory exists at the libvirt socket path (/var/run/libvirt/libvirt-sock-ro), causing the libvirt pod to enter CrashLoopBackOff.
A directory exists in place of the rendered libvirtd.conf file, causing the libvirt pod to enter Init:CrashLoopBackOff.
After the libvirt issue is resolved,
nova-computecan lose the /var/lib/nova/compute_id file, causing the nova-compute pod to enter CrashLoopBackOff.
Procedure
There are two separate libvirt failures plus one knock-on
nova-computefailure. They look similar but the fixes differ, identify the problem first using the following command:$ kubectl -n openstack get pods -o wide | grep -E 'libvirt|nova-compute'
Use the pod status to determine which problem applies:
Pod status |
Problem |
Where the damage is |
|---|---|---|
|
Problem 1 — socket path |
On the host: |
|
Problem 2 — config path |
In the pod’s |
|
Problem 3 — lost identity |
On the host: |
Note
The difference between Problem 1 and Problem 2 is CrashLoopBackOff
versus Init:CrashLoopBackOff.
Problem 1 — a directory at the libvirt socket path
To identify, on the affected node, as root use the following command:
$ stat -c '%F' /var/run/libvirt/libvirt-sock-ro # "directory" => affected
The libvirtd log on the node confirms the details:
$ grep virNetSocketNewListenUNIX /var/log/libvirt/libvirtd.log | tail -3
error : virNetSocketNewListenUNIX:514 : Failed to bind socket to
'/run/libvirt/libvirt-sock-ro': Address already in use
Note
The above message is not accurate. Nothing holds the address — bind() fails
because the path is a directory rather than a socket.
Procedural Changes:
To fix, on the affected node as root use the following command:
$ rmdir /var/run/libvirt/libvirt-sock-ro
Use rmdir rather than rm -rf. rmdir removes only empty directories,
it provides an additional safeguard by failing if the target path contains
unexpected files or content, helping to prevent accidental data loss.
Note
Deleting the pod is not required. After the directory is removed, the libvirt pod recovers automatically on its next retry, typically within 90 seconds.
Problem 2 — a directory at the rendered libvirtd.conf
To identify:
$ POD=$(kubectl -n openstack get pods --no-headers \
| awk '/^libvirt-/ && $3 ~ /Init:/ {print $1; exit}')
$ kubectl -n openstack logs "$POD" -c init-dynamic-options --tail=10
/tmp/init-dynamic-options.sh: line 12: /tmp/pod-shared/libvirtd.conf: Is a directory
Procedural Changes: To fix, two objects must be removed, not one.
Clearing only the first is not enough. First get the pod UID, where kubectl
works:
$ PODUID=$(kubectl -n openstack get pod "$POD" -o jsonpath='{.metadata.uid}')
Then on the affected node, as root run the following:
# 1. the directory in the pod's emptyDir
$ rmdir /var/lib/kubelet/pods/$PODUID/volumes/kubernetes.io~empty-dir/pod-shared/libvirtd.conf
# 2. the stale subPath bind target
$ ls -la /var/lib/kubelet/pods/$PODUID/volume-subpaths/pod-shared/libvirt/
$ rmdir /var/lib/kubelet/pods/$PODUID/volume-subpaths/pod-shared/libvirt/<N>
Note
In step 2, first list the directory contents and identify the entry to
remove rather than assuming the directory number. The number represents
a volumeMount ordinal, which can vary between builds.
Restart the pod:
$ kubectl -n openstack delete pod "$POD"
A reboot does not fix this one. The emptyDir is only destroyed when the pod
is removed from the node, so restarts and host reboots preserve the bad
directory.
Problem 3 — nova-compute will not start after libvirt is fixed
If libvirt was down long enough for the nova-compute liveness probe to fail,
the container restarts and looses its node identity file. This does not clear
when libvirt recovers. The node reports unlocked / enabled /
available while running no workloads.
To identify:
$ kubectl -n openstack logs <nova-compute-pod> -c nova-compute --tail=200 \
| grep 'No local node identity found'
Note
Key on that string. The first ERROR in the traceback is a masking error
(AttributeError: 'NoneType' object has no attribute 'stop') and will
misdiagnose every instance of this.
On the node, as root:
$ ls -l /var/lib/nova/compute_id # absent => affected
Procedural Changes: Run the following command:
$ kubectl -n openstack delete pod <nova-compute-pod>
Fix libvirt first. The nova-compute init container waits for a libvirt pod
on the same node. If libvirt is still broken, the replacement pod simply remains
in Init and nothing is gained.
Note
If the replacement pod hangs in Pending with references non-existent
secret key: host-interface, that is a separate defect and must be cleared
first.
Recommended order per host: Fix Problem 1 or 2 first, confirm libvirt reaches
1/1 Running, then check nova-compute and fix Problem 3 if required.
OpenStack Endpoint CA Certificate Required in Platform Truststore¶
When OpenStack is deployed with TLS endpoints, the OpenStack endpoint CA certificate must be trusted by the platform (imported into the platform truststore) so that the NFV-VIM can successfully establish TLS connections when making calls to OpenStack services.
If the OpenStack endpoint CA certificate is not present in the platform truststore, VIM TLS verification fails and VIM-to-OpenStack API calls fail.
Procedural Changes:
Install the CA Certificate¶
Replace <path-to-openstack-ca-cert> with the location of the OpenStack
endpoint CA certificate on the controller.
Procedure
Install the CA certificate into the system trust bundle. Skip this step if a CA with the same subject is already installed:
$ system ca-certificate-install <path-to-openstack-ca-cert>
Note
Installing a CA certificate with the same subject replaces the existing entry. Re-running this command does not create duplicate truststore entries.
Restart the VIM service to apply the newly trusted CA certificate:
$ sudo sm-restart-safe service vim
Serial Migrate Strategy May Fail on the Last Worker of a Two-Worker System¶
On systems with only two worker nodes, a serial update strategy that uses the migrate instance action may fail when processing the final worker. If the second worker begins its update sequence before the first worker has fully returned to service, no available worker remains to host the migrated instances. As a result, the update strategy stops before completion.
Procedural Changes: Delete the failed strategy, ensure the first worker has fully returned to an available state, and then recreate and reapply the strategy.
Cinder Volume Service Runs a Single Replica¶
StarlingX OpenStack runs a single cinder-volume replica on all deployment
configurations, including AIO-DX and Standard systems with more than one
controller. Cinder supports multiple concurrent volume services only in an
Active-Active configuration, which requires a Cinder cluster name, a shared
distributed lock manager, and per-node service identities. StarlingX OpenStack does not
configure these, so a single replica is the only topology currently supported.
As a result, the volume service is not redundant: if the controller hosting the
cinder-volume pod becomes unavailable, Kubernetes reschedules the pod onto
the remaining controller, and volume and snapshot operations are unavailable
until it restarts. Volumes already attached to instances are unaffected, as is
guest I/O (data plane).
The cinder-api, cinder-scheduler, and cinder-backup services
continue to run one replica per provisioned controller.
Note
Support for active-passive and active-active deployment models are
planned for a future StarlingX OpenStack release.
Procedural Changes: N/A.
Manual OpenStack Application Re-apply During Controller Lock Can Stall MariaDB Recovery¶
On AIO-DX; AIO-DX Plus and Standard systems, manually re-applying the stx-openstack
application while a controller is locked for configuration changes is not
supported. With a controller in the locked state, the system runs in a
reduced-redundancy state. Running system application-apply stx-openstack can
then scale the MariaDB StatefulSet to a single replica and leave the surviving
database pod unable to form the cluster, which stalls the apply. The apply may
become stuck in the applying state (often around 10%), transition to
apply-failed, and leave the mariadb-server-0 pod continuously restarting
(0/1 Ready).
Warning
Do not manually re-apply stx-openstack while a controller is locked for
configuration changes. If a re-apply was triggered by accident while the
controller was locked and you then observe that the mariadb-server-0
pod is stuck (0/1), follow the recovery procedure below.
Procedural Changes: Recover a stalled MariaDB cluster after a manual
re-apply of stx-openstack during a controller lock.
Symptoms:
system application-show stx-openstackshows statusapplying(often stuck around 10%) orapply-failed. Operators may notice either, sometimes only after a long delay. The recovery is the same for both.The
mariadb-serverStatefulSet is not fully ready, and themariadb-server-0pod is 0/1 Running and restarting; its log repeatstoo old to make a decision for node mariadb-server-1.
Procedure
Confirm the stuck state.
$ kubectl -n openstack get sts mariadb-server $ kubectl -n openstack get pods -l application=mariadb -o wide $ kubectl -n openstack logs mariadb-server-0 -c mariadb --tail=20
Confirm the StatefulSet shows
READY 0/1,mariadb-server-0is 0/1 Running, and its log repeatstoo old to make a decision for node mariadb-server-1.Recover MariaDB to a single healthy node by deleting the stuck pod.
$ kubectl -n openstack delete pod mariadb-server-0
Wait for the application to reach
applied.$ system application-show stx-openstack
Wait until the status is
applied.Unlock the controller you locked.
$ system host-unlock controller-<n>
Wait for
unlocked/enabled/availableand the Kubernetes nodeReady.Let the automatic re-apply scale MariaDB back to two nodes. Do not manually re-apply while reconciliation is in progress.
$ kubectl -n openstack get sts mariadb-server $ kubectl -n openstack get hr -n openstack
If any HelmRelease is not yet
READY=True, reconciliation is still in progress — wait. Only if all HelmReleases areREADY=True, the StatefulSet is still not at 2, and the app is in theappliedstate, runsystem application-apply stx-openstack.Note
system application-showcan lag; treatkubectl get hr -n openstackand the StatefulSet as authoritative.If the joining pod
mariadb-server-1stays 0/1 for several minutes (its log repeatsUpdating grastate configmap with no state transfer), delete it.$ kubectl -n openstack delete pod mariadb-server-1
Note
The StatefulSet may then roll
mariadb-server-0onto the current template; themariadb-server-0restart afterwards is normal.Verify the cluster and data.
$ kubectl -n openstack get sts mariadb-server # expect READY 2 $ DBPW=$(kubectl -n openstack get secret mariadb-dbadmin-password -o jsonpath='{.data.MYSQL_DBADMIN_PASSWORD}' | base64 -d) $ kubectl -n openstack exec mariadb-server-0 -c mariadb -- \ bash -lc "mariadb -uroot -p'$DBPW' -e \"SHOW STATUS WHERE Variable_name IN ('wsrep_cluster_size','wsrep_cluster_status','wsrep_local_state_comment','wsrep_incoming_addresses');\"" $ kubectl -n openstack exec mariadb-server-1 -c mariadb -- \ bash -lc "mariadb -uroot -p'$DBPW' -e \"SHOW STATUS WHERE Variable_name IN ('wsrep_cluster_size','wsrep_cluster_status','wsrep_local_state_comment','wsrep_incoming_addresses');\""Expect both nodes
Primary,Synced. The expectedwsrep_cluster_sizeis 2 on AIO-DX; 3 on AIO-DX Plus and Standard (two MariaDB servers plus thegarbdarbitrator).AIO-DX Plus and Standard only: re-attach the
garbdarbitrator if needed. On these configurations the healthy cluster size is 3. Ifwsrep_cluster_sizereads 2 after recovery while both servers are healthy, the arbitrator has not rejoined. Restart it.$ kubectl -n openstack delete pod -l application=garbd
The arbitrator rejoins and
wsrep_cluster_sizeshould return to 3. Skip this step on AIO-DX, which does not rungarbd.
Cinder Storage Service Catalog¶
Cinder Storage Service Catalog (SSC) does not recover from transient DNS failures when configured with NetApp volume backends.
A transient DNS outage can prevent SSC updates for NetApp backends, causing Cinder to stop detecting newly added storage pools on the NetApp system. As a result, the output of Cinder get-pools may become stale and continue to display only the NetApp pools that were registered before the DNS failure occurred.
Procedural Changes:
Following recovery from a transient DNS outage, restart the OpenStack cinder-volume pods to resume SSC updates:
$ kubectl -n openstack rollout restart deployment/cinder-volume
To help prevent future disruptions, you can pin the DNS host record for the NetApp SAN FQDN using the following command:
$ system service-parameter-add dns host-record netapp-san=<FQDN>,<IP>
This ensures that Cinder can continue to resolve the NetApp SAN endpoint even if external DNS resolution becomes temporarily unavailable.
Heat Service Depends on External DNS with FQDN Deployments¶
When OpenStack is deployed using an FQDN configuration, the Heat service relies entirely on external DNS resolution to process requests and communicate with other OpenStack services. As a result, an external DNS outage renders the Heat service unavailable. Access to the Heat API, including CLI operations, provisioning workflows, and automation processes, will fail until DNS service is restored.
Procedural Changes: During a DNS outage, manage OpenStack services directly through the OpenStack CLI rather than through Heat templates or Heat-based orchestration workflows.
Manual Removal of OpenStack Compute Service After Label Deletion¶
Any StarlingX OpenStack Release 13.0 deployment where a host ceases to function
as an OpenStack compute node after the openstack-compute-node label is removed.
A common scenario is the conversion of an OpenStack compute host into a
Kubernetes-only worker in a Hybrid Cluster.
After the openstack-compute-node label is removed and the host is unlocked,
OpenStack compute services are no longer running on the host.
However, Nova continues to retain the host’s compute service and hypervisor
records. The compute service remains in a disabled/down state, and the associated
hypervisor remains down. These records are neither removed automatically nor
aged out.
Although this condition does not affect scheduling on remaining compute hosts, because Nova does not schedule instances to disabled or down compute services, the Nova inventory no longer reflects the actual platform state. As a result, the host continues to appear in the output of commands such as OpenStack compute service list and OpenStack hypervisor list, as well as in Horizon.
$ openstack compute service list
+--------------------------------------+--------------+-----------+------+----------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+--------------------------------------+--------------+-----------+------+----------+-------+----------------------------+
| 6fc119e6-f333-4894-b1d8-33160094aa51 | nova-compute | compute-0 | nova | enabled | up | 2026-09-23T20:14:06.000000 |
| 391b1ea3-f93a-4760-b411-fab6e28e4085 | nova-compute | compute-2 | nova | enabled | up | 2026-09-23T20:14:08.000000 |
| 47e42935-826d-4131-a6c7-db77ec3ece20 | nova-compute | compute-1 | nova | disabled | down | 2026-09-23T19:39:36.000000 |
+--------------------------------------+--------------+-----------+------+----------+-------+----------------------------+
$ openstack hypervisor list
+--------------------------------------+---------------------+-----------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | State |
+--------------------------------------+---------------------+-----------------+-------+
| efacd07b-feeb-4407-b65c-176c13224880 | compute-0 | QEMU | up |
| 5127bb34-4ce0-4b08-ba92-5f9bb1e75429 | compute-1 | QEMU | down |
| e55d2a08-6449-4513-a60b-d14f8a41278e | compute-2 | QEMU | up |
+--------------------------------------+---------------------+-----------------+-------+
Procedural Changes: Once the label removal is complete, delete the compute service of the host manually. Deleting the compute service also removes the hypervisor record of the host.
# 1. Complete the label removal as documented (lock the host, remove the label, unlock the host).
# Wait until the host is unlocked / enabled / available and stx-openstack is applied.
$ system host-list
$ system application-show stx-openstack
# 2. On the active controller, load the OpenStack credentials.
$ source /var/opt/openstack/admin-openrc
# 3. Find the compute service of the host. Its State must be "down".
$ openstack compute service list --service nova-compute --host <hostname>
# 4. Delete it, using the ID returned by the previous command.
$ openstack compute service delete <service-id>
# 5. Verify that the host is no longer listed by either command.
$ openstack compute service list --service nova-compute
$ openstack hypervisor list
Note
Delete a compute service only when its State is down. A state of up indicates that the nova-compute service is still running on the host, either because the openstack-compute-node label is still assigned or because application reapply operations have not yet completed. Deleting the compute service while nova-compute is still running can leave orphaned hypervisor records.
Nova rejects the deletion request with an HTTP 409 error if the host still has instances (“Unable to delete compute service that is hosting instances. Migrate or delete the instances first.”) or contains in-progress migrations (“Unable to delete compute service that in-progress migrations. Complete the migrations or delete the instances first.”). A standard host lock operation typically migrates instances from the host before the lock completes. If the deletion is rejected, use openstack server list –all-projects –host <hostname> to identify any remaining instances, then migrate or delete them as appropriate, ensuring that any pending resize operations are either confirmed or reverted before retrying the deletion.
Deleting the compute service also removes the host from any associated host aggregates. If the host is later restored as an OpenStack compute node, re-add it to the required host aggregates.
After cleanup, the host can be returned to service as an OpenStack compute node normally. When the openstack-compute-node label is reassigned and the host is unlocked, Nova creates a new compute service and the associated hypervisor returns to the up state.
Aborted stx-openstack Application Update Can Leave MariaDB Degraded Without Alarms on AIO-DX Systems¶
On an AIO-DX system, aborting a stx-openstack application update, such as
a OpenStack Release 12.0 to OpenStack Release 13.0 that is manually
aborted or automatically aborted due to a release failure, can leave the
MariaDB cluster in a degraded state. The mariadb-server StatefulSet may be
reduced to a single replica (replicas=1), with mariadb-server-1 missing and
mariadb-server-0 stuck in a 0/1 (Running but Not Ready) state while repeatedly
failing its startup probe.
Despite the database outage, system application-show stx-openstack reports
status=applied and displays "Application recover to version [old] completed",
while fm alarm-list reports no active alarms. As a result, the control-plane
database is unavailable even though the platform indicates a successful recovery
and no fault condition.
The mariadb-server-0 pod log repeatedly reports the following message: “The data we have from the cluster is too old to make a decision for node mariadb-server-1”.
This condition occurs because the cluster-state ConfigMap still contains a stale reference to mariadb-server-1, causing MariaDB to wait for a non-existent replica and preventing the remaining node from becoming ready.
Procedural Changes:
Use the exact underscore form of the keys shown below:
Confirm the stuck state.
$ kubectl get sts -n openstack mariadb-server # replicas=1 $ kubectl get pods -n openstack | grep mariadb-server # server-0 0/1, server-1 absent
Back up the cluster-state ConfigMap.
$ kubectl get configmap -n openstack osh-openstack-mariadb-mariadb-state -o yaml > cm.backup.yaml
Remove the stale phantom server-1 entries.
$ kubectl patch configmap -n openstack osh-openstack-mariadb-mariadb-state --type merge \ p '{"data":{"safe_to_bootstrap.mariadb-server-1":null,"sample_time.mariadb-server-1":null,"seqno.mariadb-server-1":null,"uuid.mariadb-server-1":null,"version.mariadb-server-1":null}}'Restart server-0 -> bootstraps to 1/1 with data intact
$ kubectl delete pod -n openstack mariadb-server-0
Restore HA: scale back to 2 -> server-1 rejoins and re-syncs.
$ kubectl scale sts -n openstack mariadb-server --replicas=2
Verify: 2/2 ready, wsrep_cluster_size=2, no mariadb alarms, data present.
Note
After step 4, mariadb-server-0 successfully bootstraps and reaches a
1/1 Ready state with its data intact, allowing dependent pods to recover.
After step 5, mariadb-server-1 rejoins the cluster, restoring the
Galera cluster to its expected two-node configuration.
Manual Hiera Workaround Required for OVS-DPDK Deployments Before Upgrading to StarlingX Release 13.0¶
OVS-DPDK deployments require a manual Hiera workaround on all controllers before upgrading from StarlingX Release 12.0 to StarlingX Release 13.0.
When upgrading StarlingX Release 12.0 deployments using vswitch_type = ovs-dpdk,
DPDK parameters (host_core_list, pmd_core_list, socket_mem) may not be preserved
correctly during upgrade, causing OVS-DPDK networking to fail post-upgrade.
Procedural Changes: Before performing the upgrade, execute the following steps on all controllers:
Procedure
Create the manual overrides directory for both controllers.
$ sudo mkdir /etc/puppet/manual_overrides/
Identify the current DPDK Hiera values.
$ sudo find /etc -type f -name "*.yaml" -print0 2>/dev/null | sudo xargs -0 grep -nH "vswitch::dpdk" /etc/puppet/cache/hieradata/controller-0.yaml:462:vswitch::dpdk::host_core_list: '0,2' /etc/puppet/cache/hieradata/controller-0.yaml:463:vswitch::dpdk::pmd_core_list: '2' /etc/puppet/cache/hieradata/controller-0.yaml:464:vswitch::dpdk::socket_mem: '1024'
Create the override file with the correct values, from step 2, for your deployment.
# /etc/puppet/manual_overrides/override.yaml vswitch::dpdk::host_core_list: '0,2' vswitch::dpdk::pmd_core_list: '2' vswitch::dpdk::socket_mem: '1024'
Update /etc/puppet/hiera.yaml to include the manual overrides as the first item in the hierarchy.
Warning
The YAML format is critical — indentation must be exact. Do not modify what is already present in the file.
version: 5 hierarchy: - name: "Manual Overrides" datadir: /etc/puppet/manual_overrides data_hash: yaml_data path: "override.yaml" - name: "yaml" datadir: /tmp/puppet/hieradata data_hash: yaml_data paths: - runtime.yaml - host.yaml - secure_system.yaml
Enabling DHCP on a Subnet with no Free IPs Gives False Success¶
Enabling DHCP on a subnet with an exhausted allocation pool succeeds at the API level but DHCP will not function. When an operator enables DHCP on an existing subnet (enable_dhcp False→True) whose allocation pool has no free IP addresses, the Neutron API returns success (HTTP 200). However, the DHCP agent fails asynchronously to create its required port due to IP address exhaustion. This results in:
No network:dhcp port being created on the subnet.
VMs launched on that subnet failing to obtain IP addresses via DHCP/cloud-init.
The failure is visible in the DHCP agent logs (IpAddressGenerationFailure), not to the API caller.
Procedural Changes:
Before enabling DHCP on an existing subnet, verify that the allocation pool has at least one free IP:
$ openstack ip availability show <network-id>
If used_ips equals total_ips for the relevant subnet, either:
Expand the allocation pool.
openstack subnet set --allocation-pool start=<new-start>,end=<new-end> <subnet-id>Or, delete an unused static port to free an IP address.
Then enable DHCP.
$ openstack subnet set --dhcp <subnet-id>
Restore Complete Operation Can Fail on Large-Scale OpenStack Systems¶
On a large system (many hosts) running StarlingX OpenStack, the final restore step can fail with the message:
Unable to perform restore modify state.
On large-scale OpenStack systems, the restore-complete step can fail when the OpenStack application keeps the system management service heavily loaded. The restore operation itself succeeds, but the completion phase cannot finish while the service is busy.
Impact
The backup-and-restore procedure does not complete on its own on a large system.
The system appears stuck at the final step until the workaround is applied.
Smaller systems are not expected to encounter this.
Procedural Changes
Procedure
Turn off the automatic application re-evaluation so the management service is released.
$ system service-parameter-modify platform config evaluate_apps_reapply=disabled $ system service-parameter-apply platform
Wait for 30 seconds.
If the system has been stuck for a while, restart the management so it starts again.
$ sudo sm-restart-safe service sysinv-conductor $ sudo sm-query service sysinv-conductor
Confirm no controller switchover was initiated.
Rerun the restore completion step.
$ system restore-complete $ system restore-show
Result: “Restore procedure is not in progress.”
Re-enable the re-evaluation.
$ system service-parameter-modify platform config evaluate\_apps\_reapply=enabled system service-parameter-apply platform
Re-apply the applications in the following order: storage first, then the remaining applications.
$ system application-apply rook-ceph $ system application-apply stx-openstack
glance-api Pods Stay in CrashLoopBackOff After Platform Restore¶
On systems that use Ceph (rook-ceph) storage, the stx-openstack application
is reapplied when you run system restore-complete during a platform
restore. The glance-api pods start before their Ceph credentials are ready.
They stay in CrashLoopBackOff with rados.PermissionDeniedError: [errno 13] in
the pod log and do not recover on their own. Pods that wait for Glance, such
as nova-compute and pci-irq-affinity-agent, stay in Init state,
and stx-openstack can show apply-failed. The platform restore itself
and your OpenStack backup data are not affected.
Procedural Changes: Delete the glance-api pods so that they are
recreated. Run these commands on the active controller after
system restore-complete:
Procedure
Check the state of the
glance-apipods.$ export KUBECONFIG=/etc/kubernetes/admin.conf $ kubectl -n openstack get pods | grep glance-api
If the pods are in
CrashLoopBackOff, delete them. Only the stuck pods are replaced. No data is removed.$ kubectl -n openstack delete pod -l application=glance,component=api
Wait until the new
glance-apipods are Running and 1/1 ready.$ kubectl -n openstack get pods -w | grep glance-api
Verify that the pods that were waiting for Glance have left
Initstate.$ kubectl -n openstack get pods | grep -E "glance-api|nova-compute|pci-irq-affinity-agent"
Continue with the OpenStack application restore. It is expected that
stx-openstackstill showsapply-faileduntil the application restore is done.
Note
If you reapply applications manually, reapply rook-ceph before
stx-openstack. Deleting the glance-api pods helps only after the
Ceph credentials for Glance exist.
/var/opt/openstack Not Included in Backup and Restore¶
The StarlingX OpenStack platform and application backup and restore does
not save or restore the contents of /var/opt/openstack. After a restore,
the stx-openstack application is reapplied and recreates the directory with
only its default files: admin-openrc, clear-aliases.sh,
clients-wrapper.sh, local_openstackrc, and setup-aliases.sh. Any
file you added to the directory is lost, for example tenant OpenStack RC files,
certificates, custom scripts, and the images/ directory. OpenStack resources
contained in the application backup (VMs, volumes, networks, images) are not
affected.
Procedural Changes: Back up the directory yourself and restore it after the restore completes.
Backup¶
On the active controller, after the platform and application backup has completed:
Procedure
Create the archive and restrict access to it. The archive contains OpenStack credentials and certificates.
$ sudo tar czf /home/sysadmin/var-opt-openstack-$(hostname)-$(date -u +%Y%m%d%H%M%SZ).tgz -C /var/opt openstack $ sudo chown sysadmin: /home/sysadmin/var-opt-openstack-*.tgz $ sudo chmod 600 /home/sysadmin/var-opt-openstack-*.tgz
Verify the archive is readable and lists your files.
$ ls -lh /home/sysadmin/var-opt-openstack-*.tgz $ tar tzf /home/sysadmin/var-opt-openstack-<host>-<timestamp>.tgz
Copy the archive off the system, together with the platform and application backup files. Anything left on the system is lost if the system is wiped and reinstalled.
Restore
After the platform restore is complete and the stx-openstack application has
been restored and reapplied:
Procedure
Confirm that only the default files are present.
$ ls -l /var/opt/openstack
Copy the archive back to
/home/sysadminon the active controller and extract it over the directory. Usesudoso that the original owners and permissions are restored.$ sudo tar xzf /home/sysadmin/var-opt-openstack-<host>-<timestamp>.tgz -C /var/opt
Verify that your files are back with their original owner, group, and permissions, and that the OpenStack CLI still works.
$ ls -l /var/opt/openstack $ source /var/opt/openstack/admin-openrc $ openstack server list --all-projects
Note
Extracting the archive also replaces the five default files with the copies
saved at backup time. If you changed the OpenStack admin password or
endpoints after taking the backup, update admin-openrc afterwards.
Supported Deployment Configurations¶
StarlingX OpenStack Release 13.0 is supported with the following deployment configurations:
Standalone Cloud Deployments
All-In-One Simplex
All-In-One Duplex
Standard
Distributed Cloud Deployments
Subclouds
All-In-One Simplex
All-In-One Duplex
Standard
Guest OS on VMs¶
This section lists the validated Guest OSs on VMs in StarlingX OpenStack Release 13.0.
Note
Users may need to adjust the kubelet filesystem size on the controllers to successfully create bootable volumes. For huge images intended for bootable volumes, increasing the kubelet filesystem size may be necessary. The required kubelet filesystem size depends on the size of the image being used.
Booting a server using the huge image does not require further adjustments on kubelet.
system host-fs-modify controller-0/1 kubelet=<size>
Note
If using hw:cpu_realtime=yes for a VM running a real-time guest
operating system:
Enable a low-power (halt-based) idle mode in the guest so idle vCPUs release the physical CPU instead of polling. See the guest operating system documentation for the required kernel option.
Configure the stalld daemon on all compute hosts that might run the real-time VM. For more information see Configure Stall Daemon.
This prevents busy-looping real-time tasks in the guest from blocking critical host kernel tasks, which could otherwise trigger a watchdog reset.
Guest OS Compatibility Matrix¶
The following table lists the Compatibility Matrix for Guest OS, it follows a lifecycle from image booting and follows the sequence below while trying to connect to a VM during each step:
Stop/Start
Pause/Unpause
Suspend/Resume
Shelve/Unshelve
Reboot
Each GuestOS is validated 3 times for each boot type:
Boot from image
Boot from volume
Boot from snapshot
OS Image |
BIOS Boot |
UEFI Boot |
UEFI Secure Boot |
|---|---|---|---|
AlmaLinux-9 |
Yes |
Yes |
Yes |
AlmaLinux-10 |
Yes |
Yes |
Yes |
AlpineLinux |
Not validated |
Yes |
Yes |
Arch Linux (2026.02.01) |
Yes |
Yes |
Yes |
CentOS-9 (BIOS/UEFI) |
Yes |
Yes |
Yes |
CentOS-10 (BIOS/UEFI) |
Yes |
Yes |
Yes |
CirrOS |
Yes |
Yes |
Yes |
Debian-11 (Bullseye) |
Yes |
Yes |
Yes |
Debian-12 |
Yes |
Yes |
Yes |
Debian-13 (Trixie) |
Yes |
Yes |
Yes |
Fedora CoreOS (Stable) |
Yes |
Yes |
Yes |
openSUSE Leap 15.3 |
Yes |
Yes |
Yes |
openSUSE Leap 16.0 |
Yes |
Yes |
Yes |
Oracle Linux 9 |
Yes |
Yes |
Yes |
Oracle Linux 10 |
Yes |
Yes |
Yes |
Photon OS 5.0 |
Yes |
Yes |
Not validated |
RHEL 8 |
Yes |
Yes |
Yes |
RHEL 9 |
Yes |
Yes |
Yes |
RHEL 10 |
Yes |
Yes |
Yes |
Rocky Linux |
Yes |
Yes |
Yes |
Ubuntu 20.04 (Focal) |
Yes |
Yes |
Yes |
Ubuntu 22.04 (Jammy) |
Yes |
Yes |
Yes |
Ubuntu 24.04 (Noble) |
Yes |
Yes |
Yes |
Ubuntu 26.04 (Resolute) |
Yes |
Yes |
Yes |
Elxr 26.04.01 |
No |
Yes |
Yes |
Windows 11 |
Not validated |
Yes |
Yes |
Windows 2016 |
Not validated |
Yes |
Yes |
Windows 2019 |
Not validated |
Yes |
Yes |
Windows 2022 |
Not validated |
Yes |
Yes |
Windows 2025 |
Not validated |
Yes |
Not validated |
Open Source Project Versions for OpenStack¶
This section documents key open source project versions for StarlingX OpenStack.
For more information on all StarlingX OpenStack Open Source Project Versions, see https://releases.openstack.org/gazpacho/index.html.
Open Source Packages |
Version |
|---|---|
Open vSwitch |
3.5.0 |
DPDK |
24.11.4 |
libvirt |
11.3.0 |
QEMU |
10.0.8 |
MariaDB |
11.4.8 |
RabbitMQ |
3.11.28 |
memcached |
1.5.5 |
Deprecation and Removal Notices¶
This section lists capabilities that are either deprecated (still functional, but planned to be removed in a future release), and removed in StarlingX OpenStack.
Deprecated in StarlingX OpenStack 13.0¶
N/A.
Removed in StarlingX OpenStack 13.0¶
Host-based Ceph for Storage Backends¶
Host-based Ceph support for storage backends of Cinder, Glance and Nova
is deprecated in StarlingX OpenStack Release 25.09 and is replaced by rook-based Ceph.
Note
Use one of the supported StarlingX OpenStack storage backends: Rook-based Ceph, NetApp (NFS / iSCSI / FC) and/or Extended Storage Backends (ESB).
Warning
Upgrades and migrations from StarlingX OpenStack Release 12.0 systems using Host-based Ceph to StarlingX OpenStack Release 13.0 are not supported. If a migration is required, it must be performed using external migration tools and application-level migration procedures.
VLAN Transparency (Networks)¶
VLAN transparency is no longer supported by the OVS mechanism driver.
Creating networks with the --transparent-vlan option has been deprecated
and removed. To carry VLAN-tagged traffic from guest instances, use the Neutron
trunk extension (port trunking). For more information, see Enable the
Trunk Extension for Neutron.
Hardware Updates¶
See:
Bug status¶
Fixed bugs¶
This release provides fixes for a number of defects. Refer to the StarlingX bug database to review the Release 13.0 Fixed Bugs.