StarlingX OpenStack 13.0

ISO image

The pre-built ISO (Debian) for StarlingX 13.0 is located at the StarlingX mirror repo:

https://mirror.starlingx.windriver.com/mirror/starlingx/release/13.0.0/debian/trixie/amd64/monolithic/outputs/iso/

OpenStack Helm Charts:

https://mirror.starlingx.windriver.com/mirror/starlingx/release/13.0.0/debian/trixie/amd64/openstack/outputs/helm-charts/

Source Code for StarlingX 13.0

The source code for StarlingX 13.0 is available on the r/stx.13.0 branch in the StarlingX repositories.

Deployment

To deploy StarlingX 13.0, see Consuming StarlingX.

For detailed installation instructions, see StarlingX 13.0 Installation Guides.

New Features / Enhancements / Limitations

This release introduces several new features designed to improve usability and efficiency, along with targeted enhancements to existing functionality. Performance, stability, and overall user experience have been enhanced. Some features may have known limitations or constraints in this release; these are documented to help set expectations and guide usage.

Update from StarlingX OpenStack Epoxy to StarlingX OpenStack Gazpacho (SLURP)

StarlingX OpenStack Release 13.0 has been updated from OpenStack Epoxy to OpenStack Gazpacho (Skip-Level Upgrade Release Process - SLURP).

Gazpacho is designated as a SLURP release, which introduces a new upgrade version in OpenStack.

Gazpacho is the OpenStack release codename for StarlingX OpenStack Release 2026.1.

See: https://releases.openstack.org/gazpacho/index.html.

Upversion of OpenStack Services to Gazpacho (2026.1)

StarlingX OpenStack Release 13.0 updated all core OpenStack services from OpenStack 2025.1 (Epoxy) to OpenStack 2026.1 (Gazpacho). These include:

  • OpenStack Gazpacho for new installations of StarlingX OpenStack 13.0, delivered as part of the SLURP (Skip Level Upgrade Release Process) release.

  • Gazpacho on K8s v1.35 (default).

  • Functionality and services provided in the Gazpacho initial release of April 1, 2026.

This update is delivered as an in-service application update, with the following updates:

  • OpenStack configuration preserved across the application.

  • Minimal elapsed time for the application.

  • Minimal OpenStack control plane service disruption.

  • Hosted virtual machines remain in service throughout the update.

Note

Nova services run in Eventlet concurrency mode by default, in accordance with the StarlingX OpenStack Concurrency Policy. This is the default and validated configuration, and it is recommended for all deployments. Although the concurrency mode can be changed through Helm overrides, this is not recommended.

See: https://releases.openstack.org/gazpacho/index.html.

See: The Deprecation and Removal Notices section below.

Debian 13 (Trixie) OS Alignment for StarlingX OpenStack in StarlingX 13.0

StarlingX OpenStack Release 13.0 aligns with the StarlingX Release 13.0 base operating system Debian 13 (Trixie), requiring updates across OpenStack maintained host packages and containerized services.

This feature delivers updates required to ensure OpenStack components are aligned with the Trixie-based platform on StarlingX Release 13.10.

OpenStack Release 13.10 is supported on StarlingX Release 13.0 and includes the following updates:

Kubernetes versions supported in StarlingX OpenStack Release 13.0:

  • 1.32.2

  • 1.32.13

  • 1.33.0

  • 1.34.1

  • 1.34.8

  • 1.35.2

Supported default version in StarlingX OpenStack 13.0 for new installs is:

  • 1.35.2

See: OpenStack Approved Commercial Hardware

Extended Storage Backend (ESB)

This feature introduces ESB in StarlingX OpenStack, describes its capabilities and supported storage protocols, and outlines how to determine whether your type of deployment can be supported by ESB.

Earlier, StarlingX OpenStack supported only a predefined set of platform-managed storage backends (Ceph, NetApp NFS, NetApp iSCSI, and NetApp FC), which were automatically detected and configured. ESB extends this capability to additional storage backends by enabling the configuration of any upstream OpenStack Cinder driver included with the platform as a custom backend.

An extended backend is a storage backend that you define yourself. You declare it once in the Cinder Helm chart overrides, and StarlingX OpenStack handles the pod-level settings for the specified protocol, injects credentials from a Kubernetes Secret, and passes your driver configuration to Cinder unchanged. StarlingX OpenStack does not validate the driver configuration, so you are responsible for ensuring that the configuration is valid.

See:

OpenStack Ceilometer Support for Resource Usage and Telemetry Collection

This feature enables resource-owner Telemetry in StarlingX OpenStack through the deployment of OpenStack Ceilometer and its native Prometheus exporter. Ceilometer polling agents running on the control and compute planes collect resource and workload metrics from OpenStack services and hypervisors, then expose the data through a standard Prometheus-compatible endpoint for integration with Analytics, Prometheus, and Aodh.

The collected Telemetry provides visibility into tenant-level resource utilization and virtual machine workload behavior, including per-tenant storage utilization for volumes, images, and snapshots, as well as per-VM CPU utilization, memory usage, disk I/O, network I/O, and other performance metrics. The Ceilometer central agent gathers metrics from OpenStack services such as Cinder, Glance, and Neutron, while the compute agent collects virtualization metrics locally from libvirt on each compute node.

This capability enables operators and resource owners to monitor infrastructure and workload usage, support capacity planning, and improve operational observability across the OpenStack cloud.

Configure Ceilometer for Telemetry Data Collection

Ceilometer is the data collection component of the StarlingX OpenStack Telemetry project. Its role is to efficiently poll metering data from StarlingX OpenStack services, collect event data by monitoring notifications on the message bus, and publish the normalized results to various targets for storage, analysis, and alarming.

See:

Ceilometer Service Monitor for Prometheus

StarlingX OpenStack exposes two integration interfaces for external observability platforms:

  • Metrics endpoints exposed by the Ceilometer Central Exporter, Ceilometer Compute Exporter, and OpenStack Exporter, which can be scraped directly.

  • An HTTP endpoint to receive alarm actions generated by StarlingX OpenStack.

An external platform can use either or both interfaces independently.

See: Integrate with External Systems

OpenStack Aodh (Alarming)

StarlingX OpenStack uses Aodh to create and manage alarms for OpenStack services and resources. Aodh alarms are OpenStack Telemetry resources and operate independently of the platform fault management system.

See: Manage Alarms Using Aodh

OpenStack Metrics Export for Prometheus

The StarlingX OpenStack application provides observability capabilities that give you visibility into the health, performance, and operational state of your cloud. With components such as the Ceilometer exporters, OpenStack Exporter, Aodh services, and the optional prometheus platform application, you can collect and store metrics, process service events, evaluate alarms, and expose Telemetry data to monitoring and Analytics systems.

This feature enhances infrastructure-level observability of the OpenStack cloud by deploying a Prometheus OpenStack exporter within OpenStack. The exporter queries OpenStack services, including Nova, Cinder, Neutron, Glance, Keystone, Heat, and Placement, and exposes infrastructure metrics in Prometheus format for consumption by external analytics and monitoring systems.

The exported metrics provide operators with visibility into the health, utilization, and capacity of the OpenStack infrastructure. Key metrics include service and agent availability by host, hypervisor resource capacity and utilization (vCPUs, memory, and storage), tenant quota consumption and limits, network resource inventory (networks, subnets, ports, routers, and floating IPs), Placement allocation ratios, and Heat stack status. This capability enables proactive monitoring, capacity planning, and faster identification of infrastructure issues across the OpenStack environment.

See:

OVS-DPDK Support for Bonded (AE) Interfaces

StarlingX OpenStack now supports bonded (AE) data interfaces with OVS-DPDK for a validated configuration consisting of a two-member, data-class AE interface operating in active-standby mode on hosts configured with OVS-DPDK. The bonded interface can be used with a single data network of either VLAN or flat type.

See: Bond/AE and VLAN Interfaces

Configure Horizon Apache Worker Recycling

In deployments with sustained Horizon usage, the Apache WSGI worker processes that serve the dashboard may accumulate memory over time. This is expected behavior for long-running Python/Django processes. Starting with StarlingX OpenStack Release 13.0, worker recycling is enabled by default to prevent unbounded memory growth.

See: Configure Horizon Apache Worker Recycling

Dex SSO Setup and Verification

This enhancement provides details for configuring and validating Dex-based Single Sign-On (SSO) integration for Horizon and Keystone in the OpenStack environment. Users are guided through the complete deployment and verification workflow, enabling validation of each component after the OpenStack application is applied.

This applies to all supported OpenStack deployment models, including AIO-SX, AIO-DX, Standard, and Distributed Cloud subcloud deployments. Common configuration and verification steps are consolidated into a single workflow, with deployment-specific guidance provided only where required. For Distributed Cloud deployments, details for additional RedirectURI registration performed through an Ansible playbook are provided.

See: Dex SSO Setup and Verification

Known Limitations and Procedural Changes for StarlingX 13.0 OpenStack

Neutron Helm Chart Apply Fails with Overrides Using the Yaml List Syntax

The Neutron chart supported both YAML list syntax and comma-separated syntax from StarlingX OpenStack Release 12.0.

yaml list syntax

conf:
 neutron:
   DEFAULT:
     service_plugins:
     - router
     - network_segment_range
     - qos

comma-separated syntax

conf:
 neutron:
   DEFAULT:
     service_plugins: router,network_segment_range,qos

However, for StarlingX OpenStack Release 13.0, Neutron supports only the comma-separated syntax. The previously supported YAML list syntax is no longer accepted and can cause the Release 13.0 application apply operation, as well as the OpenStack application from Release 12.0 to Release 13.0, to fail.

Procedural Changes: Use the system helm-override-update CLI to replace any Neutron user overrides or apply new ones following the comma-separated syntax accepted by OpenStack 13.0. For application updates, update the overrides before triggering the system application-update, otherwise the update operation is expected to fail.

See: Enable the QoS Extension for Neutron

Nova Running-Deleted Reaper May Delete Active VMs and Attached Volumes

The Nova running_deleted_instance periodic task uses the default reap action. If the Nova database and a compute host become out of sync, such that the database marks an instance as deleted while the instance continues to run on the host, the task can destroy the running instance.

For boot-from-volume instances configured with delete-on-termination, the associated volume can also be deleted when the instance is reaped. As a result, a temporary inconsistency between the Nova database and compute host state, such as one that may occur during an application reapply operation, can lead to the permanent loss of a tenant VM and its backing volume.

Procedural Changes: Configure the Nova running_deleted_instance_action parameter to log in the Nova static overrides. This prevents the reaper task from deleting instances and volumes when a mismatch exists between the Nova database and compute host state. Instead, the discrepancy is logged, allowing an administrator to investigate and resolve the issue without risking data loss.

Apply the override and reapply stx-openstack to ensure that all nova-compute services are running with the updated setting.

conf:
  nova:
    DEFAULT:
      running_deleted_instance_action: log

Note

The poll interval must stay greater than 0 (default 60). The setting takes effect per nova-compute when its pod restarts.

Note

Setting the Nova running_deleted_instance_action parameter to log prevents the reaper task from deleting instances and volumes when a database and host state mismatch occurs. However, a separate orphan cleanup process runs whenever a nova-compute pod starts, during compute unlock, and application reapply. If a deleted-but-running instance is detected, this process can remove the instance’s libvirt domain on the next nova-compute restart. While the instance is removed, any associated volume is preserved because the cleanup process does not invoke the volume service.

This behavior is planned to be addressed in a future release. Until then, administrators should investigate and reconcile any deleted-but-running instances and avoid unnecessary nova-compute restarts while such instances exist.

Public HTTP Endpoint Unreachable Over IPv6 on IPv6-Only OAM Networks

In deployments that use an IPv6-only OAM network and expose public endpoints over HTTP (port 80), the public HTTP endpoint may be unreachable over IPv6. The network policy that opens port 80 for NGINX traffic is rendered for IPv4 only, resulting in the rule being applied to the IPv4 firewall chain but not the IPv6 chain.

As a result, clients on the OAM network, as well as stx-openstack pods running on a controller that does not host the OAM floating IP, cannot access public HTTP endpoints over IPv6. This may cause operations that depend on those endpoints to time out.

This limitation affects only HTTP (port 80) access on IPv6-only OAM deployments. Deployments that use HTTPS (port 443) or IPv4-based OAM networks are not affected.

Procedural Changes: Configure the nginx-ports-control Helm chart to use IPv6 and reapply stx-openstack.

Procedure

  1. Create an override file (e.g. /home/sysadmin/nginx-ports-control-override.yaml).

    global_network_policy:
      ingress:
        ipVersion: 6
    
  2. Apply the override and reapply the application:

    $ system helm-override-update stx-openstack nginx-ports-control openstack --values <override-file>
    
    $ system application-apply stx-openstack
    
  3. Verify that the controller does not hold the OAM floating IP and verify that the gnp-for-nginx-ports policy is rendered with ipVersion: 6 and that connections to the OAM floating IP on port 80 succeed over IPv6.

Note

This workaround configures the policy for IPv6 only. While appropriate for IPv6-only OAM deployments, it removes IPv4 support for port 80 and should not be used on dual-stack OAM networks, which require both IPv4 and IPv6 connectivity.

This update is planned to be addressed in a future release.

libvirt Pod Crash-Loop on a Wrong-Type Path and nova-compute Identity Loss

On StarlingX OpenStack Release 12.0, certain libvirt failures can persist after a controller reboot or service restart and may prevent OpenStack compute services from operating correctly.

Three related failure scenarios that have been observed are:

  • A directory exists at the libvirt socket path (/var/run/libvirt/libvirt-sock-ro), causing the libvirt pod to enter CrashLoopBackOff.

  • A directory exists in place of the rendered libvirtd.conf file, causing the libvirt pod to enter Init:CrashLoopBackOff.

  • After the libvirt issue is resolved, nova-compute can lose the /var/lib/nova/compute_id file, causing the nova-compute pod to enter CrashLoopBackOff.

Procedure

  1. There are two separate libvirt failures plus one knock-on nova-compute failure. They look similar but the fixes differ, identify the problem first using the following command:

    $ kubectl -n openstack get pods -o wide | grep -E 'libvirt|nova-compute'
    
  2. Use the pod status to determine which problem applies:

Pod status

Problem

Where the damage is

libvirt 0/1 CrashLoopBackOff

Problem 1 — socket path

On the host: /var/run/libvirt/libvirt-sock-ro

libvirt 0/1 Init:CrashLoopBackOff

Problem 2 — config path

In the pod’s emptyDir: pod-shared/libvirtd.conf

nova-compute 1/2 CrashLoopBackOff

Problem 3 — lost identity

On the host: /var/lib/nova/compute_id is missing

Note

The difference between Problem 1 and Problem 2 is CrashLoopBackOff versus Init:CrashLoopBackOff.

Problem 1 — a directory at the libvirt socket path

To identify, on the affected node, as root use the following command:

$ stat -c '%F' /var/run/libvirt/libvirt-sock-ro       # "directory" => affected

The libvirtd log on the node confirms the details:

$ grep virNetSocketNewListenUNIX /var/log/libvirt/libvirtd.log | tail -3

error : virNetSocketNewListenUNIX:514 : Failed to bind socket to
        '/run/libvirt/libvirt-sock-ro': Address already in use

Note

The above message is not accurate. Nothing holds the address — bind() fails because the path is a directory rather than a socket.

Procedural Changes:

To fix, on the affected node as root use the following command:

$ rmdir /var/run/libvirt/libvirt-sock-ro

Use rmdir rather than rm -rf. rmdir removes only empty directories, it provides an additional safeguard by failing if the target path contains unexpected files or content, helping to prevent accidental data loss.

Note

Deleting the pod is not required. After the directory is removed, the libvirt pod recovers automatically on its next retry, typically within 90 seconds.

Problem 2 — a directory at the rendered libvirtd.conf

To identify:

$ POD=$(kubectl -n openstack get pods --no-headers \
      | awk '/^libvirt-/ && $3 ~ /Init:/ {print $1; exit}')
$ kubectl -n openstack logs "$POD" -c init-dynamic-options --tail=10

/tmp/init-dynamic-options.sh: line 12: /tmp/pod-shared/libvirtd.conf: Is a directory

Procedural Changes: To fix, two objects must be removed, not one. Clearing only the first is not enough. First get the pod UID, where kubectl works:

$ PODUID=$(kubectl -n openstack get pod "$POD" -o jsonpath='{.metadata.uid}')

Then on the affected node, as root run the following:

# 1. the directory in the pod's emptyDir
$ rmdir /var/lib/kubelet/pods/$PODUID/volumes/kubernetes.io~empty-dir/pod-shared/libvirtd.conf

# 2. the stale subPath bind target
$ ls -la /var/lib/kubelet/pods/$PODUID/volume-subpaths/pod-shared/libvirt/
$ rmdir /var/lib/kubelet/pods/$PODUID/volume-subpaths/pod-shared/libvirt/<N>

Note

In step 2, first list the directory contents and identify the entry to remove rather than assuming the directory number. The number represents a volumeMount ordinal, which can vary between builds.

Restart the pod:

$ kubectl -n openstack delete pod "$POD"

A reboot does not fix this one. The emptyDir is only destroyed when the pod is removed from the node, so restarts and host reboots preserve the bad directory.

Problem 3 — nova-compute will not start after libvirt is fixed

If libvirt was down long enough for the nova-compute liveness probe to fail, the container restarts and looses its node identity file. This does not clear when libvirt recovers. The node reports unlocked / enabled / available while running no workloads.

To identify:

$ kubectl -n openstack logs <nova-compute-pod> -c nova-compute --tail=200 \
    | grep 'No local node identity found'

Note

Key on that string. The first ERROR in the traceback is a masking error (AttributeError: 'NoneType' object has no attribute 'stop') and will misdiagnose every instance of this.

On the node, as root:

$ ls -l /var/lib/nova/compute_id        # absent => affected

Procedural Changes: Run the following command:

$ kubectl -n openstack delete pod <nova-compute-pod>

Fix libvirt first. The nova-compute init container waits for a libvirt pod on the same node. If libvirt is still broken, the replacement pod simply remains in Init and nothing is gained.

Note

If the replacement pod hangs in Pending with references non-existent secret key: host-interface, that is a separate defect and must be cleared first.

Recommended order per host: Fix Problem 1 or 2 first, confirm libvirt reaches 1/1 Running, then check nova-compute and fix Problem 3 if required.

OpenStack Endpoint CA Certificate Required in Platform Truststore

When OpenStack is deployed with TLS endpoints, the OpenStack endpoint CA certificate must be trusted by the platform (imported into the platform truststore) so that the NFV-VIM can successfully establish TLS connections when making calls to OpenStack services.

If the OpenStack endpoint CA certificate is not present in the platform truststore, VIM TLS verification fails and VIM-to-OpenStack API calls fail.

Procedural Changes:

Install the CA Certificate

Replace <path-to-openstack-ca-cert> with the location of the OpenStack endpoint CA certificate on the controller.

Procedure

  1. Install the CA certificate into the system trust bundle. Skip this step if a CA with the same subject is already installed:

    $ system ca-certificate-install <path-to-openstack-ca-cert>
    

    Note

    Installing a CA certificate with the same subject replaces the existing entry. Re-running this command does not create duplicate truststore entries.

  2. Restart the VIM service to apply the newly trusted CA certificate:

$ sudo sm-restart-safe service vim

Serial Migrate Strategy May Fail on the Last Worker of a Two-Worker System

On systems with only two worker nodes, a serial update strategy that uses the migrate instance action may fail when processing the final worker. If the second worker begins its update sequence before the first worker has fully returned to service, no available worker remains to host the migrated instances. As a result, the update strategy stops before completion.

Procedural Changes: Delete the failed strategy, ensure the first worker has fully returned to an available state, and then recreate and reapply the strategy.

Cinder Volume Service Runs a Single Replica

StarlingX OpenStack runs a single cinder-volume replica on all deployment configurations, including AIO-DX and Standard systems with more than one controller. Cinder supports multiple concurrent volume services only in an Active-Active configuration, which requires a Cinder cluster name, a shared distributed lock manager, and per-node service identities. StarlingX OpenStack does not configure these, so a single replica is the only topology currently supported.

As a result, the volume service is not redundant: if the controller hosting the cinder-volume pod becomes unavailable, Kubernetes reschedules the pod onto the remaining controller, and volume and snapshot operations are unavailable until it restarts. Volumes already attached to instances are unaffected, as is guest I/O (data plane).

The cinder-api, cinder-scheduler, and cinder-backup services continue to run one replica per provisioned controller.

Note

Support for active-passive and active-active deployment models are planned for a future StarlingX OpenStack release.

Procedural Changes: N/A.

Manual OpenStack Application Re-apply During Controller Lock Can Stall MariaDB Recovery

On AIO-DX; AIO-DX Plus and Standard systems, manually re-applying the stx-openstack application while a controller is locked for configuration changes is not supported. With a controller in the locked state, the system runs in a reduced-redundancy state. Running system application-apply stx-openstack can then scale the MariaDB StatefulSet to a single replica and leave the surviving database pod unable to form the cluster, which stalls the apply. The apply may become stuck in the applying state (often around 10%), transition to apply-failed, and leave the mariadb-server-0 pod continuously restarting (0/1 Ready).

Warning

Do not manually re-apply stx-openstack while a controller is locked for configuration changes. If a re-apply was triggered by accident while the controller was locked and you then observe that the mariadb-server-0 pod is stuck (0/1), follow the recovery procedure below.

Procedural Changes: Recover a stalled MariaDB cluster after a manual re-apply of stx-openstack during a controller lock.

Symptoms:

  • system application-show stx-openstack shows status applying (often stuck around 10%) or apply-failed. Operators may notice either, sometimes only after a long delay. The recovery is the same for both.

  • The mariadb-server StatefulSet is not fully ready, and the mariadb-server-0 pod is 0/1 Running and restarting; its log repeats too old to make a decision for node mariadb-server-1.

Procedure

  1. Confirm the stuck state.

    $ kubectl -n openstack get sts mariadb-server
    $ kubectl -n openstack get pods -l application=mariadb -o wide
    $ kubectl -n openstack logs mariadb-server-0 -c mariadb --tail=20
    

    Confirm the StatefulSet shows READY 0/1, mariadb-server-0 is 0/1 Running, and its log repeats too old to make a decision for node mariadb-server-1.

  2. Recover MariaDB to a single healthy node by deleting the stuck pod.

    $ kubectl -n openstack delete pod mariadb-server-0
    
  3. Wait for the application to reach applied.

    $ system application-show stx-openstack
    

    Wait until the status is applied.

  4. Unlock the controller you locked.

    $ system host-unlock controller-<n>
    

    Wait for unlocked / enabled / available and the Kubernetes node Ready.

  5. Let the automatic re-apply scale MariaDB back to two nodes. Do not manually re-apply while reconciliation is in progress.

    $ kubectl -n openstack get sts mariadb-server
    $ kubectl -n openstack get hr -n openstack
    

    If any HelmRelease is not yet READY=True, reconciliation is still in progress — wait. Only if all HelmReleases are READY=True, the StatefulSet is still not at 2, and the app is in the applied state, run system application-apply stx-openstack.

    Note

    system application-show can lag; treat kubectl get hr -n openstack and the StatefulSet as authoritative.

  6. If the joining pod mariadb-server-1 stays 0/1 for several minutes (its log repeats Updating grastate configmap with no state transfer), delete it.

    $ kubectl -n openstack delete pod mariadb-server-1
    

    Note

    The StatefulSet may then roll mariadb-server-0 onto the current template; the mariadb-server-0 restart afterwards is normal.

  7. Verify the cluster and data.

    $ kubectl -n openstack get sts mariadb-server      # expect READY 2
    
    $ DBPW=$(kubectl -n openstack get secret mariadb-dbadmin-password -o jsonpath='{.data.MYSQL_DBADMIN_PASSWORD}' | base64 -d)
    
    $ kubectl -n openstack exec mariadb-server-0 -c mariadb -- \
        bash -lc "mariadb -uroot -p'$DBPW' -e \"SHOW STATUS WHERE Variable_name IN ('wsrep_cluster_size','wsrep_cluster_status','wsrep_local_state_comment','wsrep_incoming_addresses');\""
    
    $ kubectl -n openstack exec mariadb-server-1 -c mariadb -- \
        bash -lc "mariadb -uroot -p'$DBPW' -e \"SHOW STATUS WHERE Variable_name IN ('wsrep_cluster_size','wsrep_cluster_status','wsrep_local_state_comment','wsrep_incoming_addresses');\""
    

    Expect both nodes Primary, Synced. The expected wsrep_cluster_size is 2 on AIO-DX; 3 on AIO-DX Plus and Standard (two MariaDB servers plus the garbd arbitrator).

  8. AIO-DX Plus and Standard only: re-attach the garbd arbitrator if needed. On these configurations the healthy cluster size is 3. If wsrep_cluster_size reads 2 after recovery while both servers are healthy, the arbitrator has not rejoined. Restart it.

    $ kubectl -n openstack delete pod -l application=garbd
    

    The arbitrator rejoins and wsrep_cluster_size should return to 3. Skip this step on AIO-DX, which does not run garbd.

Kubernetes Upgrade Causes Temporary OpenStack Service Unavailability on AIO-SX

On AIO-SX systems, a Kubernetes upgrade cordons, drains, and then uncordons the node. After the node is uncordoned, stx-openstack is reapplied and OpenStack pods are recreated. During this process, control-plane pods may remain in the Init state for several minutes while their init containers wait for MariaDB and RabbitMQ to become available.

As system application-list reports application manifest status rather than pod readiness, it may show stx-openstack as applied / completed even though OpenStack control-plane services are not yet fully operational. No status field indicates this temporary service outage.

During this interval, OpenStack API requests and configuration operations, such as VM provisioning or system service-parameter-apply, can fail transiently because Keystone and other dependent services are still initializing.

Procedural Changes: Do not assume that an applied / completed application state means OpenStack is ready following a Kubernetes upgrade. Before resuming OpenStack operations, verify that all OpenStack pods are running and that the OpenStack API is responsive.

$  export KUBECONFIG=/etc/kubernetes/admin.conf kubectl get pods -n openstack | grep -Ev 'Running|Completed'

The command should return no output. Verify API availability:

$  source /var/opt/openstack/admin-openrc

$  openstack server list --all-projects

Wait until all OpenStack pods have completed initialization and API commands succeed before resuming VM operations or retrying configuration changes. Configuration values remain persisted; only the failed apply operation needs to be retried after OpenStack services have fully recovered.

Cinder Storage Service Catalog

Cinder Storage Service Catalog (SSC) does not recover from transient DNS failures when configured with NetApp volume backends.

A transient DNS outage can prevent SSC updates for NetApp backends, causing Cinder to stop detecting newly added storage pools on the NetApp system. As a result, the output of Cinder get-pools may become stale and continue to display only the NetApp pools that were registered before the DNS failure occurred.

Procedural Changes:

Following recovery from a transient DNS outage, restart the OpenStack cinder-volume pods to resume SSC updates:

$ kubectl -n openstack rollout restart deployment/cinder-volume

To help prevent future disruptions, you can pin the DNS host record for the NetApp SAN FQDN using the following command:

$ system service-parameter-add dns host-record netapp-san=<FQDN>,<IP>

This ensures that Cinder can continue to resolve the NetApp SAN endpoint even if external DNS resolution becomes temporarily unavailable.

Heat Service Depends on External DNS with FQDN Deployments

When OpenStack is deployed using an FQDN configuration, the Heat service relies entirely on external DNS resolution to process requests and communicate with other OpenStack services. As a result, an external DNS outage renders the Heat service unavailable. Access to the Heat API, including CLI operations, provisioning workflows, and automation processes, will fail until DNS service is restored.

Procedural Changes: During a DNS outage, manage OpenStack services directly through the OpenStack CLI rather than through Heat templates or Heat-based orchestration workflows.

Manual Removal of OpenStack Compute Service After Label Deletion

Any StarlingX OpenStack Release 13.0 deployment where a host ceases to function as an OpenStack compute node after the openstack-compute-node label is removed. A common scenario is the conversion of an OpenStack compute host into a Kubernetes-only worker in a Hybrid Cluster.

After the openstack-compute-node label is removed and the host is unlocked, OpenStack compute services are no longer running on the host. However, Nova continues to retain the host’s compute service and hypervisor records. The compute service remains in a disabled/down state, and the associated hypervisor remains down. These records are neither removed automatically nor aged out.

Although this condition does not affect scheduling on remaining compute hosts, because Nova does not schedule instances to disabled or down compute services, the Nova inventory no longer reflects the actual platform state. As a result, the host continues to appear in the output of commands such as OpenStack compute service list and OpenStack hypervisor list, as well as in Horizon.

$ openstack compute service list

+--------------------------------------+--------------+-----------+------+----------+-------+----------------------------+
| ID                                   | Binary       | Host      | Zone | Status   | State | Updated At                 |
+--------------------------------------+--------------+-----------+------+----------+-------+----------------------------+
| 6fc119e6-f333-4894-b1d8-33160094aa51 | nova-compute | compute-0 | nova | enabled  | up    | 2026-09-23T20:14:06.000000 |
| 391b1ea3-f93a-4760-b411-fab6e28e4085 | nova-compute | compute-2 | nova | enabled  | up    | 2026-09-23T20:14:08.000000 |
| 47e42935-826d-4131-a6c7-db77ec3ece20 | nova-compute | compute-1 | nova | disabled | down  | 2026-09-23T19:39:36.000000 |
+--------------------------------------+--------------+-----------+------+----------+-------+----------------------------+

$ openstack hypervisor list

+--------------------------------------+---------------------+-----------------+-------+
| ID                                   | Hypervisor Hostname | Hypervisor Type | State |
+--------------------------------------+---------------------+-----------------+-------+
| efacd07b-feeb-4407-b65c-176c13224880 | compute-0           | QEMU            | up    |
| 5127bb34-4ce0-4b08-ba92-5f9bb1e75429 | compute-1           | QEMU            | down  |
| e55d2a08-6449-4513-a60b-d14f8a41278e | compute-2           | QEMU            | up    |
+--------------------------------------+---------------------+-----------------+-------+

Procedural Changes: Once the label removal is complete, delete the compute service of the host manually. Deleting the compute service also removes the hypervisor record of the host.

# 1. Complete the label removal as documented (lock the host, remove the label, unlock the host).
# Wait until the host is unlocked / enabled / available and stx-openstack is applied.
$ system host-list
$ system application-show stx-openstack

# 2. On the active controller, load the OpenStack credentials.
$ source /var/opt/openstack/admin-openrc

# 3. Find the compute service of the host. Its State must be "down".
$ openstack compute service list --service nova-compute --host <hostname>

# 4. Delete it, using the ID returned by the previous command.
$ openstack compute service delete <service-id>

# 5. Verify that the host is no longer listed by either command.
$ openstack compute service list --service nova-compute
$ openstack hypervisor list

Note

  • Delete a compute service only when its State is down. A state of up indicates that the nova-compute service is still running on the host, either because the openstack-compute-node label is still assigned or because application reapply operations have not yet completed. Deleting the compute service while nova-compute is still running can leave orphaned hypervisor records.

  • Nova rejects the deletion request with an HTTP 409 error if the host still has instances (“Unable to delete compute service that is hosting instances. Migrate or delete the instances first.”) or contains in-progress migrations (“Unable to delete compute service that in-progress migrations. Complete the migrations or delete the instances first.”). A standard host lock operation typically migrates instances from the host before the lock completes. If the deletion is rejected, use openstack server list –all-projects –host <hostname> to identify any remaining instances, then migrate or delete them as appropriate, ensuring that any pending resize operations are either confirmed or reverted before retrying the deletion.

  • Deleting the compute service also removes the host from any associated host aggregates. If the host is later restored as an OpenStack compute node, re-add it to the required host aggregates.

  • After cleanup, the host can be returned to service as an OpenStack compute node normally. When the openstack-compute-node label is reassigned and the host is unlocked, Nova creates a new compute service and the associated hypervisor returns to the up state.

Aborted stx-openstack Application Update Can Leave MariaDB Degraded Without Alarms on AIO-DX Systems

On an AIO-DX system, aborting a stx-openstack application update, such as a OpenStack Release 12.0 to OpenStack Release 13.0 that is manually aborted or automatically aborted due to a release failure, can leave the MariaDB cluster in a degraded state. The mariadb-server StatefulSet may be reduced to a single replica (replicas=1), with mariadb-server-1 missing and mariadb-server-0 stuck in a 0/1 (Running but Not Ready) state while repeatedly failing its startup probe.

Despite the database outage, system application-show stx-openstack reports status=applied and displays "Application recover to version [old] completed", while fm alarm-list reports no active alarms. As a result, the control-plane database is unavailable even though the platform indicates a successful recovery and no fault condition.

The mariadb-server-0 pod log repeatedly reports the following message: “The data we have from the cluster is too old to make a decision for node mariadb-server-1”.

This condition occurs because the cluster-state ConfigMap still contains a stale reference to mariadb-server-1, causing MariaDB to wait for a non-existent replica and preventing the remaining node from becoming ready.

Procedural Changes:

Use the exact underscore form of the keys shown below:

  1. Confirm the stuck state.

    $ kubectl get sts -n openstack mariadb-server            # replicas=1
    $ kubectl get pods -n openstack | grep mariadb-server    # server-0 0/1, server-1 absent
    
  2. Back up the cluster-state ConfigMap.

    $ kubectl get configmap -n openstack osh-openstack-mariadb-mariadb-state -o yaml > cm.backup.yaml
    
  3. Remove the stale phantom server-1 entries.

    $ kubectl patch configmap -n openstack osh-openstack-mariadb-mariadb-state --type merge \
    p '{"data":{"safe_to_bootstrap.mariadb-server-1":null,"sample_time.mariadb-server-1":null,"seqno.mariadb-server-1":null,"uuid.mariadb-server-1":null,"version.mariadb-server-1":null}}'
    
  4. Restart server-0 -> bootstraps to 1/1 with data intact

    $ kubectl delete pod -n openstack mariadb-server-0
    
  5. Restore HA: scale back to 2 -> server-1 rejoins and re-syncs.

    $ kubectl scale sts -n openstack mariadb-server --replicas=2
    
  6. Verify: 2/2 ready, wsrep_cluster_size=2, no mariadb alarms, data present.

Note

After step 4, mariadb-server-0 successfully bootstraps and reaches a 1/1 Ready state with its data intact, allowing dependent pods to recover. After step 5, mariadb-server-1 rejoins the cluster, restoring the Galera cluster to its expected two-node configuration.

Manual Hiera Workaround Required for OVS-DPDK Deployments Before Upgrading to StarlingX Release 13.0

OVS-DPDK deployments require a manual Hiera workaround on all controllers before upgrading from StarlingX Release 12.0 to StarlingX Release 13.0.

When upgrading StarlingX Release 12.0 deployments using vswitch_type = ovs-dpdk, DPDK parameters (host_core_list, pmd_core_list, socket_mem) may not be preserved correctly during upgrade, causing OVS-DPDK networking to fail post-upgrade.

Procedural Changes: Before performing the upgrade, execute the following steps on all controllers:

Procedure

  1. Create the manual overrides directory for both controllers.

    $ sudo mkdir /etc/puppet/manual_overrides/
    
  2. Identify the current DPDK Hiera values.

    $ sudo find /etc -type f -name "*.yaml" -print0 2>/dev/null | sudo xargs -0 grep -nH "vswitch::dpdk"
    /etc/puppet/cache/hieradata/controller-0.yaml:462:vswitch::dpdk::host_core_list: '0,2'
    /etc/puppet/cache/hieradata/controller-0.yaml:463:vswitch::dpdk::pmd_core_list: '2'
    /etc/puppet/cache/hieradata/controller-0.yaml:464:vswitch::dpdk::socket_mem: '1024'
    
  3. Create the override file with the correct values, from step 2, for your deployment.

    # /etc/puppet/manual_overrides/override.yaml
    vswitch::dpdk::host_core_list: '0,2'
    vswitch::dpdk::pmd_core_list: '2'
    vswitch::dpdk::socket_mem: '1024'
    
  4. Update /etc/puppet/hiera.yaml to include the manual overrides as the first item in the hierarchy.

    Warning

    The YAML format is critical — indentation must be exact. Do not modify what is already present in the file.

    version: 5
    hierarchy:
      - name: "Manual Overrides"
        datadir: /etc/puppet/manual_overrides
        data_hash: yaml_data
        path: "override.yaml"
      - name: "yaml"
        datadir: /tmp/puppet/hieradata
        data_hash: yaml_data
        paths:
          - runtime.yaml
          - host.yaml
          - secure_system.yaml
    

Enabling DHCP on a Subnet with no Free IPs Gives False Success

Enabling DHCP on a subnet with an exhausted allocation pool succeeds at the API level but DHCP will not function. When an operator enables DHCP on an existing subnet (enable_dhcp False→True) whose allocation pool has no free IP addresses, the Neutron API returns success (HTTP 200). However, the DHCP agent fails asynchronously to create its required port due to IP address exhaustion. This results in:

  • No network:dhcp port being created on the subnet.

  • VMs launched on that subnet failing to obtain IP addresses via DHCP/cloud-init.

  • The failure is visible in the DHCP agent logs (IpAddressGenerationFailure), not to the API caller.

Procedural Changes:

Before enabling DHCP on an existing subnet, verify that the allocation pool has at least one free IP:

$ openstack ip availability show <network-id>

If used_ips equals total_ips for the relevant subnet, either:

  • Expand the allocation pool.

    openstack subnet set --allocation-pool start=<new-start>,end=<new-end> <subnet-id>
    
  • Or, delete an unused static port to free an IP address.

Then enable DHCP.

$ openstack subnet set --dhcp <subnet-id>

Restore Complete Operation Can Fail on Large-Scale OpenStack Systems

On a large system (many hosts) running StarlingX OpenStack, the final restore step can fail with the message:

Unable to perform restore modify state.

On large-scale OpenStack systems, the restore-complete step can fail when the OpenStack application keeps the system management service heavily loaded. The restore operation itself succeeds, but the completion phase cannot finish while the service is busy.

Impact

  • The backup-and-restore procedure does not complete on its own on a large system.

  • The system appears stuck at the final step until the workaround is applied.

  • Smaller systems are not expected to encounter this.

Procedural Changes

Procedure

  1. Turn off the automatic application re-evaluation so the management service is released.

    $ system service-parameter-modify platform config evaluate_apps_reapply=disabled
    $ system service-parameter-apply platform
    

    Wait for 30 seconds.

  2. If the system has been stuck for a while, restart the management so it starts again.

    $ sudo sm-restart-safe service sysinv-conductor
    $ sudo sm-query service sysinv-conductor
    

    Confirm no controller switchover was initiated.

  3. Rerun the restore completion step.

    $ system restore-complete
    $ system restore-show
    

    Result: “Restore procedure is not in progress.”

  4. Re-enable the re-evaluation.

    $ system service-parameter-modify platform config evaluate\_apps\_reapply=enabled
    system service-parameter-apply platform
    
  5. Re-apply the applications in the following order: storage first, then the remaining applications.

    $ system application-apply rook-ceph
    $ system application-apply stx-openstack
    

glance-api Pods Stay in CrashLoopBackOff After Platform Restore

On systems that use Ceph (rook-ceph) storage, the stx-openstack application is reapplied when you run system restore-complete during a platform restore. The glance-api pods start before their Ceph credentials are ready. They stay in CrashLoopBackOff with rados.PermissionDeniedError: [errno 13] in the pod log and do not recover on their own. Pods that wait for Glance, such as nova-compute and pci-irq-affinity-agent, stay in Init state, and stx-openstack can show apply-failed. The platform restore itself and your OpenStack backup data are not affected.

Procedural Changes: Delete the glance-api pods so that they are recreated. Run these commands on the active controller after system restore-complete:

Procedure

  1. Check the state of the glance-api pods.

    $ export KUBECONFIG=/etc/kubernetes/admin.conf
    $ kubectl -n openstack get pods | grep glance-api
    
  2. If the pods are in CrashLoopBackOff, delete them. Only the stuck pods are replaced. No data is removed.

    $ kubectl -n openstack delete pod -l application=glance,component=api
    
  3. Wait until the new glance-api pods are Running and 1/1 ready.

    $ kubectl -n openstack get pods -w | grep glance-api
    
  4. Verify that the pods that were waiting for Glance have left Init state.

    $ kubectl -n openstack get pods | grep -E "glance-api|nova-compute|pci-irq-affinity-agent"
    
  5. Continue with the OpenStack application restore. It is expected that stx-openstack still shows apply-failed until the application restore is done.

Note

If you reapply applications manually, reapply rook-ceph before stx-openstack. Deleting the glance-api pods helps only after the Ceph credentials for Glance exist.

/var/opt/openstack Not Included in Backup and Restore

The StarlingX OpenStack platform and application backup and restore does not save or restore the contents of /var/opt/openstack. After a restore, the stx-openstack application is reapplied and recreates the directory with only its default files: admin-openrc, clear-aliases.sh, clients-wrapper.sh, local_openstackrc, and setup-aliases.sh. Any file you added to the directory is lost, for example tenant OpenStack RC files, certificates, custom scripts, and the images/ directory. OpenStack resources contained in the application backup (VMs, volumes, networks, images) are not affected.

Procedural Changes: Back up the directory yourself and restore it after the restore completes.

Backup

On the active controller, after the platform and application backup has completed:

Procedure

  1. Create the archive and restrict access to it. The archive contains OpenStack credentials and certificates.

    $ sudo tar czf /home/sysadmin/var-opt-openstack-$(hostname)-$(date -u +%Y%m%d%H%M%SZ).tgz -C /var/opt openstack
    $ sudo chown sysadmin: /home/sysadmin/var-opt-openstack-*.tgz
    $ sudo chmod 600 /home/sysadmin/var-opt-openstack-*.tgz
    
  2. Verify the archive is readable and lists your files.

    $ ls -lh /home/sysadmin/var-opt-openstack-*.tgz
    $ tar tzf /home/sysadmin/var-opt-openstack-<host>-<timestamp>.tgz
    
  3. Copy the archive off the system, together with the platform and application backup files. Anything left on the system is lost if the system is wiped and reinstalled.

Restore

After the platform restore is complete and the stx-openstack application has been restored and reapplied:

Procedure

  1. Confirm that only the default files are present.

    $ ls -l /var/opt/openstack
    
  2. Copy the archive back to /home/sysadmin on the active controller and extract it over the directory. Use sudo so that the original owners and permissions are restored.

    $ sudo tar xzf /home/sysadmin/var-opt-openstack-<host>-<timestamp>.tgz -C /var/opt
    
  3. Verify that your files are back with their original owner, group, and permissions, and that the OpenStack CLI still works.

    $ ls -l /var/opt/openstack
    $ source /var/opt/openstack/admin-openrc
    $ openstack server list --all-projects
    

Note

Extracting the archive also replaces the five default files with the copies saved at backup time. If you changed the OpenStack admin password or endpoints after taking the backup, update admin-openrc afterwards.

Supported Deployment Configurations

StarlingX OpenStack Release 13.0 is supported with the following deployment configurations:

  • Standalone Cloud Deployments

    • All-In-One Simplex

    • All-In-One Duplex

    • Standard

  • Distributed Cloud Deployments

    • Subclouds

      • All-In-One Simplex

      • All-In-One Duplex

      • Standard

Guest OS on VMs

This section lists the validated Guest OSs on VMs in StarlingX OpenStack Release 13.0.

Note

Users may need to adjust the kubelet filesystem size on the controllers to successfully create bootable volumes. For huge images intended for bootable volumes, increasing the kubelet filesystem size may be necessary. The required kubelet filesystem size depends on the size of the image being used.

Booting a server using the huge image does not require further adjustments on kubelet.

system host-fs-modify controller-0/1 kubelet=<size>

Note

If using hw:cpu_realtime=yes for a VM running a real-time guest operating system:

  • Enable a low-power (halt-based) idle mode in the guest so idle vCPUs release the physical CPU instead of polling. See the guest operating system documentation for the required kernel option.

  • Configure the stalld daemon on all compute hosts that might run the real-time VM. For more information see Configure Stall Daemon.

This prevents busy-looping real-time tasks in the guest from blocking critical host kernel tasks, which could otherwise trigger a watchdog reset.

Guest OS Compatibility Matrix

The following table lists the Compatibility Matrix for Guest OS, it follows a lifecycle from image booting and follows the sequence below while trying to connect to a VM during each step:

  1. Stop/Start

  2. Pause/Unpause

  3. Suspend/Resume

  4. Shelve/Unshelve

  5. Reboot

Each GuestOS is validated 3 times for each boot type:

  • Boot from image

  • Boot from volume

  • Boot from snapshot

OS Image

BIOS Boot

UEFI Boot

UEFI Secure Boot

AlmaLinux-9

Yes

Yes

Yes

AlmaLinux-10

Yes

Yes

Yes

AlpineLinux

Not validated

Yes

Yes

Arch Linux (2026.02.01)

Yes

Yes

Yes

CentOS-9 (BIOS/UEFI)

Yes

Yes

Yes

CentOS-10 (BIOS/UEFI)

Yes

Yes

Yes

CirrOS

Yes

Yes

Yes

Debian-11 (Bullseye)

Yes

Yes

Yes

Debian-12

Yes

Yes

Yes

Debian-13 (Trixie)

Yes

Yes

Yes

Fedora CoreOS (Stable)

Yes

Yes

Yes

openSUSE Leap 15.3

Yes

Yes

Yes

openSUSE Leap 16.0

Yes

Yes

Yes

Oracle Linux 9

Yes

Yes

Yes

Oracle Linux 10

Yes

Yes

Yes

Photon OS 5.0

Yes

Yes

Not validated

RHEL 8

Yes

Yes

Yes

RHEL 9

Yes

Yes

Yes

RHEL 10

Yes

Yes

Yes

Rocky Linux

Yes

Yes

Yes

Ubuntu 20.04 (Focal)

Yes

Yes

Yes

Ubuntu 22.04 (Jammy)

Yes

Yes

Yes

Ubuntu 24.04 (Noble)

Yes

Yes

Yes

Ubuntu 26.04 (Resolute)

Yes

Yes

Yes

Elxr 26.04.01

No

Yes

Yes

Windows 11

Not validated

Yes

Yes

Windows 2016

Not validated

Yes

Yes

Windows 2019

Not validated

Yes

Yes

Windows 2022

Not validated

Yes

Yes

Windows 2025

Not validated

Yes

Not validated

Open Source Project Versions for OpenStack

This section documents key open source project versions for StarlingX OpenStack.

For more information on all StarlingX OpenStack Open Source Project Versions, see https://releases.openstack.org/gazpacho/index.html.

Open Source Packages

Version

Open vSwitch

3.5.0

DPDK

24.11.4

libvirt

11.3.0

QEMU

10.0.8

MariaDB

11.4.8

RabbitMQ

3.11.28

memcached

1.5.5

Deprecation and Removal Notices

This section lists capabilities that are either deprecated (still functional, but planned to be removed in a future release), and removed in StarlingX OpenStack.

Deprecated in StarlingX OpenStack 13.0

N/A.

Removed in StarlingX OpenStack 13.0

Host-based Ceph for Storage Backends

Host-based Ceph support for storage backends of Cinder, Glance and Nova is deprecated in StarlingX OpenStack Release 25.09 and is replaced by rook-based Ceph.

Note

Use one of the supported StarlingX OpenStack storage backends: Rook-based Ceph, NetApp (NFS / iSCSI / FC) and/or Extended Storage Backends (ESB).

Warning

Upgrades and migrations from StarlingX OpenStack Release 12.0 systems using Host-based Ceph to StarlingX OpenStack Release 13.0 are not supported. If a migration is required, it must be performed using external migration tools and application-level migration procedures.

VLAN Transparency (Networks)

VLAN transparency is no longer supported by the OVS mechanism driver. Creating networks with the --transparent-vlan option has been deprecated and removed. To carry VLAN-tagged traffic from guest instances, use the Neutron trunk extension (port trunking). For more information, see Enable the Trunk Extension for Neutron.

Hardware Updates

See:

Bug status

Fixed bugs

This release provides fixes for a number of defects. Refer to the StarlingX bug database to review the Release 13.0 Fixed Bugs.