StarlingX Kubernetes 13.0 Release Notes¶
The sections below provide a detailed list of new features, enhancements, and updates, and links to the associated user guides (if applicable).
ISO image¶
The pre-built ISO (Debian) for StarlingX 13.0 is located at the
StarlingX mirror repo:
Source Code for StarlingX 13.0¶
The source code for StarlingX 13.0 is available on the r/stx.13.0 branch in the StarlingX repositories.
Deployment¶
To deploy StarlingX 13.0, see Consuming StarlingX.
For detailed installation instructions, see StarlingX 13.0 Installation Guides.
New Features / Enhancements¶
This release introduces several new features designed to improve usability and efficiency, along with targeted enhancements to existing functionality. Performance, stability, and overall user experience have been enhanced. Some features may have known limitations or constraints in this release; these are documented to help set expectations and guide usage.
StarlingX Release 13.0 builds on previous releases by enabling organizations to deploy, manage, and upgrade infrastructure more efficiently while maintaining the high availability and reliability required for production environments. The release introduces enhancements across platform operations, workload orchestration, observability, and interoperability to help reduce operational complexity and improve deployment flexibility.
Software Management / Upgrades¶
StarlingX Upgrade Support to Release 13.0¶
StarlingX Release 13.0 supports platform software upgrades directly from StarlingX Release N-2 and from StarlingX Release N-1 to StarlingX Release 13.0, introducing N-2 to N major-release upgrades. The upgrade applies across all deployment configurations (AIO-SX, AIO-DX and AIO-DX + workers, and Standard with or without storage nodes) and all system types (Standalone, Distributed Cloud System Controller, and Subcloud). A Version N System Controller can now manage subclouds running N, N-1, and N-2 releases starting with N=13.0.
Upgrades can be driven through the software deploy CLI,
sw-manager sw-deploy-strategy, and dcmanager sw-deploy-strategy, and
may be combined with a Kubernetes upgrade to v1.35. Orchestrated rollback is
supported across all configurations.
An N-2 to N upgrade is performed directly rather than as a chained N-2 to N-1 to N sequence, completing with a single reboot per host, and requires a minimum “from” release patch level on the source system.
Note
StarlingX does not test upgrades from one release to the next. Upgrade testing is the responsibility of the user.
Note
N-2 and N-1 software patches maybe required to support the upgrade to Stx13.0. The patch build is the responsibility of the user.
Upgrading and Rollback Process Optimization¶
This feature combines the platform software upgrade and the Kubernetes upgrade into a single, streamlined procedure for AIO-SX systems (standalone and Distributed Cloud subclouds). The Kubernetes control plane is upgraded in-service ahead of the platform upgrade, and the kubelet is upgraded during the same host reboot that boots into the new platform software, eliminating the separate host cordon and uncordon cycle.
A new software system-deploy init and delete command set frames the
combined operation. The procedure supports rollback on manual abort or failure,
returning the system to its original platform and Kubernetes versions while
preserving application information. Orchestration is available at both the
subcloud level and the Central cloud level through an added attribute that
specifies the target Kubernetes version. The existing separate platform-only
and Kubernetes-only upgrade procedures remain available.
See:
StarlingX Software Management Componentization¶
This feature introduces finer-grained software management for StarlingX. The internal packaging of a Product Patch Release is restructured into a set of components (metapackages) that can be managed individually, while the platform continues to be released as complete Product Major and Product Patch Releases.
In StarlingX Release 13.0, the metapackages are:
swmgmt – the software management framework
distcloud – Distributed Cloud software
infra – StarlingX Flock Services (minus swmgmt and distcloud)
k8s-v<version> – versioned Kubernetes software components
k8s-common – common Kubernetes-related code supporting all versions
base – everything else (OS, drivers, kernel)
After uploading a Product Patch Release, operators can list the component patch releases it contains and choose to deploy the entire release as before, or selectively deploy a subset of components subject to inter-component dependencies. Selective component / metapackage deployments is supported via ‘software deploy’ commands and ‘sw-manager sw-deploy-strategy’ orchestration commands.
Note
Legacy commands and procedures for deploying full Product Patch Releases remain unchanged. In this release, component-level deployment is available on standalone systems only.
See:
Platform Components Updates (Linux OS, Kernel, Drivers, Debian, Base Components)¶
The platform base operating system and foundational components are up-versioned to their latest supported long-term-support levels. Kernel commits enabling DPLL timing-module firmware and configuration updates through the devlink interface are backported. Support is added for sub-NUMA clustering and configurable per-interface NIC packet queues.
Platform upgrades include data migration and configuration handling for PostgreSQL, Puppet, RabbitMQ, and OpenStack.
Cgroups v2 is now the default, with a service parameter available to revert to cgroups v1 for exceptional cases.
See:
Platform OpenStack Services Updated to Epoxy¶
StarlingX OpenStack services (Keystone, Barbican, Horizon) have been updated from Victoria (2020.2) to Epoxy (2025.1), aligning with the upstream Debian Trixie packages.
Barbican Role-Based Access Control (RBAC)¶
Barbican now enforces the upstream “new defaults” RBAC policy.
See: Keystone Account Roles.
devlink Support for GNR-D Timing Module¶
The devlink utility provides a standardized interface for device management in Linux, including firmware update operations. Users can identify a ZL3073x device, verify its firmware version, update the firmware, and validate the update.
Note
Firmware version upgrades are outside the scope of StarlingX. This capability is provided solely to facilitate user-required firmware version changes when necessary. Users are responsible for planning, validating, and executing any firmware version updates.
Support for Sub-NUMA Clustering¶
StarlingX supports Sub-NUMA Clustering (SNC) on Intel platforms and Nodes Per Socket (NPS) on AMD platforms. This setting partitions a processor into multiple NUMA domains and is configured in the server firmware. The platform detects and adapts to the configured NUMA topology on boot.
StarlingX supports multi-NUMA-node configurations on single-socket and multi-socket servers. The NUMA topology is determined by the processor’s Sub-NUMA Clustering (Intel) or Nodes Per Socket (AMD) firmware setting.
See: Change the SNC/NPS Setting
Configure Number of NIC Packet Queues (Channels) per Interface¶
NIC channels (queues) can be configured per interface on systems with multiple network interfaces. By default, the number of NIC queues is configured automatically on a system.
By default, the number of queues is configured to the number of platform cores for platform interfaces, and the number of application cores for SR-IOV interfaces.
This feature targets StarlingX memory optimization to facilitate field deployment of GNR-D nodes with 128 GB memory for Samsung vRAN application.
See: NIC Queue Configuration.
Kubernetes v1.35.2 and Container Components Up-version in StarlingX 13.0¶
The Kubernetes version supported in StarlingX Release 13 is v1.35.2, and all container infrastructure components are updated to versions compatible with Kubernetes v1.35.2.
Core container runtime components updated include containerd, runc, crictl, and etcd (upgraded to 3.6.9).
Note
Platform upgrades from StarlingX Release 11 and Release 12 to Release 13 are supported, including Kubernetes version upgrades.
Containerized Application Updates¶
All StarlingX containerized platform applications are up-versioned to their latest upstream releases compatible with Kubernetes v1.35.2, incorporating current CVE fixes. The maintained applications include
cert-manager
nginx-ingress
oidc-auth
istio
rook-ceph
platform-integ-apps
kubevirt
oran-o2
dell-storage
snmp
portieris
power-metrics
intel-device-plugins
metrics-server
ptp-notification
deployment-manager
security-profiles-operator
kernel-module-management
ipsec-policy-operator
node-interface-metrics-exporter
Helm/FluxCD application infrastructure
Distributed Cloud¶
Subcloud Enrollment Duration Reduction Ph1 - AIO-SX 20 mins¶
This feature reduces the time required to enroll a standalone StarlingX system as a subcloud within a distributed cloud, focusing on single-server (AIO-SX) deployments. It is the first phase of a multi-phase effort toward an eventual sub-10-minute enrollment goal. In this phase, enrollment and configuration completes in 20 minutes or less for patch-current systems and around 45 minutes or less for systems that require a software update during enrollment. The optimization applies to distributed cloud across supported configurations.
The feature streamlines multiple stages of the enrollment path. It removes redundant deployment steps, defers the audit that delays OAM network reconfiguration, refactors Keystone reconfiguration into a dedicated utility while eliminating repeated service restarts, consolidates fragmented certificate renewal behind a single expiry check, and reuses a shared authentication token and session across Ansible playbook execution. SR-IOV interface configuration is applied in bulk to avoid frequent host updates and redundant configuration passes.
Note
SR-IOV interface bulk configuration could be used separately based on field needs.
Restore Without Reinstall for Pre-installed Node¶
This feature lets operators restore a StarlingX system from a backup archive onto a factory-installed node without first performing an OS and platform reinstall. Restore now offers two options from a single backup image: the existing restore to a pre-staged, OS-only system, and a new restore to a pre-installed system that has already been bootstrapped and configured at the factory (optionally including workload software such as vCSR or a generic golden image). Since the install step is eliminated and only site-specific configuration needs to be applied, the system returns to service far more quickly. The capability applies to AIO-SX standalone cloud and Distributed Cloud AIO-SX subclouds, with no additional hardware dependency.
The feature supports both remote and local restore procedures, with and without vCSR. Remote restore is performed using BMC Redfish virtual media, and local restore is performed using a USB device or the local install utility. It uses cloud-init to seed the site-specific configuration and stage the backup contents, minimizing reboots during recovery. The platform automatically detects the current system state to select the correct restore method, and it validates that the backup and target are compatible. For AIO-SX standalone clouds, the in-place restore targets completion are within 15 minutes.
See:
vCSR ISP Local Site Enrollment and Disaster Recovery¶
This feature introduces local, on-site procedures for enrolling and recovering All-in-one (AIO-SX) subclouds that run virtual Cell Site Router (vCSR) software in ISP (Internet Service Provider) backhaul deployments without remote BMC access. In these sites, connectivity to the customer core network is provided only through an IPsec tunnel that becomes available once the node is up and running, so system setup, subcloud enrollment, and disaster recovery must be driven locally by field personnel rather than remotely through the Distributed Cloud manager.
The feature adds a local system setup step that uses a cloud-init NoCloud seed image, auto-mounted from a local USB device or Redfish virtual media, to apply the site-specific configuration, credentials, networking, and vCSR settings needed to bring the node into communication with the System Controller. From this common local setup, field personnel can initiate several operations against the System Controller DC Manager API:
enrollment of a factory-installed subcloud;
system restore from a local backup archive;
system restore from a remote backup transferred and staged from the System Controller; and
restore to factory defaults through a new recovery bootloader option for systems that have become unresponsive.
A new DCManager enrollment option recognizes systems that have already completed the enrollment-init phase locally so that the step is not repeated. If enrollment fails, the node can be returned to a factory-default state, and for hardware-replacement cases the replacement node can be brought up and restored with the site-specific details held by the System Controller. In this release, vCSR is validated with the Juniper Cloud-Native Router (JCNR).
See:
Golden Image for Factory / Warehouse Install¶
This feature introduces a factory “golden image” process for StarlingX, allowing systems to be pre-installed, configured, and validated in a controlled lab or factory environment before being deployed at scale from a warehouse. A validated baseline can be captured and reproduced across many identical systems, which are then made unique and production-ready during enrollment or site-specific restore at the deployment location. The capability applies to standalone cloud factory installation and to Distributed Cloud subcloud enrollment.
Note
Only systems with identical hardware configurations can share the same golden image. The image must be created from and validated on the exact hardware configuration on which it will be deployed.
The feature supports two cloning approaches, Smart Disk Clone and Install and Restore, so a warehouse can quickly reproduce a validated baseline and then perform subcloud enrollment or a site-specific restore. Since cloned systems start out identical, the platform regenerates all system-unique identifiers during enrollment, including the system and host UUIDs, the platform install UUID, the Linux machine-id, the Ceph File System Identifier (FSID), and the Keystone region name, after integrity checks so the factory baseline is not invalidated. Security material is also refreshed: SSH host keys are regenerated and platform server and client certificates are updated based on the provided root and intermediate CAs and site-specific parameters.
Security¶
Stronger Encryption Algorithms for Data and Communications - 256 / 384 / 512 bits Ph1¶
This feature is the first phase of an effort to strengthen the cryptography that protects StarlingX data and communications. It delivers configurable control over the TLS versions and Cipher suites accepted by the platform’s HTTPS endpoints, and support for stronger X.509 certificate key types and longer key sizes across the platform’s certificate authority and core service certificates. It applies to all StarlingX configurations, including standalone and Distributed Cloud deployments.
Operators can now set the minimum TLS version (TLS 1.2 or TLS 1.3) and a restricted list of allowed Cipher suites for the externally visible StarlingX HTTPS server endpoints; the StarlingX REST APIs, StarlingX Horizon, the registry.local container registry, the local LDAP service, and the OIDC endpoints through new platform configuration service parameters. New installations default to a curated set of strong AEAD cipher suites that provide forward secrecy through ECDHE key exchange, with legacy protocols and weak Ciphers excluded. The feature also raises certificate key strength: the platform CA and core platform certificates default to ECDSA P-384 on new installs, with validation that enforces minimum key sizes.
See:
StarlingX PKI Implementation Enhancement Ph1 - EJBCA System Application (WITHOUT HSM)¶
This feature is the first phase of a multi-release enhancement to the StarlingX public key infrastructure. It integrates the mature, open-source EJBCA project as an optional certificate authority (CA) solution, packaged and delivered as a StarlingX system application. The application is available across all StarlingX configurations, including standalone (AIO-SX, AIO-DX, Standard) and Distributed Cloud (System Controller and subcloud) deployments.
The EJBCA system application deploys a scalable and highly available EJBCA instance backed by a PostgreSQL database with primary and standby replicas and automatic failover. It exposes a standards-based Certificate Management protocols V2 (CMPv2) interface on the StarlingX OAM network for certificate lifecycle operations. Workload applications hosted on StarlingX can obtain signed certificates through a cert-manager issuer configured to talk to EJBCA, while external devices and applications with network access to StarlingX can manage and certificates directly using CMPv2. In this release, EJBCA is intended for the certificates of guest workload and external applications; StarlingX platform certificates continue to be managed as before and will adopt EJBCA in a future release.
See:
Configurable Parameters for Inactive Session Termination and Suspended Account Access due to Consecutive Invalid Login Attempts¶
This feature introduces configurable security parameters for automatic termination of inactive user sessions and temporary suspension of account access after a defined number of consecutive invalid login attempts. These controls help users enforce security policies, reduce the risk of unauthorized access, and align with compliance requirements.
See:
Unified Identity Management (OIDC/Dex) Ph2 - Horizon and StarlingX CLI¶
This is the second phase of unified identity management for StarlingX, extending OIDC authentication coverage to the Horizon web administration GUI and enhancing the usability of OIDC authentication for StarlingX command-line clients. Keystone authentication remains fully supported and is still the default; Keystone and OIDC logins can be used simultaneously by different users.
Key capabilities:
Horizon login page adds an OIDC login option working with all supported Dex backends (Local LDAP, remote Windows Active Directory, and remote OIDC).
StarlingX CLIs gain
oidc-loginplugin support, sharing OIDC identity provider configuration and cached tokens in $KUBECONFIG with kubectl.Server-side OIDC token validation moves to offline (local) validation using cached public keys, improving performance and resiliency when connectivity to the Dex identity provider is lost.
Recommended email, username, and name attribute mappings are corrected and tested across all connector types.
Default Local LDAP connector settings updated on new installs; existing configurations unchanged on upgrade.
See:
Networking¶
Open vSwitch (OVS) L2 Forwarding¶
This feature adds Layer 2 forwarding to StarlingX networking, enabling a Granite Rapids-D (GNR-D) server to act as a Layer 2 bridge between servers for backhaul and management network operations. It targets deployments that have no adjacent Cell Site Hub Router and connects the GNR-D servers directly to aggregate Service Edge (SE) and Route Engine (RE) routers over a Layer 2 network. The capability is delivered through a new Open vSwitch (OVS) platform application and is supported across standalone and Distributed Cloud configurations.
OVS bridges traffic between SR-IOV physical and virtual functions and internal or patch ports, and forwards traffic between redundant servers for network access and redundancy. It supports BMC zero-touch provisioning forwarding for a second GNR-D node, cloud OAM and management traffic redirection, and third-party O-RAN Radio Unit (ORU) management traffic forwarding over dedicated VLAN, together with VLAN tagging and trunking, MAC address learning and aging, and IPv4 and IPv6 unicast and multicast bridging. Bidirectional Forwarding Detection (BFD) monitors each network endpoint for fast link-failure detection and topology changes. Configuration is declarative: a new Kubernetes operator and two custom resource definitions, OVSBridge and OVSPort, describe the desired bridge and port state, and a node-local agent applies it with idempotent reconciliation.
See:
Cloud Platform Calico Network Operator¶
StarlingX introduces the Cloud Platform Calico Network Operator to manage the lifecycle of Calico CNI networking across all hosts in a StarlingX system. It replaces the previous manifest-based installation with an operator-driven model that follows the standard Kubernetes operator paradigm, continuously reconciling the actual state of the cluster network to the desired state.
The operator provides:
Automatic platform and configuration detection.
Simplified upgrade procedure with retry and rollback on failure.
Correct host upgrade order enforcement (controllers before workers).
User-specified overrides applied to the Calico Installation resource.
IPv4-only, IPv6-only, and dual-stack support.
A Calico Ingress Gateway implementing the Kubernetes Gateway API.
Existing deployments are migrated from the manifest-based method during the standard platform upgrade. Major, minor, and patch upgrades of the operator are integrated into the Kubernetes cluster network upgrade framework.
See:
System Configuration Guide
Admin Guide
Ethernet Frequency Synchronization (SyncE) as Additional Timing Source¶
Synchronous Ethernet (SyncE) adds a physical-layer frequency synchronization source to StarlingX, distributing timing information across an Ethernet network from a leader clock to subordinate devices. This release adds support for the Ethernet Synchronization Messaging Channel (ESMC) and delivers O-RAN compliant SyncE notifications to the O-DU (O-RAN Distributed Unit).
Capabilities:
ESMC processing on incoming and outgoing SyncE links.
Incoming frequency Quality Level (QL) passed through to outgoing ESMC messages.
Configurable priority of SyncE relative to GNSS.
Defined failover across SyncE, GNSS, PTP, local holdover, and free-run.
Alarm raised when SyncE is configured but not available.
O-RAN Cloud Notification API SyncE Extended Lock State and Clock Quality Change notifications.
Extended holdover exceeding 20 days (depending on timing module hardware).
See:
Counters for Timing Sources¶
This feature provides customer-facing counters and metrics for the timing
sources used by StarlingX, giving operators visibility into the health and
stability of PTP synchronization. The ptp-metrics-exporter feature is
shipped as an OpenMetrics-compliant option alongside the ptp-notification
application.
Exposed metrics include:
Time spent in each synchronization state over the last measurement period
Sync messages received and lost
Average drift of the reference timing
Distribution of synchronization quality
Metrics can be labeled by interface, domain, and grandmaster identity and are scoped at the system, host, and interface levels. The endpoint is configurable (enable/disable and port selection), disabled by default, and designed for integration with Wind River Analytics and Prometheus-based observability systems.
The counters apply to both primary and backup timing sources and are available across all supported StarlingX configurations.
See: PTP Metrics Exporter.
Storage¶
Rook-Ceph Application¶
The Rook-Ceph application remains on the Ceph release and transitions its
management interface from the deprecated Ceph Manager / RESTful plugin to the
Manager / dashboard plugin API. A new app-netapp-storage application is
introduced, deploying NetApp Trident through the Trident Operator and replacing the
previous Ansible-based installation.
See:
Ceph Storage Health Monitoring: Dashboard API Replaces RESTful API¶
The ceph-manager storage health monitor has been re-designed to use the
Ceph Dashboard API for cluster health polling, replacing the former Ceph
RESTful module API (port 7999).
The Ceph Dashboard module is now enabled by default when the Rook Ceph application is installed. No manual user-override is required to enable it.
The
stx-ceph-managerDocker image (based onfm-apiandoslolibraries) has been replaced with a lightweight Python script running inside the existingceph-config-helperimage. This eliminates build dependencies that were incompatible with Debian Trixie.Health polling is performed via the Dashboard REST API over HTTPS on port 8448 (cluster-internal). The former RESTful module on port 7999 remains available but is no longer used for health monitoring.
Platform alarms (FM API) and Keystone authentication continue to use the management network interface as before.
Migration impact:
No manual action is required during upgrade. The Dashboard is enabled automatically and the health monitor switches to the new API transparently.
Previous user-overrides that explicitly enabled the Dashboard (
cephClusterSpec.dashboard.enabled: true) are harmless but no longer necessary.The Ceph RESTful module can be disabled using the following override if it is not needed for other integrations:
~(keystone_admin)$ system helm-override-update rook-ceph rook-ceph-provisioner rook-ceph \ --set mgr_restful.enabled=False ~(keystone_admin)$ system application-apply rook-ceph
See: Install Rook Ceph
Portworx Pure Storage¶
StarlingX enables Portworx software-defined storage as a persistent storage solution for both platform and containerized network function (CNF) workloads. The feature supports the use of separately licensed Portworx software with Pure Storage hardware (FlashArray for block storage and FlashBlade for file and object storage), without any Portworx proprietary software being redistributed by StarlingX.
StarlingX delivers the packages and kernel configuration updates required to deploy Portworx Enterprise from its upstream Helm chart. The Kernel Module Manager application builds and deploys the Portworx kernel module onto targeted storage workers. Both PX-STOREV1 and PX-STOREV2 storage cluster types are supported. All Pure Storage capabilities remain accessible to CNF applications.
The feature applies to all supported StarlingX configurations across both standalone and Distributed Cloud deployments.
LVM (Logical Volume Manager) Storage¶
This feature introduces LVM-based local storage on StarlingX as a
complementary option to Ceph. Both storage solutions are supported, and
operators can choose to use one or both based on their business needs. The
capability is delivered through a new lvm-csi platform application, a Container
Storage Interface driver that lets Kubernetes dynamically provision and manage
persistent volumes backed by Linux LVM logical volumes on each node’s local
disks.
The LVM CSI driver supports dynamic provisioning of both thick and thin volumes, raw block volumes, volume expansion, and snapshots and restoration of thinly provisioned volumes. Because the storage is local to a node, pods that use an LVM CSI volume are scheduled to the node where the volume resides, and in multi-node deployments a scheduler extension places pods based on available disk capacity. The platform is extended to let operators define named volume groups through the existing system inventory APIs and Deployment Manager, assign a provisioner function and provisioning capabilities (thick or thin) to each group, and manage their lifecycle including creation, deletion, and backup and restore.
See:
NetApp Ansible Playbook Installation Method¶
Users can configure an external NetApp deployment as the storage backend, after
system installation using a StarlingX-provided application netapp-storage.
This application provides standard lifecycle management through the
system application-upload, system application-apply, and
system application-remove commands, as well as configuration via Helm
overrides.
Warning
The legacy playbook and its associated roles are retained solely for
backward compatibility. New installations using the playbook method are
unsupported. If used inadvertently, uninstall the backend and reinstall
using the netapp-storage application.
See: Configure an External NetApp Deployment as the Storage Backend.
Miscellaneous¶
StarlingX Prometheus Integration¶
This feature integrates Prometheus as a platform application on StarlingX,
delivered as the new app-prometheus platform application built on the
industry-standard kube-prometheus-stack and Prometheus Operator pattern.
The application deploys:
Prometheus Operator with full CRD support (Prometheus, ServiceMonitor, PodMonitor, PrometheusRule, Alertmanager)
Prometheus Server with persistent storage and configurable data retention (default 15 days)
Optional Node Exporter, kube-state-metrics, and Alertmanager (disabled by default)
Applications register scrape targets by creating ServiceMonitor or PodMonitor resources. The Prometheus endpoint is not exposed externally by default; when external access is required, it is served over HTTPS through a Kubernetes Ingress with TLS certificates issued by cert-manager.
Component versions:
kube-prometheus-stack: v82.x
Prometheus Operator: v0.89.0
Prometheus Server: v3.10.0
Alertmanager: v0.31.1
Note
Grafana is intentionally not included due to AGPL (GNU Affero General Public License) licensing restrictions.
Known Limitations and Procedural Changes for StarlingX 13.0¶
Kubernetes Memory Manager Policies¶
The interaction between the kube-memory-mgr-policy=static
and the Topology Manager policy “restricted” can result in pods failing to be
scheduled or started even when there is sufficient memory. This
occurs due to the restrictive design of the NUMA-aware memory manager, which
prevents the same NUMA node from being used for both single and multi-NUMA
allocations.
Procedural Changes: It is important for users to understand the implications of these memory management policies and configure their systems accordingly to avoid unexpected failures.
For detailed configuration options and examples, refer to the Kubernetes documentation at https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/.
Alarm 900.024 Raised When Uploading N-1 Patch Release to the System Controller¶
When uploading an N-1 patch release to the System Controller, alarm 900.024 (Obsolete Patch) will be triggered.
This behavior is specific to the System Controller and occurs only when uploading an N-1 patch
Procedural Changes: This warning can be safely ignored.
Kubevirt Limitations¶
The following limitations apply to Kubevirt in StarlingX Release 24.09:
Limitation: Kubernetes does not provide CPU Manager detection.
Procedural Changes: Add
cpumanagerto Kubevirt:apiVersion: kubevirt.io/v1 kind: KubeVirt metadata: name: kubevirt namespace: kubevirt spec: configuration: developerConfiguration: featureGates: - LiveMigration - Macvtap - Snapshot - CPUManagerCheck the label, using the following command:
~(keystone_admin)]$ kubectl describe node | grep cpumanager where `cpumanager=true`
Limitation: Huge pages do not show up under cat /proc/meminfo inside a guest VM. Although, resources are being consumed on the host. For example, if a VM is using 4GB of Huge pages, the host shows the same 4GB of huge pages used. The huge page memory is exposed as normal memory to the VM.
Procedural Changes: You need to configure Huge pages inside the guest OS.
See the Installation Guides at Installation Guides https://docs.windriver.com/ <https://docs.windriver.com/> for more details.
Limitation: Virtual machines using Persistent Volume Claim (PVC) must have a shared ReadWriteMany (RWX) access mode to be live migrated.
Procedural Changes: Ensure PVC is created with RWX.
$ class=cephfs --access-mode=ReadWriteMany $ virtctl image-upload --pvc-name=cirros-vm-disk-test-2 --pvc-size=500Mi --storage-class=cephfs --access-mode=ReadWriteMany --image-path=/home/sysadmin/Kubevirt-GA-testing/latest-manifest/kubevirt-GA-testing/cirros-0.5.1-x86_64-disk.img --uploadproxy-url=https://10.111.54.246 -insecure
Note
Live migration is not allowed with a pod network binding of bridge interface type ()
Live migration requires ports 49152, 49153 to be available in the virt-launcher pod. If these ports are explicitly specified in the masquarade interface, live migration will not function.
For live migration with SR-IOV interface:
specify networkData: in cloudinit, so when the VM moves to another node it will not loose the IP config
specify nameserver and internal FQDNs to connect to cluster metadata server otherwise cloudinit will not work
fix the MAC address otherwise when the VM moves to another node the MAC address will change and cause a problem establishing the link
Example:
cloudInitNoCloud: networkData: | ethernets: sriov-net1: addresses: - 128.224.248.152/23 gateway: 128.224.248.1 match: macAddress: "02:00:00:00:00:01" nameservers: addresses: - 10.96.0.10 search: - default.svc.cluster.local - svc.cluster.local - cluster.local set-name: sriov-link-enabled version: 2Limitation: Snapshot CRDs and controllers are not present by default and needs to be installed on StarlingX.
Procedural Changes: To install snapshot CRDs and controllers on Kubernetes, see:
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml
Additionally, create
VolumeSnapshotClassfor Cephfs and RBD:cat <<EOF>cephfs-storageclass.yaml — apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-cephfsplugin-snapclass driver: cephfs.csi.ceph.com parameters: clusterID: 60ee9439-6204-4b11-9b02-3f2c2f0a4344 csi.storage.k8s.io/snapshotter-secret-name: ceph-pool-kube-cephfs-data csi.storage.k8s.io/snapshotter-secret-namespace: default deletionPolicy: Delete EOF
cat <<EOF>rbd-storageclass.yaml — apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-rbdplugin-snapclass driver: rbd.csi.ceph.com parameters: clusterID: 60ee9439-6204-4b11-9b02-3f2c2f0a4344 csi.storage.k8s.io/snapshotter-secret-name: ceph-pool-kube-rbd csi.storage.k8s.io/snapshotter-secret-namespace: default deletionPolicy: Delete EOF .. note:: Get the cluster ID from : ``kubectl describe sc cephfs, rbd``Limitation: Live migration is not possible when using configmap as a filesystem. Currently, virtual machine instances (VMIs) cannot be live migrated as
virtiofsdoes not support live migration.Procedural Changes: N/A.
Limitation: Live migration is not possible when a VM is using secret exposed as a filesystem. Currently, virtual machine instances cannot be live migrated since
virtiofsdoes not support live migration.Procedural Changes: N/A.
Limitation: Live migration will not work when a VM is using ServiceAccount exposed as a file system. Currently, VMIs cannot be live migrated since
virtiofsdoes not support live migration.Procedural Changes: N/A.
Docker Network Bridge Not Supported¶
The Docker Network Bridge, previously created by default, is removed and no longer supported as the default bridge IP address collides with addresses already in use.
As a result, docker can no longer be used for running containers. This impacts building docker images directly on the host.
Procedural Changes: Create a Kubernetes pod that has network access, log in to the container, and build the docker images.
Upper Case Characters in Host Names Cause Issues with Kubernetes Labelling¶
Upper case characters in host names cause issues with kubernetes labelling.
Procedural Changes: Host names should be in lower case.
Kubernetes Taint on Controllers for Standard Systems¶
In Standard systems, a Kubernetes taint is applied to controller nodes in order to prevent application pods from being scheduled on those nodes; since controllers in Standard systems are intended ONLY for platform services. If application pods MUST run on controllers, a Kubernetes toleration of the taint can be specified in the application’s pod specifications.
Procedural Changes: Customer applications that need to run on controllers on Standard systems will need to be enabled/configured for Kubernetes toleration in order to ensure the applications continue working after an upgrade. It is suggested to add the Kubernetes toleration to your application prior to upgrading.
You can specify toleration for a pod through the pod specification (PodSpec). For example:
spec:
....
template:
....
spec
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
effect: "NoSchedule"
See: Taints and Tolerations.
Application Fails After Host Lock/Unlock¶
In some situations, application may fail to apply after host lock/unlock due to previously evicted pods.
Procedural Changes: Use the kubectl delete command to delete the evicted pods and reapply the application.
Application Apply Failure if Host Reset¶
If an application apply is in progress and a host is reset it will likely fail. A re-apply attempt may be required once the host recovers and the system is stable.
Procedural Changes: Once the host recovers and the system is stable, a re-apply may be required.
Platform CPU Usage Alarms¶
Alarms may occur indicating platform cpu usage is greater than 90% if a large number of pods are configured using liveness probes that run every second.
Procedural Changes: To mitigate either reduce the frequency for the liveness probes or increase the number of platform cores.
Pods Using isolcpus¶
The isolcpus feature currently does not support allocation of thread siblings for cpu requests (i.e. physical thread +HT sibling).
Procedural Changes: For optimal results, if hyperthreading is enabled then isolcpus should be allocated in multiples of two in order to ensure that both SMT siblings are allocated to the same container.
KubeVirt VMs are Not Scheduled After Backup and Restore with wipe_ceph_osds=true¶
When a backup and restore operation is executed with wipe_ceph_osds=true,
KubeVirt VMs are not scheduled.
Procedural Changes: After a backup and restore operation with wipe_ceph_osds=true,
the VM image needs to be uploaded to the DataVolume to reinitialize the VM.
Stop VMs Before Platform Rollback¶
StarlingX rollback will fail if KubeVirt VMs are not stopped before executing the activate-rollback operation.
Procedural Changes: If the KubeVirt application is installed, stop running VMs before initiating the platform rollback. Running VMs during a rollback will cause ive migration and VM management failures after the rollback completes. For details, see Stop VMs Before Platform Rollback.
Cgroup v2 Migration and cgroup v1 Deprecation¶
Starting with StarlingX, the system uses cgroup v2 (unified hierarchy) by default. cgroup v1 is deprecated and will be removed in a future release. Before upgrading to 13.0 or deploying applications on a freshly installed 13.0 system, customers should review the “Procedural Changes” sections below and take the applicable actions to ensure cgroup v2 compatibility.
Container OOM Behavior
With cgroup v2, the out-of-memory (OOM) kill behavior for Kubernetes pods differs from cgroup v1.
memory.oom.groupis enabled by default for Kubernetes pod cgroups under cgroup v2. When a pod reaches its memory limit, the kernel terminates all processes in the pod cgroup rather than only the process that triggered the OOM condition. On cgroup v1, the OOM killer typically targeted only the highest-RSS process in the cgroup.As a result, a multi-process workload that briefly exceeded its memory limit on cgroup v1 (where only a single process was killed and the pod often continued running) may, on cgroup v2, have all of its processes terminated at once. This can cause the pod to enter
CrashLoopBackOff, and the container may terminate with exit code 137.This behavior does not lower a pod’s memory limit or affect pods that stay within their limits. It only changes what happens when a pod exceeds the memory limit it was already configured with.
Procedural Changes: For multi-process workloads, confirm that the pod’s memory
requestsandlimitsreflect the workload’s actual peak memory usage, so that a transient overshoot does not trigger a whole-pod OOM kill under cgroup v2.Applications, Monitoring Tools, and Scripts Using cgroup Paths
The cgroup filesystem layout has changed from separate per-controller hierarchies to a unified hierarchy. Applications, monitoring tools, workloads, and custom scripts that depend on cgroup v1 paths, controller directories, or file names must be updated for cgroup v2.
For example, cgroup v1 applications may read files such as
memory.usage_in_bytesorcpuacct.usagefrom a per-controller directory, locating it by appending their cgroup path from/proc/self/cgroupto a controller directory such as/sys/fs/cgroup/memory. In cgroup v2 these per-controller directories and files do not exist, so such applications may fail to obtain container-level resource information. Custom scripts that parse cgroup paths to identify pods may also need updating, as the cgroup path and naming convention have changed.On cgroup v2 systems, reading
/proc/self/cpusetfrom inside a pod returns/instead of the full kubepods cgroup path, such as/kubepods/burstable/pod<UID>/<CID>. This is expected kernel behavior due to cgroup namespace isolation on the unified hierarchy. CPU pinning usingwindriver.com/isolcpuscontinues to work correctly.Procedural Changes: Update workloads, applications, monitoring tools, and automation to use the cgroup v2 unified hierarchy and read resource information directly from
/sys/fs/cgroup/rather than hard-coding cgroup v1 paths or file names. For example, usecpu.stat,memory.current, andmemory.maxinstead of cgroup v1-specific files.If scripts or automation read
/proc/self/cpusetinside a container to determine the pod cgroup path or CPU assignment, update them to use/sys/fs/cgroup/cpuset.cpus.effective, which returns the effective CPU list (for example,2-3) and should be used to verify CPU pinning on cgroup v2.
Kubernetes Conformance Test Failure with cpu-manager-policy=static¶
On systems configured with cpu-manager-policy=static, the Kubernetes
conformance tests for “Pod InPlace Resize” of Guaranteed QoS pods fail with
the error “Resize is infeasible for Guaranteed Pods alongside CPU Manager
policy static”. This is caused by an upstream kubelet restriction (PR#128287)
that unconditionally blocks all in-place resource resizing for Guaranteed pods
on nodes with the static CPU manager policy, even for pods using fractional
milliCPU requests that only run on the shared CPU pool.
Procedural Changes: To run Kubernetes conformance tests, the test system
must be configured as a system controller, where cpu-manager-policy=none.
This allows the conformance tests to pass without requiring any kubelet patches
or alpha feature gates.
An upstream fix (KEP-5554) is targeted for a future Kubernetes release.
Kubernetes Conformance Test Failure for ServiceAccountIssuerDiscovery with ES384¶
The Kubernetes conformance test [sig-auth] ServiceAccounts
ServiceAccountIssuerDiscovery should support OIDC discovery of service account
issuer [Conformance] fails because the platform signs service account tokens
with ES384 (EC P-384), which is not in the conformance test’s supported
algorithm list (RS256, ES256). This is an upstream test limitation
(https://github.com/kubernetes/kubernetes/issues/141010); the ES384 algorithm
is fully supported by the Kubernetes API server for token signing and
validation. An upstream fix is available in Kubernetes 1.37 and later.
Procedural Changes: This conformance test failure does not affect platform functionality; service account tokens signed with ES384 are valid and accepted by the Kubernetes API server. Wind River is working with upstream Kubernetes to update the 1.36 conformance suite so that it will pass with ES384 tokens.
Limitation for Day-2 Deployment Manager operations¶
After completing Day-1 operations and initiating a
Day-2 update for the Host resource, a system config update strategy is
generated. Consequently, alarms indicating the presence of this strategy in the
system are triggered.
If a new Day-2 update is executed immediately after another update, and before the previous strategy is created, it may lead to unexpected results .
Before proceeding with Day-2 operations, use the following Procedural Changes:
Procedural Changes: Wait for any alarms related to the system config update
strategy to clear, which indicates the completion of the strategy. Once the
alarms are cleared execute a new Day-2 update using either reconfiguration or
playbook re-application to apply new changes that were not applied in the
previous update.
Subcloud Restore to N-1 Release with Additional Patches¶
Before restoring a subcloud to the latest or a specific patch level of the N-1 release, you must first upload the corresponding pre-patched ISO for that patch level.
Procedural Changes: N/A.
Subcloud install or restore to the previous release¶
If the System Controller is on StarlingX Release 12, subclouds can be deployed or restored to either StarlingX Release 11 or StarlingX Release 12.
The following operations have limited support for subclouds of the previous release:
Subcloud error reporting
The following operations are not supported for subclouds of the previous release:
Orchestrated subcloud kubernetes upgrade
Procedural Changes: N/A.
Subcloud Upgrade with Kubernetes Versions¶
Before upgrading the platform, update Kubernetes to the highest version supported by your current platform release, as the new platform version requires this specific Kubernetes version. Orchestrated Kubernetes upgrades are not supported for N-1 subclouds. For example, before upgrading to StarlingX Release 12, verify that the System Controller and all subclouds are running Kubernetes v1.32.2 (the highest version supported by StarlingX Release 11).
Procedural Changes: N/A.
Enhanced Parallel Operations for Distributed Cloud¶
No parallel operation should be performed while the System Controller is being patched.
Only one type of parallel operation can be performed at a time. For example, subcloud prestaging or upgrade orchestration should be postponed while batch subcloud deployment is still in progress.
Examples of parallel operation:
any type of
dcmanager orchestration(prestage, sw-deploy, kube-upgrade, kube-rootca-update)concurrent
dcmanager subcloud adddcmanager subcloud-backup/subcloud-backup restorewith –group option
Procedural Changes: N/A.
Subcloud Prestage Post Restore¶
Subcloud backups do not include prestaged software and container images for new release deployments. If you restore a subcloud from backup, you must prestage the subcloud again before deploying the new release.
See:
Procedural Changes: N/A.
IPsec Certificate Renewal Post Duplex/Standard Subcloud Rehoming¶
After rehoming an AIO-DX or Standard subcloud, the IPsec certificate must be renewed on all subcloud nodes and certain services must be restarted. This is required to ensure successful software updates and upgrades. Contact Wind River Customer Support at https://www.windriver.com/services#support for the Ansible playbook that automates these tasks across all applicable subclouds.
Procedural Changes: N/A.
Subcloud Deploy, Enroll or Rehome with an ECDSA system-local-ca¶
If the System Controller’s system-local-ca uses an ECDSA key, subclouds
running the previous release cannot be deployed, enrolled, or rehomed. Starting
with StarlingX Release 13.0, new installations generate an ECDSA P-384
system-local-ca by default unless an alternative configuration is specified.
Systems upgraded from earlier releases retain their existing system-local-ca
and are not affected by this change.
Procedural Changes: When bootstrapping a new System Controller that will
manage previous release subclouds, ensure system-local-ca uses an RSA key,
either by providing RSA-based system_local_ca_cert and
system_local_ca_key overrides, or, if you do not provide them, by setting
system_platform_certificate.algorithm to RSA.
On a System Controller that is already deployed, check the system-local-ca
key type. If it uses an ECDSA key, run the
Update system-local-ca or Migrate Platform Certificates to use Cert Manager
procedure to change it to RSA. Once all subclouds are upgraded to StarlingX
Release 13.0, you can change it back to ECDSA.
Unable to create Kubernetes Upgrade Strategy for Subclouds using Horizon GUI¶
When creating a Kubernetes Upgrade Strategy for a subcloud using the Horizon GUI, it fails and displays the following error:
kube upgrade pre-check: Invalid kube version(s), left: (v1.24.4), right:
(1.24.4)
Procedural Changes: Use the following steps to create the strategy:
Procedure
Create a strategy for subcloud Kubernetes upgrade using the dcmanager kube-upgrade-strategy create --to-version <version> command.
Apply the strategy using the Horizon GUI or the CLI using the command dcmanager kube-upgrade-strategy apply.
Apply a Kubernetes Upgrade Strategy using Horizon
Procedural Changes: N/A.
k8s-coredump only supports lowercase annotation¶
Creating K8s pod core dump fails when setting the
starlingx.io/core_pattern parameter in upper case characters on the pod
manifest. This results in the pod being unable to find the target directory
and fails to create the coredump file.
Procedural Changes: The starlingx.io/core_pattern parameter only accepts
lower case characters for the path and file name where the core dump is saved.
Quartzville Tools¶
The following celo64e and nvmupdate64e commands are not supported due to a known issue in Quartzville tools that crashes the host.
Procedural Change: Reboot the host using the boot screen menu.
Connectivity Issues on E825 (Granite Rapids D Integrated NIC) when Enabling SyncE TX Signal¶
Enabling the SyncE TX clock on E825 (tx_clk=synce in the PTP clock instance) results in a single, expected link flap while the NIC physical interface adjusts to drive the signal.
When platform interfaces are also configured on the E825 NIC, the link interruption may result in system alarms, and in AIO-DX or Standard deployments, may trigger a SWACT if the configuration is applied during runtime. The following alarms are raised:
400.005 / 401.005: Communication failure detected with peer
100.106 - 100.111: port / interface failed
Note
The 100.xxx alarm ID depends on the affected interface.
Warning
If the E825 link is established over a 1G link, connectivity may be completely disrupted.
Procedural Changes: Avoid configuring tx_clk=synce on E825 ports. If
SyncE TX signal is expected, it is recommended to enable it only on specific
PTP ports, avoiding the usage on all ports on each NIC.
SyncE DPLL Pin Priorities and GNSS Pins Are Fixed on Intel E810 NICs¶
On Intel E810 NIC adapters, the DPLL pin configuration is managed by the NIC firmware and cannot be modified by StarlingX. This affects SyncE frequency synchronization on E810 adapters in two ways:
The priority ordering of frequency input sources cannot be changed, including the priority of the onboard OCXO reference pin.
The GNSS input pins cannot be disconnected or reconnected through software. The firmware silently accepts the request but does not change the pin state.
On E810 adapters operating in automatic selection mode, the firmware always prefers a GNSS reference over a recovered SyncE clock when GNSS is present.
Procedural Changes: N/A
Incoming SyncE Is Not Supported on Intel E830 NICs¶
The Intel E830 NIC supports outgoing (source) SyncE only. It cannot receive or relay an incoming SyncE signal, so it cannot be used as a SyncE boundary clock (relay). Configuring an E830 to receive and relay incoming SyncE does not work: the recovered quality level is not propagated and the outgoing ESMC advertises the DNU (Do Not Use) quality level.
Procedural Changes: Use an Intel E825 NIC for configurations that receive and relay incoming SyncE quality levels.
SyncE Frequency Source Priority Relative to GNSS Is Limited on E825 (GNR-D) NICs¶
On E825 (Granite Rapids D integrated NIC) adapters, the priority of an
incoming SyncE frequency reference relative to GNSS can be configured within a
single synce4l instance by representing each reference as an external clock
source (ext_src) and setting its internal_prio parameter (a lower value
is higher priority). This allows an operator to prefer a recovered SyncE clock
over GNSS for frequency, avoiding reference flapping when GNSS is unstable.
The following constraints apply:
Priority control is only effective for references configured as
synce4lexternal clock sources. A SyncE reference must be configured as a recovered clockext_src(for example,CLK_78M125_NAC0_SYNCE0); SyncE port-level priority has no effect on E825.When two references are configured with the same
internal_prio, GNSS is selected.Coordinated, system-wide arbitration of frequency source priority across a full hybrid timing configuration — where GNSS discipline and incoming SyncE are handled through separate control paths — is not provided in this release.
Procedural Changes: To prefer SyncE over GNSS within a synce4l
instance, configure both references as ext_src sections and assign the SyncE
source a lower internal_prio value than GNSS.
Add / delete operations on pods results in errors¶
Under some circumstances, add / delete operations on pods results in error getting ClusterInformation: connection is unauthorized: Unauthorized and also results in pods staying in ContainerCreating/Terminating state. This error may also prevent users from locking a host.
Procedural Changes: If this error occurs run the following kubectl describe pod -n <namespace> <pod name> command. The following message is displayed:
error getting ClusterInformation: connection is unauthorized: Unauthorized
Limitation: There is also a known issue with the Calico CNI that may occur in rare occasions if the Calico token required for communication with the kube-apiserver becomes out of sync due to NTP skew or issues refreshing the token.
Procedural Changes: Delete the calico-node pod (causing it to automatically restart) using the following commands:
$ kubectl get pods -n kube-system --show-labels | grep calico
$ kubectl delete pods -n kube-system -l k8s-app=calico-node
Application Pods with SRIOV Interfaces¶
Application Pods with SR-IOV Interfaces require a restart-on-reboot: “true” label in their pod spec template.
Pods with SR-IOV interfaces may fail to start after a platform restore or Simplex upgrade and persist in the Container Creating state due to missing PCI address information in the CNI configuration.
Procedural Changes: Application pods that require|SRIOV| should add the label restart-on-reboot: “true” to their pod spec template metadata. All pods with this label will be deleted and recreated after system initialization, therefore all pods must be restartable and managed by a Kubernetes controller (i.e. DaemonSet, Deployment or StatefulSet) for auto recovery.
Pod Spec template example:
template:
metadata:
labels:
tier: node
app: sriovdp
restart-on-reboot: "true"
PTP O-RAN Spec Compliant Timing API Notification¶
The v2 API conforms to O-RAN.WG6.O-Cloud Notification API-v02.01 with the following exceptions, that are not supported in StarlingX
O-RAN Custom cluster names
Procedural Changes: See the PTP-notification v2 document for further details: https://docs.starlingx.io/api-ref/ptp-notification-armada-app/api_ptp_notifications_definition_v2.html
ptp4l error “timed out while polling for tx timestamp” reported for NICs using the Intel ice driver¶
NICs using the Intel® ice driver may report the following error in the ptp4l
logs, which results in a PTP port switching to FAULTY before
re-initializing.
Note
PTP ports frequently switching to FAULTY may degrade the accuracy of
the PTP timing.
ptp4l[80330.489]: timed out while polling for tx timestamp
ptp4l[80330.489]: increasing tx_timestamp_timeout may correct this issue, but it is likely caused by a driver bug
Note
This is due to a limitation with the Intel® ice driver as the driver cannot
guarantee the time interval to return the timestamp to the ptp4l user
space process which results in the occasional timeout error message.
Procedural Changes: The Procedural Changes recommended by Intel is to increase the
tx_timestamp_timeout parameter in the ptp4l config. The increased
timeout value gives more time for the ice driver to provide the timestamp to
the ptp4l user space process. Timeout values of 50ms and 700ms have been
validated. However, the user can use a different value if it is more suitable
for their system.
~(keystone_admin)]$ system ptp-instance-parameter-add <instance_name> tx_timestamp_timeout=700
~(keystone_admin)]$ system ptp-instance-apply
Note
The ptp4l timeout error log may also be caused by other underlying
issues, such as NIC port instability. Therefore, it is recommended to
confirm the NIC port is stable before adjusting the timeout values.
PTP is not supported on Broadcom 57504 NIC¶
PTP is not supported on the Broadcom 57504 NIC.
Procedural Changes: Do not configure PTP instances on the Broadcom 57504 NIC.
ptp-notification application is not supported during bootstrap¶
Deployment of
ptp-notificationduring bootstrap time is not supported due to dependencies on the system PTP configuration which is handled post-bootstrap.Procedural Changes: N/A.
The helm-chart-attribute-modify command is not supported for
ptp-notificationbecause the application consists of a single chart. Disabling the chart would renderptp-notificationnon-functional.Procedural Changes: N/A.
The ptp-notification-demo App is not a System-Managed Application¶
The ptp-notification-demo app is provided for demonstration purposes only. Therefore, it is not supported on typical platform operations such as Upgrades and Backup and Restore.
Procedural Changes: NA
Silicom TimeSync (STS) Card limitations¶
Silicom and Intel based Time Sync NICs may not be deployed on the same system due to conflicting time sync services and operations.
PTP configuration for Silicom TimeSync (STS) cards is handled separately from StarlingX host PTP configuration and may result in configuration conflicts if both are used at the same time.
The sts-silicom application provides a dedicated
phc2sysinstance which synchronizes the local system clock to the Silicom TimeSync (STS) card. Users should ensure thatphc2sysis not configured via StarlingX PTP Host Configuration when the sts-silicom application is in use.Additionally, if StarlingX PTP Host Configuration is being used in parallel for non-STS NICs, users should ensure that all
ptp4linstances do not use conflictingdomainNumbervalues.When the Silicom TimeSync (STS) card is configured in timing mode using the sts-silicom application, the card goes through an initialization process on application apply and server reboots. The ports will bounce up and down several times during the initialization process, causing network traffic disruption. Therefore, configuring the platform networks on the Silicom TimeSync (STS) card is not supported since it will cause platform instability.
Procedural Changes: N/A.
N3000 Image in the containerd cache¶
The StarlingX system without an N3000 image in the containerd cache fails to configure during a reboot cycle, and results in a failed / disabled node.
The N3000 device requires a reset early in the startup sequence. The reset is
done by the n3000-opae image. The image is automatically downloaded on bootstrap
and is expected to be in the cache to allow the reset to succeed. If the image
is not in the cache for any reason, the image cannot be downloaded as
registry.local is not up yet at this point in the startup. This will result
in the impacted host going through multiple reboot cycles and coming up in an
enabled/degraded state. To avoid this issue:
Ensure that the docker filesystem is properly engineered to avoid the image being automatically removed by the system if flagged as unused. For instructions to resize the filesystem, see Increase Controller Filesystem Storage Allotments Using the CLI
Do not manually prune the N3000 image.
Procedural Changes: Use the procedure below.
Procedure
Lock the node.
~(keystone_admin)]$ system host-lock controller-0
Pull the (N3000) required image into the
containerdcache.~(keystone_admin)]$ crictl pull registry.local:9001/docker.io/starlingx/n3000-opae:stx.8.0-v1.0.2
Unlock the node.
~(keystone_admin)]$ system host-unlock controller-0
Deploying an App using nginx controller fails with internal error after controller.name override¶
An Helm override of controller.name to the nginx-ingress-controller app may result in errors when creating ingress resources later on.
Example of Helm override:
Procedural Changes: N/A.
Unsupported Calico Operator Services¶
The Calico Operator (tigera-operator) can manage several additional Calico
services, including the APIServer, Goldmane, Whisker, and the CSI Node Driver.
These services are not validated or supported in StarlingX. Only the core
Calico networking and the optional Calico Ingress Gateway are supported.
Procedural Changes: N/A.
Calico Ingress Gateway Services Have No External IP¶
StarlingX does not provide a native LoadBalancer implementation. As a result,
Gateway services created through the Calico Ingress Gateway remain in the
EXTERNAL-IP: <pending> state and are reachable only through a NodePort.
Procedural Changes: Access the gateway services using the assigned NodePort. If an external IP is required, deploy a third-party LoadBalancer provider separately.
Gateway API CRD Memory Cost Persists After Disabling the Ingress Gateway¶
Enabling the Calico Ingress Gateway registers the Gateway API and Envoy
Gateway CRDs, which increase kube-apiserver memory usage by
approximately 281 MiB. This cost persists even if the Ingress Gateway is
later disabled, because the CRDs remain registered in the cluster.
These measurements are for the control plane only (envoy-gateway). Each
Gateway resource created will spawn an additional Envoy dataplane pod.
Procedural Changes: N/A.
Full Disk Wipe for Older Release Reinstallations¶
Once StarlingX Release 13.0 (Debian Trixie) has been installed on a host,
reinstalling an earlier release (for example, StarlingX Release 11 or 12) on the
same hardware requires a full disk wipe, including the platform_backup
partition.
StarlingX Release 13 (Trixie) uses a newer version of e2fsprogs that
enables the orphan_file and metadata_csum_seed ext4 filesystem features
when it creates the platform_backup partition. When an earlier release is
installed, the installer preserves the existing platform_backup partition,
but the older e2fsck does not recognize these features. The filesystem
check then fails and the host drops into emergency mode on boot, with an error
similar to:
[FAILED] Failed to start File System Check on /dev/disk/by-label/platform_backup
...
You are in emergency mode.
Note
This affects a fresh Release 13.0 install followed by a reinstall of an
earlier release on the same hardware. It does not affect software upgrade
or restore, which do not reformat the platform_backup partition.
Procedural Changes: Before installing an earlier release on hardware that
previously ran StarlingX Release 13.0, perform a full disk wipe (including
the platform_backup partition), then install the earlier release:
~(keystone_admin)]$ sudo wipedisk --force --include-backup
~(keystone_admin)]$ sudo sgdisk -o /dev/sda
Repeat the sgdisk command for each disk on the host (for example,
/dev/sdb, /dev/nvme0n1).
If the earlier release was installed without first wiping the disk and the host has already dropped into emergency mode, recover as follows:
Access the host console (using the BMC console or IPMI Serial over LAN (SoL)) and reboot the node.
At the GRUB menu, highlight the default boot entry and press
eto edit it. Appendinit=/bin/bashto the line beginning withlinux(orlinuxefi), then pressCtrl+X(orF10) to boot into a root shell.From the root shell, wipe the
platform_backuppartition:# mount -o remount,rw / # wipefs -a /dev/disk/by-label/platform_backup
Reboot the host and reinstall the earlier release.
Operating System Noise on Application-Isolated cores / Cyclic Test Performance Degradation¶
Current analysis indicates no confirmed application impact. Due to limited visibility into specific application requirements and tolerances, a definitive assessment cannot be made.
While internal tests show some performance degradation in cyclictest metrics, when validated against StarlingX internal benchmarks, there are no corresponding external application requirements for Cascade Lake or Ice Lake platforms that are being violated. Requirements for GNR-D continue to be met. The observed degradation is acceptable for most use cases, though this is not guaranteed.
An impact related to osnoise has been observed during the system initialization
phase. This is transient in nature, and there is currently no evidence
confirming a sustained impact on application performance. Ongoing analysis
is focused on validating duration, root cause, and ensuring stable
performance during steady-state operation.
Note
Current analysis indicates that upstream Linux changes contributed to the observed degradation.
See:
Procedural Changes: Allow up to 50 seconds after launching processes on application-isolated cores before performing time-sensitive or latency-sensitive operations to ensure OS noise has stabilized.
BPF is disabled¶
BPF cannot be used in the PREEMPT_RT/low latency kernel, due to the inherent incompatibility between PREEMPT_RT and BPF, see, https://lwn.net/Articles/802884/.
Some packages might be affected when PREEMPT_RT and BPF are used together. This includes the following, but not limited to these packages.
libpcap
libnet
dnsmasq
qemu
nmap-ncat
libv4l
elfutils
iptables
tcpdump
iproute
gdb
valgrind
kubernetes
cni
strace
mariadb
libvirt
dpdk
libteam
libseccomp
binutils
libbpf
dhcp
lldpd
containernetworking-plugins
golang
i40e
ice
Procedural Changes: Wind River recommends not to use BPF with real time kernel. If required it can still be used, for example, debugging only.
Control Group parameter¶
The control group (cgroup) parameter kmem.limit_in_bytes has been deprecated, and results in the following message in the kernel’s log buffer (dmesg) during boot-up and/or during the Ansible bootstrap procedure: “kmem.limit_in_bytes is deprecated and will be removed. Please report your use case to linux-mm@kvack.org if you depend on this functionality.” This parameter is used by a number of software packages in StarlingX, including, but not limited to, systemd, docker, containerd, libvirt etc.
Procedural Changes: NA. This is only a warning message about the future deprecation of an interface.
Subcloud Reconfig may fail due to missing inventory file¶
The dcmanager subcloud reconfig command may fail due to a missing file /var/opt/dc/ansible/<subcloud_name>_inventory.yml.
Procedural Changes: Provide the floating OAM IP address of the subcloud using the “–bootstrap-address” argument. For example:
~(keystone_admin)]$ dcmanager subcloud reconfig --sysadmin-password <password> --deploy-config deployment-config.yaml --bootstrap-address <floating_OAM_IP_address> <subcloud_name>
Increased CPU Usage After Removal of intel_idle.max_cstate=0¶
CPU usage may increase after removal of the intel_idle.max_cstate=0 setting.
The increase may be observed across multiple processes.
No performance impact is expected under active workloads, as the system does not spend time in idle states during normal operation.
See: Configurable Power Manager
Procedural Changes: Intel recommends using the intel_idle driver.
Console Session Issues during Installation¶
After bootstrap and before unlocking the controller, if the console session times
out (or the user logs out), systemd does not work properly. fm, sysinv and
mtcAgent do not initialize.
Procedural Changes: If the console times out or the user logs out between bootstrap and unlock of controller-0, then, to recover from this issue, you must re-install the ISO.
Power Metrics Application in Real Time Kernels¶
When executing Power Metrics application in Real Time kernels, the overall scheduling latency may increase due to inter-core interruptions caused by the MSR (Model-specific Registers) reading.
Due to intensive workloads the kernel may not be able to handle the MSR reading interruptions resulting in stalling data collection due to not being scheduled on the affected core.
Dell iDRAC Virtual Media Issues During Remote Installation¶
On Dell PowerEdge XR8720t systems with Intel Granite Rapids-D XCC processors and iDRAC version 1.30.10.50 (Build 25), the Dell iDRAC virtual media may experience intermittent failures during remote ISO installation.
These failures include CD mount errors, USB device resets, I/O errors on the virtual media device (sr0), and critical medium errors, which can lead to kernel panic during the initial installation, failed ostree repository synchronization, or installation failure.
Additionally, it was observed that there is an impact on the performance of the iDRAC virtual media, resulting in file transfer times longer than reference numbers during remote ISO installation.
Procedural Changes: Retry the remote ISO installation until it completes successfully.
Dell iDRAC Boot Failure on System Reboot¶
On Dell PowerEdge XR8720t systems with Intel Granite Rapids-D XCC processors and iDRAC version 1.30.10.50 (Build 25), the system may intermittently fail to boot after a reboot.
After a reboot, the boot process fails and the message “Boot Failed: starlingx” is displayed. The system then gets stuck while attempting alternative boot options. Dell iDRAC lifecycle logs may show SSD failures around the same time.
Procedural Changes: If the system fails to boot after a reboot, perform an additional reboot to allow the system to properly boot.
Dell GNSS Issues¶
In some Dell PowerEdge XR8720t system samples with Intel Granite Rapids-D XCC processors used during laboratory testing, GNSS issues were observed. These issues are suspected to be caused by mechanical problems.
Due to these problems, the GNSS module was either not detected or operating incorrectly.
Procedural Changes: Contact Dell support for hardware assistance.
Applications Using FEC VF Resources on the VFIO Driver During Upgrade¶
Applications that use FEC VF resources bound to the vfio-pci driver
should not be running during a platform or Kubernetes upgrade. A VF held
through vfio-pci is not released during the FEC PF reset on reboot,
which can stall the upgrade (for example, a deploy activate timeout with a
250.001 config-out-of-date alarm).
If the sriov-fec-daemon restarts for any reason, the application should also
be restarted so that the FEC Operator configuration can be re-applied.
Procedural Changes: Scale down, stop or remove applications holding FEC
VF resources bound to vfio-pci before an upgrade. If the
sriov-fec-daemon restarts, release and reconfigure the VF interface (or
restart the application) to recover.
See: Additional instructions for applications using VF interface in case of VFIO mode
Open File Limit Changes on Debian Trixie¶
On systems running Debian Trixie, the default maximum number of open files per process is significantly higher than in previous releases (approximately 1,073,741,816 compared to 1,048,576). Applications that automatically size internal resources based on the available file descriptor limit may allocate excessive memory and can become vulnerable to out-of-memory (OOM) terminations.
To maintain compatibility with earlier releases, the container runtime enforces an open-file limit of 1,048,576 for containers. Applications that require a higher number of open files should explicitly configure their own file descriptor limits.
Note
In a future release, the container runtime’s explicit open file limit may be removed to align with the lower upstream default, and the system-wide maximum may be lowered to ~1,048,576. Applications that require more open files than the default should set their own limit.
Procedural Changes: N/A.
Sub-NUMA Clustering (SNC) / NUMA Per Socket (NPS) Support¶
The platform supports up to 4 NUMA nodes per host, either physical or logical (via Sub-NUMA Clustering (SNC) / NUMA Per Socket (NPS)). The AMD “L3 Cache as NUMA Domain” firmware setting (ACPI SRAT (System Resource Affinity Table) L3 Cache as NUMA Domain)), that presents each Last-Level Cache (LLC/CCX) as a separate NUMA domain, is not supported.
Procedural Changes: N/A.
Software Delete Operations on the System Controller¶
On System Controllers, the software delete operation can only be performed on the previous release after all subclouds have been successfully upgraded to the target release.
Procedural Changes: Before executing software delete on the System Controller, ensure that every subcloud in the system has completed its upgrade or update process. Attempting to delete a release earlier is not supported and may impact subcloud operations.
Warning
Do not delete any previous release from the System Controller until all subclouds have been upgraded or updated.
Restart Required for containerd to Apply Config Changes for AIO-SX¶
On AIO-SX systems, certain container images were removed from the registry due to the image garbage collector and changes introduced during the Kubernetes upgrade. This may impact workloads that rely on specific image versions.
Procedural Changes: Increasing the Docker filesystem size will help retain the
image in the containerd cache. Additionally, only for AIO-SX it is
recommended to restart containerd after the Kubernetes upgrade.
BMC Password¶
To update the BMC password, the BMC must be de-provisioned and then re-provisioned
Procedural Changes: In order to update the BMC password, de-provision the BMC, and then re-provision it again with the new password.
Backup and Restore Playbook fails due to self-triggered “backup in progress”/”restore in progress” flag¶
Backup and Restore causes the Playbook to fail due to self-triggered “backup in progress” / “restore in progress” flag.
Procedural Changes: Retry the backup after manually removing the flag /etc/platform/.backup_in_progress if it has been more than 10 minutes based on the error message:
"backup has already been started less than x minutes ago.
Wait to start a new backup or manually remove the backup flag in
/etc/platform/.backup_in_progress "
For a “restore in progress” flag, reinstall and retry the restore operation.
Distributed Cloud Upgrade to StarlingX 13.0 or Later May Abort Due to a Non-Conformant Admin Endpoint Certificate¶
Starting with StarlingX 13.0, the platform is based on Debian Trixie and Python 3.13, which enables strict X.509 certificate validation by default. A server certificate must now carry an Authority Key Identifier that chains to the Subject Key Identifier of its issuing CA, as described in RFC 5280.
On a Distributed Cloud system originally installed on earlier releases,
the Distributed Cloud admin endpoint certificate chain (issued by
cert-manager in the dc-cert namespace) may not carry these identifiers,
because the cert-manager versions in those releases did not populate them.
The root CA is valid for 5 years, so this chain is long-lived and is carried
forward unchanged across subsequent upgrades. Earlier releases accepted the
certificate because Python 3.9 did not enforce the strict validation.
A system is only affected if the original chain is still in place. If the Distributed Cloud root CA was renewed after the system moved to a later release, the reissued chain carries the required identifiers and the issue is not present.
After the system controller is upgraded to 13.0 or later, calls that verify
this certificate fail with CERTIFICATE_VERIFY_FAILED ... Missing Authority Key
Identifier. An upgrade orchestrated through sw-manager can then
abort at the controller swact step (apply-reason = "swact-hosts timed out")
even though the swact completed and the cluster is healthy on the newly active
controller. Creating a new deployment strategy can fail for the same reason.
Detection: On the system controller, check whether the admin endpoint certificate chain carries the required identifiers:
# Leaf certificate
sudo openssl x509 -in /etc/ssl/private/admin-ep-cert.pem -noout -text \
| grep -q "Authority Key Identifier" \
&& echo "LEAF: has Authority Key Identifier - OK" \
|| echo "LEAF: no Authority Key Identifier - NOT OK"
# Root CA
sudo openssl x509 -in /etc/pki/ca-trust/source/anchors/dc-adminep-root-ca.crt -noout -text \
| grep -q "Subject Key Identifier" \
&& echo "ROOT: has Subject Key Identifier - OK" \
|| echo "ROOT: no Subject Key Identifier - NOT OK"
Procedural Changes: If either detection check reports NOT OK, renew the
Distributed Cloud root CA while the system is running an earlier release or later and
before it is upgraded, so that the chain is reissued with the required
identifiers. Wait for the new certificate to be written to disk, then restart
the dcmanager-audit and dccertmon processes on the system controller so
they pick up the new root CA:
kubectl -n dc-cert delete secret dc-adminep-root-ca-certificate
sleep 120
sudo pkill -9 -f dcmanager-audit
sudo sm-restart-safe service dccertmon
Wait 60 seconds for the audit to run again, then confirm the subclouds are managed and online:
dcmanager subcloud list
If the upgrade has already aborted because of this issue, renew the root CA as above (including the subcloud recovery step) and then restart the VIM on the affected controller so that it picks up the regenerated certificate:
sudo sm-restart service vim
90-Day Password Expiration Enforced for OpenStack Admin Account¶
The OpenStack admin account is subject to a default 90-day password expiration policy. If the password is not updated before the expiration date, the account is locked and OpenStack CLI operations can no longer be performed using that account.
Procedural Changes: To prevent the OpenStack admin account from password expiration, run the following command before the password expires: --os-region-name SystemController user set admin --ignore-password-expiry.
Alternatively, update the admin password before the 90-day expiration period elapses.
Recovering from a Password Expiration Lockout:
If the admin password has expired and OpenStack CLI commands can no longer be executed:
Open the Horizon Web Dashboard in a web browser.
Sign in using the admin username and the expired password.
When prompted, change the password.
Enter the current password and specify a new compliant password.
After the password is updated, OpenStack CLI access is automatically restored.
Note
The password cannot be reset through the OpenStack CLI after the account has been locked due to password expiration. Access must first be restored through the Horizon Web Dashboard.
See 90-Days Password Expiration Rule for more information.
Multiple trusted CA certificates with same Distinguished Name are not supported¶
Trusted CA (ssl_ca) certificates must have unique Distinguished Names (DNs). When a new trusted CA certificate is installed with a DN that matches an existing certificate, the system treats it as a replacement and overwrites the existing certificate.
Procedural Changes: N/A.
Kubernetes Root CA Certificates¶
The Kubernetes Root CA certificate and key are automatically generated with a default 10-year expiration, and are intended for internal use only.
For external access to kube-apiserver, the proxy (HAproxy) authenticates itself using the Rest API/GUI certificate (system-restapi-gui-certificate), which supports Intermediate CAs. The issuer (system-local-ca) can be customized at bootstrap. See Ansible Bootstrap Configurations for more information.
Procedural Changes: N/A.
External Authentication to kube-apiserver Using Client Certificates¶
SSL termination for external connections to kube-apiserver is now handled
by HAProxy, which establishes a new connection to the API server on behalf of
the external client. As a result, client certificate authentication is now
restricted to the admin user (kubernetes-admin). Token-based authentication
remains fully supported and unchanged.
Procedural Changes: N/A.
Password Expiry does not work on LDAP user login¶
On Debian, the warning message is not being displayed for Active Directory users, when a user logs in and the password is nearing expiry. Similarly, on login when a user’s password has already expired, the password change prompt is not being displayed.
Procedural Changes: It is recommended that users rely on Directory administration tools for “Windows Active Directory” servers to handle password updates, reminders and expiration. It is also recommended that passwords should be updated every 3 months.
Note
The expired password can be reset via Active Directory by IT administrators.
Upgrade activation: cert-manager does not start issuing certificates after upversion¶
During upgrade activation and upversioning, cert-manager usually takes less than a minute to be available and start issuing certificates. Occassionally, cert-manager can take more time than expected. This behavior is associated with an Open source issue. For more details see https://github.com/cert-manager/cert-manager/issues/7138#issuecomment-2422983418.
Since the cert-manager application is required, the upgrade activation will fail if the app takes too long to be available after the upversion. The following log will be displayed in /var/log/software.log
Error from server (NotFound): secrets "stx-test-cm" not found
certificate.cert-manager.io "stx-test-cm" deleted
software-controller-daemon: software_controller.py(837): INFO: 15 received
from deploy-activate with deploy-state activate-failed
software-controller-daemon: software_controller.py(870): INFO: Received
deploy state changed to DEPLOY_STATES.ACTIVATE_FAILED, agent deploy-activate
Procedural Changes: Cert-manager should recover by itself after a few minutes. If required, the following certificate used for test purposes in the upgrade activation can be created manually to ensure cert-manager is ready before reattempting the upgrade.
cat <<eof> cm_test_cert.yml
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
creationTimestamp: null
name: system-local-ca
spec:
ca:
secretName: system-local-ca
status: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
creationTimestamp: null
name: stx-test-cm
namespace: cert-manager
spec:
commonName: stx-test-cm
issuerRef:
kind: ClusterIssuer
name: system-local-ca
secretName: stx-test-cm
status: {}
eof
$ kubectl apply -f cm_test_cert.yml
$ rm cm_test_cert.yml
$ kubectl wait certificate -n cert-manager stx-test-cm --for=condition=Ready --timeout 20m
# Verify that the TLS secret associated with the cert was created, using the following:
$ kubectl get secret -n cert-manager stx-test-cm
cert-manager cm-acme-http-solver pod fails¶
On a multinode setup, when you deploy an acme issuer to issue a certificate,
the cm-acme-http-solver pod might fail and stays in “ImagePullBackOff” state
due to the following defect https://github.com/cert-manager/cert-manager/issues/5959.
Procedural Changes:
If you are using the namespace “test”, create a docker-registry secret “testkey” with local registry credentials in the “test” namespace.
~(keystone_admin)]$ kubectl create secret docker-registry testkey --docker-server=registry.local:9001 --docker-username=admin --docker-password=Password*1234 -n test
Use the secret “testkey” in the issuer spec as follows:
apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: stepca-issuer namespace: test spec: acme: server: https://test.com:8080/acme/acme/directory skipTLSVerify: true email: test@test.com privateKeySecretRef: name: stepca-issuer solvers: - http01: ingress: podTemplate: spec: imagePullSecrets: - name: testkey class: nginx
Vault application is not supported during bootstrap¶
The Vault application cannot be configured during bootstrap.
Procedural Changes:
The application must be configured after the platform nodes are unlocked /
enabled / available, a storage backend is configured, and platform-integ-apps
is applied. If Vault is to be run in HA configuration (3 vault server pods)
then at least three controller / worker nodes must be unlocked / enabled / available.
Vault application support for running on application cores¶
By default the Vault application’s pods will run on platform cores. When changing the core selection from platform cores to application cores the following additional procedure is required for the vault application.
Procedural Changes:
“If static kube-cpu-mgr-policy is selected and when overriding the label
app.starlingx.io/component for Vault namespace or pods, there are two
requirements:
The Vault server pods need to be restarted as directed by Hashicorp Vault documentation. Restart each of the standby server pods in turn, then restart the active server pod.
Ensure that sufficient hosts with worker function are available to run the Vault server pods on application cores.
See: Kubernetes CPU Manager Policies.
Restart the Vault Server pods¶
The Vault server pods do not restart automatically.
Procedural Changes: If the pods are to be re-labelled to switch execution from platform to application cores, or vice-versa, then the pods need to be restarted.
Under kubernetes the pods are restarted using the kubectl delete pod command. See, Hashicorp Vault documentation for the recommended procedure for restarting server pods in HA configuration, https://support.hashicorp.com/hc/en-us/articles/23744227055635-How-to-safely-restart-a-Vault-cluster-running-on-Kubernetes.
Ensure that sufficient hosts are available to run the server pods on application cores¶
The standard cluster with less than 3 worker nodes does not support Vault HA on the application cores. In this configuration (less than three cluster hosts with worker function):
Procedural Changes:
When setting label app.starlingx.io/component=application with the Vault app already applied in HA configuration (3 vault server pods), ensure that there are 3 nodes with worker function to support the HA configuration.
When applying Vault for the first time and with
app.starlingx.io/componentset to “application”: ensure that the server replicas is also set to 1 for non-HA configuration. The replicas for Vault server are overriden both for the Vault Helm chart and the Vault manager Helm chart:cat <<EOF > vault_overrides.yaml server: extraLabels: app.starlingx.io/component: application ha: replicas: 1 injector: extraLabels: app.starlingx.io/component: application EOF cat <<EOF > vault-manager_overrides.yaml manager: extraLabels: app.starlingx.io/component: application server: ha: replicas: 1 EOF $ system helm-override-update vault vault vault --values vault_overrides.yaml $ system helm-override-update vault vault-manager vault --values vault-manager_overrides.yaml
Platform and Kubernetes Upgrades fail if Portieris is applied¶
Platform and Kubernetes Upgrades fail if Portieris is applied.
Procedural Changes: Before performing platform or Kubernetes upgrades, you must remove the Portieris application. Once the upgrade is complete, you can reinstall Portieris as usual.
Harbor cannot be deployed during bootstrap¶
The Harbor application cannot be deployed during bootstrap due to the bootstrap deployment dependencies such as early availability of storage class.
Procedural Changes: N/A.
Windows Active Directory¶
Limitation: The Kubernetes API does not support uppercase IPv6 addresses.
Procedural Changes: The issuer_url IPv6 address must be specified as lowercase.
Limitation: The refresh token does not work.
Procedural Changes: If the token expires, manually replace the ID token. For more information, see, Configure Kubernetes Client Access.
Limitation: TLS error logs are reported in the oidc-dex container on subclouds. These logs should not have any system impact.
Procedural Changes: NA
Security Audit Logging for K8s API¶
A custom policy file can only be created at bootstrap in apiserver_extra_volumes.
If a custom policy file was configured at bootstrap, then after bootstrap the
user has the option to configure the parameter audit-policy-file to either
this custom policy file (/etc/kubernetes/my-audit-policy-file.yml) or the
default policy file /etc/kubernetes/default-audit-policy.yaml. If no
custom policy file was configured at bootstrap, then the user can only
configure the parameter audit-policy-file to the default policy file.
Only the parameter audit-policy-file is configurable after bootstrap, so
the other parameters (audit-log-path, audit-log-maxsize,
audit-log-maxage and audit-log-maxbackup) cannot be changed at
runtime.
Procedural Changes: NA
Configure TLS Version and Restricted Cipher List¶
# |
Limitation |
Affected Ports |
Reason |
Impact |
|---|---|---|---|---|
1 |
TLS 1.3 cipher selection cannot be restricted |
9001 (Docker Registry), 9002 (Registry Token Server), 443 (nginx-ingress), 30556 (OIDC Dex), 30555 (OIDC Client) |
Go’s crypto/tls runtime hardcodes all 3 TLS 1.3 cipher suites and does not allow application-level restriction. |
All 3 TLS 1.3 suites are strong — no security concern. |
2 |
Registry Token Server does not enforce platform tls-min-version |
9002 (Registry Token Server) |
Token server binary only accepts -tlscert and -tlskey flags. No CLI option or config for TLS minimum version. Source code hardcodes MinVersion: tls.VersionTLS12. |
TLS 1.2 is always accepted regardless of platform tls-min-version setting. |
3 |
lighttpd does not support TLS 1.3 cipher filtering |
8443 (Horizon) |
lighttpd’s ssl.cipher-list uses SSL_CTX_set_cipher_list() which only controls TLS 1.2. No ssl.ciphersuites directive is available. |
All 3 TLS 1.3 suites are always offered. Low impact. |
4 |
OIDC Dex bypasses HAProxy |
30556 (Dex NodePort) |
Dex is exposed as a Kubernetes NodePort — traffic goes directly to the pod, not through HAProxy. |
TLS enforcement depends on Dex’s own config (applied via helm overrides). |
5 |
Dex ciphers hardcoded in source |
30556 (Dex) |
Dex binary has 8 ciphers compiled in its Go source. No configuration mechanism to override them. |
Not configurable via platform parameters. Platform default ciphers are a subset — no compat issue. |
Procedural Changes: N/A.
System Controller and AIO-DX: Retrying an Upgrade may Fail due to Metapackage Metadata¶
In a two-controller system after upgrading from StarlingX Release 11 and rolling back to Release 11, the reattempt to upgrade may fail due to missing metapackage metadata files.
Procedural Changes: Remove the following file on both controllers before reattempting the upgrade.
$ rm -f /etc/platform/.metapackage_initial_setup
Software Delete Operations on the System Controller¶
On System Controllers, the software delete operation can only be performed on the previous release after all subclouds have been successfully upgraded to the target release.
Procedural Changes: Before executing software delete on the System Controller, ensure that every subcloud in the system has completed its upgrade or update process. Attempting to delete a release earlier is not supported and may impact subcloud operations.
Warning
Do not delete any previous release from the System Controller until all subclouds have been upgraded or updated.
ISO/SIG Upload to Central Cloud Fails when Using sudo¶
To upload a software patch or major release to the System Controller region
using the --os-region-name SystemController option, the upload command must be
authenticated with Keystone.
Procedural Changes: Do not use sudo with the --os-region-name SystemController
option. For example, avoid using sudo software upload <software release>
command.
Note
When using the -local option, you must provide the absolute path to the
release files.
Note
When using software upload commands with --os-region-name SystemController
to upload a software patch or major release to the System Controller
region, Keystone authentication is required.
Important
Do not use sudo in combination with the --os-region-name SystemController
option. For example, avoid using:
$ sudo software --os-region-name SystemController upload <software-release>
Instead, ensure the command is executed with proper authentication and without sudo.
For more information see, Upload Software Releases Using the CLI
RT Throttling Service not running after Lock/Unlock on Upgraded Subclouds¶
During the upgrade process, the USM post-upgrade script modifies systemd
presets to define which services should be automatically enabled or disabled.
As part of this process, any user-enabled custom services may be set to
“disabled” after the upgrade completes.
Since this change occurs post-upgrade, systemd will not automatically
re-enable the affected service during subsequent lock / unlock operations.
By default, USM disables custom services not explicitly listed in the systemd
presets. Since service definitions can vary between releases, USM relies on
these presets to determine enablement status per host during the upgrade.
If a custom service is not included in the presets, it will be marked as
disabled and remain inactive after lock / unlock even following a successful
upgrade.
Log message during the upgrade:
controller-0 usm-initialize[3061]: info Removed
/etc/systemd/system/multi-user.target.wants/sysctl-rt-sched-apply.service
Procedural Changes: Once the upgrade to StarlingX Release 11 completes, run the service-enable and service-start commands for all custom / user services before issuing the first lock / unlock (or reboot).
The enable and start commands for this service are required only once prior to the initial lock / unlock operation. After this step is completed, there is no further need to manually start or enable custom services, as the USM post-upgrade script has already run during the upgrade process.
sw-manager sw-deploy-strategy apply fails¶
sw-manager apply fails to apply the patch.
Note
The Procedural Changes is applicable only if the sw-manager sw-deploy-strategy
fails with the following issues.
To show the operation is in an aborted state due to a timeout, run the following command.
~(keystone_admin)]$ sw-manager sw-deploy-strategy show Strategy Patch Strategy: strategy-uuid: 2082ab5e-a387-4b6a-be23-50ac23317725 controller-apply-type: serial storage-apply-type: serial worker-apply-type: serial default-instance-action: stop-start alarm-restrictions: strict current-phase: abort current-phase-completion: 100% state: aborted apply-result: timed-out apply-reason: abort-result: success abort-reason:
If step 1 fails with ‘timed-out’ results, check if the timeout has occurred due to step-name ‘wait-alarms-clear’ using the command below.
To display results ‘wait for alarm’ that has timed out and run the following command.
~(keystone_admin)]$ sw-manager sw-deploy-strategy show --details step-name: wait-alarms-clear timeout: 2400 seconds start-date-time: 2024-03-27 19:21:15 end-date-time: 2024-03-27 20:01:16 result: timed-out
To list the 750.006 alarm, use the following command.
~(keystone_admin)]$ fm alarm-list +----------+---------------------------+--------------------+----------+---------------+ | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp | +----------+---------------------------+--------------------+----------+---------------+ | 750.006 | A configuration change | platform-integ-apps| warning | 2024-03-27T| | | requires a reapply of the | | | 19:21:15. | | | platform-k8s_application= | | | 471422 | | | integ-apps application. | | | | +----------+---------------------------+--------------------+----------+---------------+
VIM orchestrated patch strategy failed with the 900.103 alarm being triggered.
~(keystone_admin)]$ fm alarm-list +----------+---------------------------+--------------------+----------+---------------+ | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp | +----------+---------------------------+--------------------+----------+---------------+ | 900.103 | Software patch auto-apply | orchestration=sw- | critical | 2024-03-26T03T| | | failed | | | | +----------+---------------------------+--------------------+----------+---------------+
Procedural Changes - Option 1
Check the system for existing alarms using the fm alarm-list command. If the existing alarms can be ignored then use the sw-manager sw-deploy-strategy create --alarm-restrictions relaxed command to ignore any alarms during patch orchestration
If the alarm was not ignored in using the command in step 1 and the issue is seen when you encounter patch apply failure, check if alarm ‘750.006’ is present on the system.
Delete the failed strategy using the following command.
~(keystone_admin)]$ sw-manager sw-deploy-strategy delete
Create a new strategy.
~(keystone_admin)]$ sw-manager sw-deploy-strategy create --alarm-restrictions relaxed
Apply the strategy.
~(keystone_admin)]$ sw-manager sw-deploy-strategy apply
Procedural Changes - Option 2
Create a new strategy (alarm-restrictions are not relaxed).
~(keystone_admin)]$ sw-manager sw-deploy-strategy create
Apply the strategy.
~(keystone_admin)]$ sw-manager sw-deploy-strategy apply
When the
sw-deploy-strategyis in progress, and when at ‘wait-alarms-clear’ step (this can be found from ‘sw-manager patch strategy show –details | grep “step-name”’), check if alarm 750.006 is present, then execute the below steps.Execute the command.
~(keystone_admin)]$ system application-apply platform-integ-apps
This will re-apply the application and clear the alarm ‘750.006’.
If the alarm still persists after step 3, manually delete the alarm using fm alarm-delete <uuid of alarm 750.006> command.
kube-apiserver Cannot be Configured with a Minimum TLS Version of 1.3¶
In StarlingX Release 12.0, the Kubernetes kube-apiserver cannot be configured
with a minimum TLS version of 1.3. The default configuration remains TLS 1.2,
which can still be configured with a restricted set of cipher suites if
required.
The rest of the TLS 1.3 support introduced in this release is unaffected and
works as expected. Only the kube-apiserver minimum TLS 1.3 case is
impacted.
The kube-apiserver minimum TLS version cannot be set to 1.3. This limitation
is tied to the third-party docker.io/openstackhelm/ceph-config-helper:ubuntu_jammy_18.2.2-1-20240312
image. The kubectl binary bundled in that image is v1.16.2, built with
Go 1.12.10, which predates TLS 1.3 client support. This image is consumed by
platform-integ-apps, Rook Ceph, the Baremetal Migration scripts, and
Backup and Restore (BnR).
A newer image (ubuntu_jammy_18.2.2-1-20241210, kubectl v1.29.7 / Go 1.22.5) adds TLS 1.3 support; however, it also introduces additional CVEs (1 critical, 16 high). Therefore, this new image was not adopted for this release.
This limitation is tied to the third-party image dependency and is planned to be addressed in a future release, where the strategic fix is to create a self-contained image containing only the necessary binaries to be used by storage procedures, and the current ceph-config-helper will be removed.
Procedural Changes: No action is required to run with the secure default of
TLS 1.2, because it is the default value of tls-min-version for
kube-apiserver.
Host Lock Limitation on Multinode Environments with Rook Ceph¶
On multinode environments, locking all nodes containing OSDs and monitors will affect negatively Rook Ceph leading to cluster instability and availability problems.
Procedural Changes: Ensure at least one OSD is running on an unlocked host and at least one monitor is running on an unlocked host at all times. A single host can satisfy both conditions if it contains both an OSD and a monitor.
Install Failure on Rook Ceph with Replication Misconfiguration on AIO-DX¶
On AIO-DX, if the OSD distribution is not compliant with the replication factor and failure domain configured, the Rook Ceph application will fail to install.
** Procedural Changes ** For more information on how to configure OSD distribution, replication factor and failure domain, see: Failure domain and Replication Factor for deployment models Rook Ceph OSD Management Rook Ceph Tiers and Chassis
Upgrade Failure on AIO-DX Ceph Monitor After Unlock¶
During the software upgrade on a AIO-DX there is a rare issue that can affect Ceph monitors. The affected fixed monitor can get stuck in a reboot loop.
The problem and the affected monitor can be identified by finding both the alarm and a message in the logs:
Alarm sample:
200.006: <HOSTNAME> is degraded due to the failure of its 'ceph-fixed-mon (mon.<HOSTNAME>, )' process.
Message “Bad table magic number” to be found on the faulty monitor log:
$ grep "Bad table magic number" /var/log/ceph/ceph-mon.<HOSTNAME>.log
Procedural Changes: Recreate the faulty monitor:
Warning
This should only be executed on the host that has the problem with the faulty monitor.
Follow the steps to recreate the monitor:
$ sudo rm -f /etc/pmon.d/ceph-fixed-mon.conf
$ sudo /etc/init.d/ceph stop mon-${HOSTNAME}
$ sudo rm -rf /var/lib/ceph/data/ceph-${HOSTNAME}/
$ sudo ceph-mon --cluster ceph --mkfs \
--id ${HOSTNAME} \
--keyring /dev/null \
--fsid $(ceph fsid) \
--mon-data /var/lib/ceph/data/ceph-${HOSTNAME}
$ sudo /etc/init.d/ceph-init-wrapper start mon.${HOSTNAME}
$ sudo ln -s /etc/ceph/ceph-fixed-mon.conf.pmon /etc/pmon.d/ceph-fixed-mon.conf
Deployment Manager Incorrectly Handles Rook Ceph Placement Labels¶
On Day-1 operations, using Deployment Manager option deploymentScope: bootstrap, when a system is bootstrapped without defining the labels ceph-mgr-placement: enabled and ceph-mon-placement: enabled in the deployment-config.yaml on Host Custom Resource, the system completes initialization and becomes online, but the Deployment Manager may report INSYNC=false.
On Day-2 operations, using Deployment Manager option deploymentScope: principal, these labels will be deleted from the system. As a result, the Ceph cluster transitions to an Unhealthy state.
Procedural Changes:To prevent the issue, the following labels must be defined on all Rook Ceph cluster Hosts Custom Resource on the Deployment Manager YAML before bootstrap:
labels:
ceph-mgr-placement: enabled
ceph-mon-placement: enabled
For running deployments, re-apply Rook Ceph application after defining the labels on the Deployment Manager.
# Apply the updated config
$ kubectl apply -f deployment-config.yaml
# Re-apply Rook Ceph
$ system application-apply rook-ceph
Rook Ceph Application Limitation During Floating Monitor Removal¶
On a AIO-DX system, removing the floating monitor using “system controllerfs-modify ceph-float –functions=”” may lead to temporary system instability, including the possibility of uncontrolled swacts.
Procedural Changes: To avoid this issue, ensure that all finalizers are removed from the floating monitor Rook Ceph chart after its deletion, using the following command:
$ kubectl patch hr rook-ceph-floating-monitor -p '{"metadata":{"finalizers":[]}}' --type=merge
Host fails to Lock During an Upgrade¶
After adding multiple OSDs simultaneously configured in the Ceph cluster, some OSDs may remain in a configuring state even though the cluster is healthy and the OSD is deployed. This is an intermittent issue that only occurs on systems with Ceph storage backend configured with more than one OSD per host. This causes the system host-lock command to fail with the following error:
$ system host-lock controller-<id>
controller-<id> : Rejected: Can not lock a controller with storage devices
in 'configuring' state.
Since system host-lock on the controller fails and the OSD is still in
the configuring state, the upgrade is blocked from proceeding.
Procedural Changes: Use the following steps to proceed with the upgrade.
List the OSDID in the ‘configuring’ state using the following command:
$ system host-stor-list <hostname>
Identify the OSD using the following command:
$ ceph osd find osd.<OSDID>
If the OSD is found manually update the database inventory using the
stor uuid:$ sudo -u postgres psql -U postgres -d sysinv -c "UPDATE i_istor SET state='configured' WHERE uuid='<STOR_UUID>';";
Rook Ceph Application Limitation¶
After applying Rook Ceph application in an AIO-DX configuration the
800.001 - Storage Alarm Condition: HEALTH_WARN alarm may be triggered.
Procedural Changes: Restart the pod of the monitor associated with the
slow operations detected by Ceph. Check ceph -s.
Remove all OSDs on a Host on Rook Ceph¶
The procedure to remove OSDs will not work as expected when removing all
OSDs from a host. The Ceph cluster gets stuck with a HEALTH_WARN state.
Note
Use the Procedural change only if the cluster is stuck in HEALTH_WARN
state after removing all OSDs on a host.
Procedural Changes:
Check the cluster health status.
Check crushmap tree.
Remove the host(s) that are empty in the command executed before
Check the cluster health status.
Critical Alarm 800.001 After Backup and Restore on AIO-SX Systems¶
A Critical alarm 800.001 may be triggered after running the Restore Playbook. The alarm details are as follows:
~(keystone_admin)]$ fm alarm-list
+-------+----------------------------------------------------------------------+--------------------------------------+----------+---------------+
| Alarm | Reason Text | Entity ID | Severity | Time Stamp |
| ID | | | | |
+-------+----------------------------------------------------------------------+--------------------------------------+----------+---------------+
| 800. | Storage Alarm Condition: HEALTH_ERR. Please check 'ceph -s' for more | cluster= | critical | 2024-08-29T06 |
| 001 | details. | 96ebcfd4-3ea5-4114-b473-7fd0b4a65616 | | :57:59.701792 |
| | | | | |
+-------+----------------------------------------------------------------------+--------------------------------------+----------+---------------+
Procedural Changes: To clear this alarm run the following commands:
Note
Applies only to AIO-SX systems.
FS_NAME=kube-cephfs
METADATA_POOL_NAME=kube-cephfs-metadata
DATA_POOL_NAME=kube-cephfs-data
# Ensure that the Ceph MDS is stopped
sudo rm -f /etc/pmon.d/ceph-mds.conf
sudo /etc/init.d/ceph stop mds
# Recover MDS state from filesystem
ceph fs new ${FS_NAME} ${METADATA_POOL_NAME} ${DATA_POOL_NAME} --force
# Try to recover from some common errors
sudo ceph fs reset ${FS_NAME} --yes-i-really-mean-it
cephfs-journal-tool --rank=${FS_NAME}:0 event recover_dentries summary
cephfs-journal-tool --rank=${FS_NAME}:0 journal reset
cephfs-table-tool ${FS_NAME}:0 reset session
cephfs-table-tool ${FS_NAME}:0 reset snap
cephfs-table-tool ${FS_NAME}:0 reset inode
sudo /etc/init.d/ceph start mds
Intermittent Installation of Rook-Ceph on Distributed Cloud¶
While installing rook-ceph, if the installation fails, this is due to
ceph-mgr-provision not being provisioned correctly.
Procedural Changes: It is recommended to use the system application-remove rook-ceph --force to initiate rook-ceph installation.
Storage Nodes are Not Considered Part of the Kubernetes Cluster¶
When running the system kube-host-upgrade-list command the output must only display controller and worker hosts that have control-plane and kubelet components. Storage nodes do not have any of those components and so are not considered a part of the Kubernetes cluster.
Procedural Changes: Do not include Storage nodes as part of the Kubernetes upgrade.
Optimization with a Large Number of OSDs¶
As Storage nodes are not optimized, you may need to optimize your Ceph configuration for balanced operation across deployments with a high number of OSDs. This results in an alarm being generated even if the installation succeeds.
800.001 - Storage Alarm Condition: HEALTH_WARN. Please check ‘ceph -s’
Procedural Changes: To optimize your storage nodes with a large number of OSDs, it is recommended to use the following commands:
~(keystone_admin)]$ ceph osd pool set kube-rbd pg_num 256
~(keystone_admin)]$ ceph osd pool set kube-rbd pgp_num 256
Storage Nodes Recovery on Power Outage¶
Storage nodes take 10-15 minutes longer to recover in the event of a full power outage.
Procedural Changes: NA
Ceph Recovery on an AIO-DX System¶
In certain instances Ceph may not recover on an AIO-DX system, and remains in the down state when viewed using the :command”ceph -s command; for example, if an OSD comes up after a controller reboot and a swact occurs, or other possible causes for example, hardware failure of the disk or the entire host, power outage, or switch down.
Procedural Changes: There is no specific command or procedure that solves the problem for all possible causes. Each case needs to be analyzed individually to find the root cause of the problem and the solution. It is recommended to contact Customer Support at, http://www.windriver.com/support.
Restrictions on the Size of Persistent Volume Claims (PVCs)¶
There is a limitation on the size of Persistent Volume Claims (PVCs) that can be used for all StarlingX Releases.
Procedural Changes: It is recommended that all PVCs should be a minimum size of 1GB. For more information, see, https://bugs.launchpad.net/starlingx/+bug/1814595.
Failure to Clean Up platform-integ-apps Files/Helm Release¶
The System Controller does not have Ceph configured,
so the platform-integ-apps is not installed and the images are not
automatically downloaded to registry.central when upgrading the platform.
The missing images on the subclouds are:
registry.central:9001/docker.io/openstackhelm/ceph-config-helper:ubuntu_jammy_18.2.2-1-20240312
registry.central:9001/quay.io/cephcsi/cephcsi:v3.16.2
registry.central:9001/registry.k8s.io/sig-storage/csi-attacher:v4.10.0
registry.central:9001/registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.15.0registry.central:9001/registry.k8s.io/sig-storage/csi-provisioner:v6.0.0
registry.central:9001/registry.k8s.io/sig-storage/csi-resizer:v2.0.0
registry.central:9001/registry.k8s.io/sig-storage/csi-snapshotter:v8.4.0
If the System Controller does not have Ceph configured and the subclouds have Ceph configured, then the images need to be manually uploaded to the registry.central before starting the upgrade of the subclouds.
To push the images to the registry.central, run the following commands on the System Controller:
# Change the variables according to the setup
REGISTRY_PREFIX="server:port/path"
REGISTRY_USERNAME="admin"
REGISTRY_PASSWORD="password"
sudo docker login registry.local:9001 --username ${REGISTRY_USERNAME} --password ${REGISTRY_PASSWORD}
for image in\
docker.io/openstackhelm/ceph-config-helper:ubuntu_jammy_18.2.2-1-20240312
quay.io/cephcsi/cephcsi:v3.16.2
registry.central:9001/registry.k8s.io/sig-storage/csi-attacher:v4.10.0 \
registry.central:9001/registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.15.0 \
registry.central:9001/registry.k8s.io/sig-storage/csi-provisioner:v6.0.0 \
registry.central:9001/registry.k8s.io/sig-storage/csi-resizer:v2.0.0 \
registry.central:9001/registry.k8s.io/sig-storage/csi-snapshotter:v8.4.0
do
sudo docker pull ${REGISTRY_PREFIX}/${image}
sudo docker tag ${REGISTRY_PREFIX}/${image} registry.local:9001/${image}
sudo docker push registry.local:9001/${image}
done
Note
The images need to be pushed to the registry.central registry before upgrading the subclouds.
Procedural Changes: N/A.
Bare-Metal Ceph to Rook-Ceph Migration Limitations¶
Standard with Dedicated Storage does not support the In-Service method. Only Redeploy and Export-Import are available for this deployment type.
Storage Backend Deletion After Removing rook-ceph Post-Migration¶
After migrating from Ceph to Rook, removing the rook-ceph application and
then attempting to delete the ceph-rook storage backend fails with the
following error:
[sysadmin@controller-0 ~(keystone_admin)]$ system storage-backend-delete ceph-rook-store --force
Missing required backend parameter: failure_domain
Procedural Changes: To allow the removal, set the failure domain before deleting the storage backend:
Simplex system:
$ system storage-backend-modify ceph-rook-store --failure-domain osd
Multinode system:
$ system storage-backend-modify ceph-rook-store --failure-domain host
After this, the failure_domain error no longer occurs when running
system storage-backend-delete ceph-rook-store --force.
Bare-Metal Ceph to LVM Migration Limitations¶
Migration to LVM is only supported on AIO-SX deployments.
LVM (TopoLVM) does not support
ReadWriteMany(RWX) access mode. During migration, PVCs with RWX access mode are automatically changed toReadWriteOnce(RWO).In a Redeploy, PVCs are recreated without preserving the original PV
reclaimPolicy. The new PVs use the reclaim policy defined by the LVM StorageClass, since the PVCs remain pending due toWaitForFirstConsumer.
Trident CSI Registration Failure During Kubernetes Upgrade¶
On AIO-DX systems with NetApp Trident storage, the Kubernetes upgrade
may time out at the lock-hosts step. After a controller reboots for the
kubelet upgrade, the trident-node-linux pod can fail to register the CSI
driver csi.trident.netapp.io, preventing volume unmounts and leaving
StatefulSet pods stuck in Terminating state.
A trident-node-health audit Deployment is included in the
netapp-storage application to detect and recover from the most common
cause (boot timing). In rare cases involving a TLS certificate mismatch
between the Trident controller and nodes, the automated recovery is not
sufficient and manual intervention is required. The following observations
have been noticed:
trident-node-linuxpods inCrashLoopBackOfftrident-main logs show
x509: ECDSA verification failurekubectl get csinodeshows no driver on affected nodeslock-hoststimes out after 900s
Procedural Changes: Regenerate the Trident TLS certificate bundle:
$ kubectl -n trident get secret trident-csi -o yaml \
> /home/sysadmin/trident-csi.backup.yaml
$ kubectl -n trident delete secret trident-csi
$ kubectl -n trident rollout restart deployment trident-operator
$ kubectl -n trident rollout restart deployment trident-controller
$ kubectl -n trident rollout restart daemonset trident-node-linux
# Wait for all trident-node-linux pods to be 2/2 Running
$ kubectl -n trident get ds trident-node-linux
$ kubectl get csinode
Then retry the upgrade.
Deployctl Tool¶
Limitation: The deployctl tool builds a configuration definition file from a running system to generate a sample yaml file. The tool is unable to convert the information to align with the Deployment Manager naming conventions.
Procedure
You are able to specify a ‘system name’ that has upper case characters that are not supported by Deployment Manager, for example:
./deployctl build -n deployment -s deployment-config-baremin-sriov-license-1G --minimal-config
Procedural Changes: DO NOT use unsupported characters when specifying a ‘system name’ when creating the deployment config file using the deployctl tool.
You are able to specify ‘system configs’ using characters not supported by Deployment Manager, for example, to create a datanetwork with a name using unsupported chars:
system datanetwork-add sriov_net flat
Procedural Changes: DO NOT use unsupported characters for naming ‘system configs’ when creating the deployment config file using the deployctl tool.
Limitation: If the deployctl tool is used for the Central Region after the subclouds are added, the user needs to edit the generated yaml file to remove the subcloud routes.
Procedural Changes: It is recommended to review the generated file and make sure the config generated is what is intended for use in the deployment.
Incorrect Password in yaml File¶
Using the incorrect admin password in the deployment-config.yaml file will require a system reconfiguration.
Procedural Changes: Ensure the correct admin password is specified. If an incorrect password was used, update the yaml file and reconfigure the system.
Deprecated Notices in Stx 13.0¶
Kubernetes APIs¶
Kubernetes APIs that will be removed in K8s 1.27 are listed below:
See: https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-27
Containerd Schema 1 images¶
Support for Docker Schema 1 images has been disabled in containerd v2.0.
Users should ensure all container images are using Docker Schema 2 or
OCI image format. Docker Schema 1 images will no longer be supported
and may fail to pull or run.
cgroup v1 Deprecation and Transition to cgroup v2¶
Support for cgroup v1 is being deprecated in preparation for a full
transition to cgroup v2. Applications that rely on cgroup v1
should be updated to use cgroup v2 when available, to ensure forward
compatibility.
Support Lifecycle
StarlingX Release 13:
cgroup v1 is deprecated. cgroup v2 is enabled by default and supported.
Future Releases:
cgroup v1 will be fully removed. cgroup v2 is available.
Deprecation of containerd¶
Support for containerd is deprecated in StarlingX 13.0 and will be removed in a
future release. No impact to containerized applications is expected as a result
of this change.
Ingress nginx¶
The upstream Ingress nginx project has announced that no updates will be provided after March 2026. StarlingX currently uses Ingress nginx to implement the Kubernetes Ingress API and plans to continue using it until further notice. Since the Ingress API in Kubernetes is now frozen and the Gateway API is the recommended replacement, StarlingX is targeting a future release to introduce support for the Kubernetes Gateway API. Applications currently relying on the Ingress nginx API are encouraged to migrate to the Gateway API once it becomes available in StarlingX.
Legacy BIOS Support¶
Legacy BIOS boot support is deprecated in StarlingX Release 13.0. While legacy BIOS remains functional in this release, it is no longer actively maintained and may be removed in a future StarlingX release and replaced by UEFI-only boot.
UEFI is required to support key platform capabilities including Secure Boot and hardware TPM 2.0. Existing tools and environments that rely on Legacy BIOS (both physical machines and virtual machines) should be transitioned to UEFI in preparation for a future release where Legacy BIOS support may be removed altogether.
Note
No immediate action is required for StarlingX Release 13.0. However, users are encouraged to begin evaluating their UEFI readiness to avoid disruption in future upgrades.
Drivers¶
The following drivers are deprecated:
iqvlinux
intel-opae-fpga
kubernetes-power-manager¶
Intel has stopped support for the kubernetes-power-manager application. This
is still being supported by StarlingX and will be removed in a future release.
For more information, see Configurable Power Manager.
Static Configuration for Hardware Accelerator Cards¶
Static configuration for hardware accelerator cards was deprecated in StarlingX Release 10.0 and will be discontinued in future releases. Use FEC operator instead.
See Switch between Static Method Hardware Accelerator and SR-IOV FEC Operator
N3000 FPGA Firmware Update Orchestration¶
The N3000 FPGA Firmware Update Orchestration was deprecated in StarlingX Release 10.0. For more information, see N3000 FPGA Overview for more information.
show-certs.sh Script¶
The show-certs.sh script that is available when you ssh to a controller was
deprecated in StarlingX Release 11.
The new response format of the ‘system certificate-list’ RESTAPI / CLI now
provides the same information as provided by show-certs.sh.
Alarms and Event Logs Deprecation¶
- The following Alarms are deprecated:
900.101 - Software patch auto-apply in progress 900.102 - Software patch auto-apply aborting 900.103 - Software patch auto-apply failed
- The following Event Logs are deprecated:
900.111 - Software patch auto-apply start 900.112 - Software patch auto-apply inprogress 900.113 - Software patch auto-apply rejected 900.114 - Software patch auto-apply cancelled 900.115 - Software patch auto-apply failed 900.116 - Software patch auto-apply completed 900.117 - Software patch auto-apply abort 900.118 - Software patch auto-apply aborting 900.119 - Software patch auto-apply abort rejected 900.120 - Software patch auto-apply abort failed 900.121 - Software patch auto-apply aborted
These will be removed in a future release. These are superceded by the USM Software 900.2xx series of alarms and event logs.
oidc-auth Command¶
The oidc-auth command is a legacy authentication method that works only with
LDAP or WAD backends. It is deprecated in favour of oidc-login.
The oidc-login plugin provides an improved user experience when using OIDC
authentication with Kubernetes CLI (kubectl).
For migration details, see Configure Kubeconfig for OIDC Login.
Restful API Ceph Storage Health Monitoring¶
While the Restful API is still supported and available, it is no longer being used for Ceph storage health monitoring.
Bare Metal Ceph¶
Host-based Ceph is deprecated in StarlingX Release 11.0 and will be removed in a future release.
Impact: Existing deployments using Host-based Ceph will require migration prior to upgrading to StarlingX future releases.
Recommendation: For new deployments, adopt Rook-Ceph to avoid service disruption during the transition from Bare Metal Ceph to Rook.
Migration Guidance: Detailed procedures for migration during upgrades and other scenarios will be provided in a future release.
Removed in Stx 13.0¶
O-RAN O2 IMS¶
In the context of hosting a RAN Application on Cloud Platform, the O-RAN O2 Application provides and exposes the IMS and Deployement Management Service APIs of the O2 interface between the O-Cloud (Cloud Platform) and the Service Management and Orchestration (SMO), in the O-RAN Architecture.
As O2 specifications evolve, O2 IMS capability falls outside the current StarlingX design scope and is supported by Wind River Conductor. StarlingX continues to support the O2 Kubernetes Deployement Management Service (DMS) interface in alignment with O-RAN Alliance specifications.
Debian Bullseye (11) Base OS¶
Debian Bullseye (11.3+) has reached the end of its upstream LTS support and is no longer supported as a base operating system. StarlingX Release 13.0 is available only on Debian Trixie (13.5), which ships with Python 3.13 as the default Python version.
igb Drivers¶
igb (delivered by Linux v6.12.40). This has been discontinued with StarlingX 13.0 and enabled by in-tree Intel FPGA (DFL/OPAE) support. Enabled the in-tree Device Feature List (DFL) stack in both kernel-rt and kernel-std configs so that Intel FPGA / OPAE cards (e.g. Intel PAC N3000) are supported by the mainline drivers, without relying on the out-of-tree opae-intel-fpga-driver DKMS (Dynamic Kernel Module Support) package.
Ceph Bare-Metal Migration via System CLI/API¶
The system storage-backend-modify ceph-store migration_type=<type> CLI
command and corresponding Sysinv API endpoint have been removed in
StarlingX Release 13.0. All migrations must now be performed using the
Ansible playbooks.
Legacy JSON Policy Format No Longer Supported¶
Keystone and Barbican policy definition files have been migrated from
JSON to YAML format. The enforce_new_defaults option is enabled for
Barbican. Custom policy overrides using the legacy JSON format are no
longer supported and must be converted to YAML if customization is
required.
Barbican RBAC Roles Updated to Keystone-Aligned Model¶
Barbican now enforces the new RBAC policy with enforce_scope and
enforce_new_defaults enabled. The legacy Barbican-specific roles
(creator, observer, audit) are no longer effective. Access control is
now based on Keystone-aligned roles (admin, member, reader).
Hardware Updates¶
See:
Bug status¶
Fixed bugs¶
This release provides fixes for a number of defects. Refer to the StarlingX bug database to review the Release 13.0 Fixed Bugs.