StarlingX Userspace transition to Debian¶
Storyboard: https://storyboard.openstack.org/#!/story/list?status=active&project_group_id=86
This specification will discuss the current challenge and proposed solution with the loss of on-going support for CentOS-8 [1] and CentOS Linux in general. Red Hat and the CentOS Project have decided to move to CentOS Stream, which is a “rolling preview” of what’s next in RHEL according to this Red Hat blog [2]. Due to the nature of this type of release process there is no guarantee of getting the level of support needed to meet the needs of the StarlingX community. This specification will layout options that could be used to replace the existing CentOS-7 base OS with an alternative.
Problem description¶
Recently, the CentOS Project announced that it would stop supporting CentOS Linux 8 as of the end of 2021 with the replacement being CentOS Stream. CentOS Stream is described as a rolling preview for RHEL with a 5 year support window. It’s also been pointed out that CentOS Stream will receive critical fixes later than RHEL or via package updates, thus removing package versions that the StarlingX Community might rely on. In doing this, Red Hat has turned CentOS into a “testing” release forcing organizations looking for a “stable” release to adapt to RHEL.
Evaluation of Options¶
The StarlingX community is now in a position to make an important decision regarding the Base OS. StarlingX is currently CentOS & RPM based, there are alternatives in the GNU/Linux distribution space. The following chart shows the set of options along data that will help to make the best decision.
Distro |
Package System |
Time to EOL |
Commercial |
Kernel Version |
Low Latency Kernel |
HW Support |
Effort (t-shirt) |
Package Translation |
|---|---|---|---|---|---|---|---|---|
Fedora |
RPM |
13 month, no LTS |
no |
5.8 |
buildable |
buildable |
Med |
no |
CentOS-Stream |
RPM |
Rolling/5yrs |
no |
4.18 |
yes |
yes |
Med |
no |
RHEL |
RPM |
10 years |
yes |
4.18 |
yes |
yes |
Med |
no |
Rocky |
RPM |
Unknown |
no |
4.18 |
yes |
yes |
Med |
no |
CloudLinux |
RPM |
10 yrs |
yes |
4.18 |
yes |
yes |
Med |
no |
AlmaLinux |
RPM |
8 yrs |
no |
4.18 |
yes |
yes |
Med |
no |
Oracle Linux |
RPM |
10yrs |
yes |
4.18 |
yes |
yes |
Med |
no |
Amazon Linux |
RPM |
5 yrs |
yes |
4.14/4.19 |
buildable |
buildable |
?? |
no |
SUSE |
RPM |
10+3 |
yes |
4.12 |
yes |
yes |
Med |
limited |
openSUSE |
RPM |
10+3 |
no |
4.12 |
yes |
yes |
Med |
limited |
Debian |
DEB |
5 yrs + $$ELTS 2yrs |
no |
5.10 |
yes |
buildable |
XL |
yes |
Ubuntu |
DEB |
5+5 |
yes |
5.4/5.8 |
yes |
yes |
XL |
yes |
OpenEmbedded / Yocto Project |
any |
2 yrs |
no |
5.10 |
yes |
buildable |
XL |
yes |
Alpine Linux |
APK |
rolling/ 1.5 yrs |
no |
5.10 |
buildable |
buildable |
?? |
yes |
The Hardware Support column shows either yes or buildable, means that either the distribution contain a kernel with some of the latest hardware support, wheras buildable means we would have to backport patches as needed. Package Translation column shows if a given row requires package meta-data to be translated or converted from RPM to a new package system.
Based on the available data of the existing distros a number of these can be ruled out. Any distros that are only available from commercial vendors such as Red Hat, Canonical, Oracle, and SUSE are not viable choices. Due to the potential for additional disruptions because a vendor makes some strategic change (Red Hat/CentOS, Docker, …), future cost of support, possible license issues, or branding. While Oracle Linux is available with free redistribution rights, there is no guarentee that this will continue in the future and there is no clear upstream strategy, which could impact quality of the release. While there are some new non-commercial distributions available such as RockyLinux and AlmaLinux (CloudLinux’s free version) that are attempting to provide a similar solution as CentOS Linux did, they are young communities with no track record on how they will be able to support their community members.
While Base OS should be a non-differentiation item, the kernel might be, therefore looking at newer kernel options that supports newer hardware should be important, along with a kernel that is updated regularly. What remains is a limited number of non-commercial candidates. Of those, it is important a distribution be selected that has a strong community that can provide long term support for critical fixes and a security process to address CVEs. This would also suggest using the latest kernel available from Debian or OE/Yocto.
The remaining distros such as OE/Yocto and Alpine are not considered by operators/deployers to be enterprise / datacenter ready. OpenSuSE is primarily supported by SUSE and has similar problem of being tied to a Commercial Vendor, in addition to using an older kernel. This leaves Debian, while this would be a major change to the package and build system, the Debian community is well structured and supports a wide variety of packages and hardware. Debian has a robust community with a large userspace beyond the standard open source userspace applications, such as Kubernetes, helm, ceph and OpenStack packages.
Use Cases¶
This change will most impact StarlingX Community Developers with limited impact to Operators and End Users.
System Admin: Has to deal with distro related tools and stuff in /etc and difference with sysconfig tools, also ansible and puppet.
Operators deal with WebUI, but may deal with CLI and have some interaction on the shell.
End Users should not see any difference as they work with the OpenStack and Kubernetes interfaces
Proposed change¶
Shift Base Operating System from CentOS to Debian Stable Release.
This will entail starting a new project to create a build system based around Debian packaging, this would be in parallel to the existing CentOS as the current base will need to be supported for some time to come. Since many of the existing Debian build tools operate on a merge of upstream source with a Debian meta-data sub-directory a new set of repos would be needed to support this. Development can still occur on the current set of repos, tools can be used to merge between the two sets of repos. A new specification will be written to address the new build tools once the proof of concept is completed.
Which exact Debian build tools [3] to use is still under evaluation.
This work will require a complete switch away from CentOS once the initial release of the Debian base occurs, unless the community has the additional resources to support maintaining and testing both a CentOS and Debian base.
Debian stable releases happen every few years, Debian11 (Bullseye) is currently the in-flight release under test. The Debian team has started the version lockdown will become the stable release before the end of the year. This will line up nicely with StarlingX’s OS transition.
Alternatives¶
Given the amount of effort to pivot to a new Base OS vs staying the course with CentOS, its important to make the decision sooner rather than later if any of the above alternatives are not viable.
1) Do nothing, stay with CentOS-7, which is supported until July 2024 and see how the new RHEL derivatives such as RockyLinux or AlmaLinux play out. This would require continuing the CentOS-8 work in order to be prepared for such a transition. By doing nothing or the minimum to move to CentOS-8, the risk increases if the new communities do not work out.
2) Use CentOS Stream, same work to move to RockyLinux, but will also require continued updating of userspace as CentOS Stream moves forward and increased testing. The on-going effort to support a rolling style release would require constant up-rev of packages and/or backporting CVEs and other critical fixes. Additionally, there may be a delay getting a CVE or fix into the StarlingX packages.
3) Another option would be to put more resources into the multi-os project and complete the OpenEmbedded implementation. As mentioned above, operators and deployers of StarlingX don’t view OE as a viable base userspace. Additionally, this is still a proof of concept.
Data model impact¶
Changing the Base OS will impact a number of different aspects of the system installation and deployment since Debian does not use anaconda and has a different sysconfig interface. The ansible, puppet and config-file scripts and packages will need to be modified to account for the difference in Debian.
This will not impact any data models between the flock and associated interfaces with Kubernetes or OpenStack components.
REST API impact¶
The change to the Base OS should not impact the various REST APIs that are part of the flock.
Security impact¶
This change will update a large part of the userspace to newer versions of the package base, along with the kernel. Moving to newer version will address some CVEs and improve security, there will be an impact to the scanning tool we use to monitor and report security issues.
Other end user impact¶
The base os package names will be different for the operator will need to understand these difference when doing on-system administration.
This will not directly affect any of the python clients.
Developer impact¶
StarlingX Community Developers/Designers may need to learn and understand a new build and packaging system. Developers may also need to modify code areas that interfaces with sysadmin functions that are different between CentOS and Debian
Upgrade impact¶
The upgrade and patching mechanisms will need to be changed, this will be looked at in more details with additional specifications. The upgrade to Debian will be done as part of a normal StarlingX platform upgrade as described in [4]. Since a platform upgrade is done one host at a time, the key consideration is that the upgraded hosts (running a Debian based release) must be able to communicate with un-upgraded hosts (running a CentOS based release). This will be done by ensuring key services (e.g. DRBD, Ceph, Kubernetes) are running compatible software versions on the Debian and CentOS releases.
Implementation¶
Assignee(s)¶
- Primary assignee:
TDB
- Other contributors:
TDB
Repos Impacted¶
All repos will be impacted by this change and it will require packaging and tooling changes.
Work Items¶
Each of these will likely need additional specifications:
Build Tools and Mirror creation
Creating an installable image
Updating installation process
Testing¶
Lots of testing will be required, this will require modification to the test infrastructure in order to handle the Debian environment.
Documentation Impact¶
Any documentation that references CentOS specific command for packaging will need to be update to Debian commands. Developer documentation will need to have the workflows updated to reflect Debian workflows vs our current workflows.
References¶
Footnotes
History¶
Release Name |
Description |
|---|---|
STX-6.0 |
Introduced |
