StarlingX: Layered Build

Storyboard: https://storyboard.openstack.org/#!/story/2006166

This specification separates the StarlingX build into multiple layers:

  • base: A small group of packages (bash, python, etc) that are core packages common throughout the system that are useful to build separately to avoid issues with build dependencies

  • distro: Packages such as the kernel, CEPH, and other thirdparty packages that are updated and built by StarlingX, but are not part of the common services. These packages are updated less frequently, and most StarlingX developers should not need to recompile them

  • flock: StarlingX packages, such as sysinv, mtce, etc.

Problem description

The current StarlingX build is a monolithic build, building all modified distro packages, all out-of-tree packages, in addition to the StarlingX components. As a result, developers experience very long build times:

  • Barrier to adoption for new users

  • Productivity impediment to existing users

Use Cases

  1. Developers that want to update only StarlingX flock software

  2. Developers that want to update StarlingX distro software

Proposed change

Separate the StarlingX build into multiple layers, with the build output of one layer feeding into the next.

The first step will be to restructure the current repos to align with the proposed layers. This will require setting up a few new repos, with various packages moving from one repo to another (while maintaining git history). The proposed moves can be seen in the following spreadsheet: https://docs.google.com/spreadsheets/d/1zURL1UlDST8lnvw3dMlNWN6pkLX6EVF6TDBwNR9TQik/edit?usp=sharing

Three layer-specific manifests will be created in starlingx/manifest, ensuring only those repos required for the given layer are downloaded. The build tools will operate the downloaded repos. In addition, there will be a master manifest that includes all repos, allowing developers working on features impacting multiple layers to build everything. Certain repos, such as starlingx/root, would be common to all manifests.

The formal loadbuild will build each layer as needed (or as scheduled), with the build artifacts published to a public StarlingX yum repository. The flock layer build, for example, would then use the build output from the base and distro layer builds, rather than rebuilding everything.

A developer working only with the flock packages, then, would not need to build the base and distro layers, as the build would pull the required packages from the yum repository. This will greatly reduce the overall build times for developers.

The LST and yum repo config files processed by download_mirror.sh will also need to be split up and relocated along layer boundaries. New LST files will be added to enumerate packages needed from lower layers. The formal build can optionally publish LST files that would capture the outputs of a layer build.

Alternatives

This proposal allows for a layered build with minimal changes to the existing build tools, rather than adopting a new build system.

Data model impact

N/A

REST API impact

N/A

Security impact

N/A

Other end user impact

N/A

Performance Impact

N/A

Other deployer impact

N/A

Developer impact

Developers will see improvements in build times, as they will no longer need to build the entire system.

Certain packages will move to a new repo, which will impact the propagation of fixes in those packages back to the stx-2.0 branch.

Upgrade impact

N/A

Implementation

Assignee(s)

Primary assignee:

Scott Little <scott.little@windriver.com>

Other contributors:

Don Penney <don.penney@windriver.com>

Overview

In the root directory of any git:

  • centos_build_layer.cfg = A file that declares which layer will be built from this git (compiler, distro, flock). build-pkgs will build all layers named in any centos_build_layer.cfg files found.

  • centos_* as a prefix on any of the traditional lst files (centos_tarball-dl.lst, centos_rpms_centos.lst. centos_rpms_3rdparties.lst, centos_rpms_centos3rdparties.lst). We expect this to list primarily source (srpms or tarballs) rpms. Binary rpms would only be listed in the flock layer to match up with any image.inc entries.

In the stx-tools git:

  • LST and YUM files will be relocated to centos-mirror-tools/config/<os>/<layer>/. LST files are expected to only list binaries. All source entries have been moved into other gits.

  • Optionally, there will be a special layer called ‘mock’ that lists the binaries required to initialize a mock environment, something that all layers will require in order to compile (at least for centos). Without this virtual layer, these binaries would have to be listed repeatedly in each layer (compiler, distro, flock).

  • For compiler and distro layers, only binaries that satisfy a BuildRequires need be listed, plus the transitive Requires of those packages.

  • For flock layer, additional binaries must be added to satisfy the transitive Requires of packages named in an image.inc.

  • A layer may reference binaries produced by lower layers. Distro may reference compile layer binaries. Flock may reference both distro and compile layers binaries.

  • The yum configuration for a layer will be able to reference repositories produced by lower layers, via additional files in centos-mirror-tools/config/<os>/<layer>/yum.repos.d/. e.g. StarlingX_compile_layer.repo, StarlingX_distro_layer.repo

New bash library lst_utils.sh (stx-tools): Functions to produce a final version of a lst files, assembling the fragments from the various gits and from the centos-mirror-tools/config/<os>/<layer>/ directories. Once the final lst has been assembled, it is passed on for normal processing as we have had before. This library will be used by:

  • centos-mirror-tools/dl_tarball.sh

  • centos-mirror-tools/download_mirror.sh

  • toCOPY/generate-cgcs-centos-repo.sh

  • toCOPY/populate_downloads.sh

  • centos-mirror-tools/stx_mirror_scripts/*.sh (Used by cengn mirroring process)

Modifications to download scripts … dl_tarball.sh, download_mirror.sh, generate-cgcs-centos-repo.sh, populate_downloads.sh (stx-tools):

  • New command line options to specify (or override) which layers are to be downloaded. Absent this argument, it should be possible to infer a reasonable default from centos_build_layer.cfg files found in the subgits.

In the stx-root git:

  • centos_required_build_layer.cfg = A file that lists all layers required to build the iso, and a cengn url where they might be found. Listed layers will be checked against all currently downloaded subgits’ centos_build_layer.cfg files. If a required layer is not present, then additional artifacts must be downloaded from official build (cengn) at the provided url. e.g. centos_<layer>_iso_image.inc files representing the missing layers. (see below)

Modifications to package build scripts … build-pkgs, build-srpms-parallel, build-srpms-serial, build-rpms-parallel, build-rpms-serial (cgcs-root):

  • Don’t fail on absence of release-info.inc… at least for non-flock layer builds. Instead we will either not supply a platform_release rpm macro, or populate it with ‘00.00’. TBD

  • If multiple layers are present, then their packages will compile concurrently, subject only to BuildRequire constraints, as they do today. No effort will be made to ensure one layer compiles to completion before the next layer starts.

Modifications to iso build script … build-iso

  • Look for rpms in both built and dowloaded repos in some places where currently it is hard coded to look only in the built repo. e.g. A flock only build will download grub2 from the distro layer, rather than build it.

Official build and mirror (CENGN):

  • Build and publish layers independently.

  • Only the flock build publishes iso, docker images, charts etc.

  • Each layer publishes new output artifacts:

    • centos_<layer>_iso_image.inc - Merge of centos_iso_image.inc in the layer.

    • centos_stx_<layer>_rpms.lst - List of rpms built by the layer.

      i.e Addional rpms that an upper layer should download.

    • centos_stx_<layer>_srpms.lst - List of srpms built by the layer.

Repos Impacted

All repos will be impacted

Work Items

Restructure repos Implement layered builds

Dependencies

N/A

Testing

Ensure that the builds are successful, and that changes in other layers are picked up appropriately.

Sanity and regression testing on built ISO.

Documentation Impact

Build documentation will need to be updated to reflect the layered build.

References

Proposed package moves: https://docs.google.com/spreadsheets/d/1zURL1UlDST8lnvw3dMlNWN6pkLX6EVF6TDBwNR9TQik/edit?usp=sharing

History

Revisions

Release Name

Description

stx-3.0

Introduced