Build StarlingX Docker Images¶

Building StarlingX Docker images consists of three components:

StarlingX base image, which provides the operating system for the Docker image
Python wheels, providing an installation source for pip when installing python modules via loci
Docker image build, using image directives files

The images are currently built using either a Docker file or via loci (https://github.com/openstack/loci), which is an OpenStack image builder.

Environment Setup ¶

The Docker image build scripts (build-stx-base.sh, build-stx-images.sh, build-wheel-tarball.sh) and the helm chart packaging script (build-helm-charts.sh) should be run inside the StarlingX builder container, which provides the necessary build environment and Docker-in-Docker capability.

To enter the builder container:

stx shell

Once inside the builder container, the required environment variables (MY_REPO, MY_WORKSPACE, DEBIAN_SNAPSHOT, etc.) are automatically set.

For complete setup instructions on creating the build environment, refer to the StarlingX Build Guide.

Base image ¶

The StarlingX base image is the operating system image that provides the base for the StarlingX Docker images. This is built using the build-stx-base.sh tool in stx-root/build-tools/build-docker-images.

Currently, we build base images for Debian with different codenames (e.g., bullseye, trixie) and architectures (e.g., amd64) - one for use with the stable build stream images, and one for the dev build stream (bleeding edge) images:

The stable base image is configured with repo commands to point to the StarlingX build output as the source for packages to be installed in the images. After setting up the repo configuration, an apt upgrade is performed to update installed packages with versions from the StarlingX build, to try to align with the host OS as much as possible.
The dev base image does not point to the StarlingX build, as packages will come primarily from upstream sources.

The base image is passed into the StarlingX Docker image build command as an argument.

Base image tags follow the format: ${OS_CODENAME}-${OS_ARCH}-${BUILD_STREAM}-${IMAGE_VERSION}

Where IMAGE_VERSION should be:

Local builds: ${BRANCH}-${USER} (e.g., master-slittle1)
Official builds: ${BRANCH}-${TIMESTAMP} (e.g., master-20231214T070059Z)

When using --push --latest --latest-tag ${BRANCH}-latest, an additional tag is created pointing to the most recent build (e.g., master-latest), while keeping versioned tags for specific builds.

Legacy tag backward compatibility¶

For debian-bullseye-amd64 builds, legacy tags (without OS_CODENAME and OS_ARCH) are also created automatically to maintain compatibility with existing deployments:

Legacy base image tag: stx-${OS}:${BUILD_STREAM}-${VERSION}
Legacy app image tag: ${PREFIX}-${OS}-${BUILD_STREAM}-${VERSION}

For non-bullseye or non-amd64 builds, legacy tags are not created unless explicitly requested via the --os-label-legacy option on build-stx-images.sh.

Note

The examples below use DOCKER_REGISTRY to specify where to push built images. For local development and testing, you can set up a private Docker registry. See Create and Bootstrap from a Local External Private Docker Registry for instructions on creating a local registry.

Use PUSH_ARGS to optionally push images to a registry and clean up local copies after the build. Omit PUSH_ARGS to keep images locally without pushing.

Example stable build command:

# Verify build environment variables are set (typically provided by build environment)
# OS=debian
# OS_CODENAME=bullseye
# OS_ARCH=amd64

BUILD_STREAM=stable
BRANCH=master
IMAGE_VERSION=${BRANCH}-${USER}
LATEST_TAG=${BRANCH}-latest
DOCKER_USER=${USER}
DOCKER_REGISTRY=192.168.0.1:9001 # Some private registry you've setup for your testing, for example

# Optional: Push to registry and clean up local images after build
PUSH_ARGS="--push --clean"

# Optional: Keep images locally without pushing
# PUSH_ARGS=""

time $MY_REPO/build-tools/build-docker-images/build-stx-base.sh \
    --os ${OS} \
    --os-codename ${OS_CODENAME} \
    --os-arch ${OS_ARCH} \
    --stream ${BUILD_STREAM} \
    --version ${IMAGE_VERSION} \
    --user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
    --latest --latest-tag ${LATEST_TAG} \
    --repo "deb [trusted=yes] ${REPOMGR_DEPLOY_URL}deb-local-binary ${OS_CODENAME} main" \
    --repo "deb [trusted=yes] ${REPOMGR_DEPLOY_URL}deb-local-binary-containers ${OS_CODENAME} main" \
    --repo "deb [trusted=yes] ${REPOMGR_DEPLOY_URL}deb-local-build ${OS_CODENAME} main" \
    ${PUSH_ARGS}

Note

The REPOMGR_DEPLOY_URL variable is automatically set in the builder container and points to the local aptly repository.

For container builds, you need to specify multiple –repo arguments to include:

deb-local-binary: Base binary packages (common layer)
deb-local-binary-containers: Container-specific packages
deb-local-build: Your locally built StarlingX packages (highest priority)

The deb-merge-all repository used for ISO builds does not include the containers layer by default. Container builds require packages from deb-local-binary-containers that are not included in the ISO merge.

Example dev build command:

# Verify build environment variables are set (typically provided by build environment)
# OS=debian
# OS_CODENAME=bullseye
# OS_ARCH=amd64

BUILD_STREAM=dev
BRANCH=master
IMAGE_VERSION=${BRANCH}-${USER}
LATEST_TAG=${BRANCH}-latest
DOCKER_USER=${USER}
DOCKER_REGISTRY=192.168.0.1:9001 # Some private registry you've setup for your testing, for example

# Optional: Push to registry and clean up local images after build
PUSH_ARGS="--push --clean"

# Optional: Keep images locally without pushing
# PUSH_ARGS=""

time $MY_REPO/build-tools/build-docker-images/build-stx-base.sh \
    --os ${OS} \
    --os-codename ${OS_CODENAME} \
    --os-arch ${OS_ARCH} \
    --stream ${BUILD_STREAM} \
    --version ${IMAGE_VERSION} \
    --user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
    --latest --latest-tag ${LATEST_TAG} \
    --local \
    ${PUSH_ARGS}

Note

Dev builds use the –local flag to build from locally built StarlingX packages in your build environment.

If you are not making changes to any source packages that need to be installed in your designer-built images, you can use the official StarlingX built stx-base image. For example: https://hub.docker.com/r/starlingx/stx-debian/tags

Stable base image: starlingx/stx-debian:bullseye-amd64-master-stable-latest
Dev base image: starlingx/stx-debian:bullseye-amd64-master-dev-latest

Building wheels ¶

A wheel is a package format that provides a pre-built python module. We collect or build a set of python modules in wheel format and store them in a tarball, which can be passed to loci when building the StarlingX Docker images. We have two groups of wheels in the StarlingX build:

Base wheels - wheels that come from upstream source
StarlingX wheels - wheels produced by the StarlingX build

The build-wheel-tarball.sh tool in stx-root/build-tools/build-wheels is used to build and collect wheels and generate the wheels tarball. It uses two sub-tools (located in the same directory) to build and/or collect the two groups of wheels.

If you are not modifying any python modules, you can use the official StarlingX built wheels tarball:

Base wheels¶

The base wheels are built and/or collected by the build-base-wheels.sh script, which is called from build-wheel-tarball.sh. It uses a Docker file in stx-root/build-tools/build-wheels/docker to setup a wheel-builder container and runs the docker-build-wheel.sh script. This script uses a wheels.cfg file as input (eg. dev-wheels.cfg), which provides a list of wheels and build/download directives. The wheels.cfg file can specify wheel/module sources as:

pre-built wheel file to be downloaded
source git repo
source tarball
source zip

In addition, when building the “dev” wheels tarball, the build-base-wheels.sh script will pull the loci/requirements:master-${OS} image, extracting the wheels from that image to provide the initial set. This allows us to keep the dev wheels tarball at the latest upstream versions, with the exception of wheels that we explicitly build.

Example build command:

BUILD_STREAM=stable

${MY_REPO}/build-tools/build-wheels/build-wheel-tarball.sh \
    --os ${OS} \
    --os-codename ${OS_CODENAME} \
    --stream ${BUILD_STREAM}

This will produce a wheels tarball in your workspace:

${MY_WORKSPACE}/std/build-wheels-${OS}-${OS_CODENAME}-${BUILD_STREAM}/stx-${OS}-${BUILD_STREAM}-wheels.tar

StarlingX wheels¶

The StarlingX build provides support for producing python wheels during the build. For Debian, this means updating the package debian/rules file to build the wheel and package it in a -wheels package. The names of the wheels packages to be included in the tarball are listed in the wheels.inc files in the corresponding repo (ie. debian_stable_wheels.inc).

Building images ¶

The StarlingX Docker images are built using a set of image directives files, with the base image and wheels tarball as input. The images are built by the build-stx-images.sh tool, in stx-root/build-tools/build-docker-images. The build-stx-images.sh tool will search the StarlingX repos for a corresponding docker_images.inc file which contains a list of subdirectories that contain the associated image directives files, which are processed and built.

The docker_images.inc files follow these naming patterns:

${OS}_${OS_CODENAME}_${BUILD_STREAM}_docker_images.inc (preferred format) Example: debian_bullseye_stable_docker_images.inc
${OS}_${BUILD_STREAM}_docker_images.inc (backward compatibility, will be removed) Example: debian_dev_docker_images.inc

Application image tags follow the format: ${PREFIX}-${OS}-${OS_CODENAME}-${OS_ARCH}-${BUILD_STREAM}-${VERSION}.${UPDATE_VER}

The following diff provides an example of changes made to a debian/rules file to add building a wheel to a package (example for reference):

# Example showing wheel build integration in debian/rules
# Add python3-pip and python3-wheel to Build-Depends in debian/control

override_dh_auto_build:
    dh_auto_build
    python3 setup.py bdist_wheel

override_dh_auto_install:
    dh_auto_install
    mkdir -p debian/tmp/wheels
    cp dist/*.whl debian/tmp/wheels/

# Create a separate wheels package in debian/control:
# Package: <package-name>-wheels
# Architecture: all
# Description: Python wheels for <package-name>

The get-stx-wheels.sh script, called by build-wheel-tarball.sh, will gather the set of -wheels packages, defined by the corresponding wheels.inc files, and extract the wheel files, making them available to the build-wheel-tarball.sh tool.

Wheels tarball¶

The build-wheel-tarball.sh tool, after successfully calling build-base-wheels.sh and get-stx-wheels.sh, will collect the wheels built or downloaded and prep the tarball. It will also download the OpenStack requirements.txt and upper-constraints.txt files, which are used by loci when installing the python modules. The upper-constraints.txt file is modified based on the collected/built wheels, allowing us to override or append module specifications. The upper-constraints.txt file in the StarlingX wheels tarball then reflects the content of the tarball, to ensure the desired module versions are installed.

Building images ¶

The StarlingX Docker images are built using the build-stx-images.sh tool, in stx-root/build-tools/build-docker-images, using the image directives files for build instructions, with the base image and wheels as input.

Image directives files¶

The image directives files provide the build arguments necessary for building a specific image. The first required option is BUILDER, which can be either “docker” or “loci”.

“docker”¶

Images with BUILDER set to “docker” are built using a Docker file. The only other required option in the image directives file for “docker” builds is the LABEL, or image name (ie. stx-libvirt). The Docker file can use the StarlingX base image as its “FROM” by including the following at the top:

ARG BASE
FROM ${BASE}

The BASE is passed by build-stx-images.sh as a build argument.

Options supported by BUILDER=docker image directives files include:

LABEL: the image name
PROJECT: main project name
DOCKER_REPO: main project source git repo
DOCKER_REF: git branch or tag for main project source repo (default “master”)
DOCKER_PATCHES: list of patch files to apply to DOCKER_REPO, relative to the local dir
DOCKER_CONTEXT: path to build context source, relative to the local dir (default “docker”)
DOCKER_FILE: path to Dockerfile, relative to the local dir (default “docker/Dockerfile”)

Note

DOCKER_CONTEXT and DOCKER_FILE are mutually exclusive to DOCKER_REPO, DOCKER_REF and DOCKER_PATCHES.

For an example of a BUILDER=docker image, see https://opendev.org/starlingx/oidc-auth-armada-app/src/branch/master/dex/centos/dex.stable_docker_image

“loci”¶

The loci project (https://github.com/openstack/loci) provides a mechanism for building images using a python module as the main project source. The image directives file for BUILDER=loci images allows you to specify supporting python modules or packages to be installed, in addition to specifying the main project source repo and/or branch. In addition, the build-stx-images.sh supports specifying an additional customization command that is applied to the loci-built image.

Options supported by BUILDER=loci image directives files that are passed on to loci include:

LABEL: the image name
PROJECT: main project name
PROJECT_REPO: main project source git repo
PROJECT_REF: git branch or tag for main project source repo
PIP_PACKAGES: list of python modules to be installed, beyond those specified by project dependencies or requirements
DIST_PACKAGES: additional packages to be installed (eg. Debian packages from repo, configured by base image)
PROFILES: bindep profiles supported by project to be installed (eg. apache)

In addition, you can specify a bash command in the CUSTOMIZATION option, in order to do a modification on the loci-built image.

Example: stx-upstream/openstack/python-nova/centos/stx-nova.dev_docker_image

BUILDER=loci
LABEL=stx-nova
PROJECT=nova
PROJECT_REPO=https://github.com/openstack/nova.git
PIP_PACKAGES="pycrypto httplib2 pylint"
DIST_PACKAGES="openssh-clients openssh-server libvirt e2fsprogs"
PROFILES="fluent nova ceph linuxbridge openvswitch configdrive qemu apache"
CUSTOMIZATION="yum install -y openssh-clients"

In a case where the image is built without a main project source git repo, where the main project source is just coming from a wheel, you can set PROJECT to infra, and loci skips the git clone steps. For example, stx-nova-api-proxy: stx-nfv/nova-api-proxy/centos/stx-nova-api-proxy.dev_docker_image

BUILDER=loci
LABEL=stx-nova-api-proxy
# Set PROJECT=infra and PROJECT_REPO=nil because we are not cloning a repo
PROJECT=infra
PROJECT_REPO=nil
PIP_PACKAGES="api_proxy eventlet oslo.config oslo.log \
              paste PasteDeploy routes webob keystonemiddleware pylint"

Image build command¶

Example image build command, using the official StarlingX base image and wheels:

# Verify build environment variables are set (typically provided by build environment)
# OS=debian
# OS_CODENAME=bullseye
# OS_ARCH=amd64

BUILD_STREAM=stable
BRANCH=master

# Option 1: Use official StarlingX base image
DEBIAN_BASE=starlingx/stx-debian:${OS_CODENAME}-${OS_ARCH}-${BUILD_STREAM}-${BRANCH}-latest

# Option 2: Use locally built base image (must include registry prefix)
# DEBIAN_BASE=${DOCKER_REGISTRY}/${DOCKER_USER}/stx-debian:${OS_CODENAME}-${OS_ARCH}-${BUILD_STREAM}-${BRANCH}-${DOCKER_USER}

# Option 1: Use official StarlingX wheels
WHEELS=https://mirror.starlingx.windriver.com/mirror/starlingx/master/debian/bullseye/amd64/monolithic/latest_docker_image_build/outputs/wheels//stx-debian-${BUILD_STREAM}-wheels.tar

# Option 2: Use locally built wheels
# WHEELS=${MY_WORKSPACE}/std/build-wheels-${OS}-${OS_CODENAME}-${BUILD_STREAM}/stx-${OS}-${BUILD_STREAM}-wheels.tar

# Option 1: Use official StarlingX registry (Docker Hub)
DOCKER_USER=starlingx
DOCKER_REGISTRY=docker.io

# Option 2: Use local private registry
# DOCKER_USER=${USER}
# DOCKER_REGISTRY=192.168.0.1:9001 # Some private registry you've setup for your testing, for example

# Image versioning
IMAGE_VERSION=${BRANCH}-${USER}  # For local builds, or ${BRANCH}-${TIMESTAMP} for official builds
LATEST_PREFIX=${BRANCH}-latest   # Tag for most recent build

# Optional: Push to registry and clean up local images after build
PUSH_ARGS="--push --clean"

# Optional: Keep images locally without pushing
# PUSH_ARGS=""

time $MY_REPO/build-tools/build-docker-images/build-stx-images.sh \
    --os debian \
    --os-codename ${OS_CODENAME} \
    --os-arch ${OS_ARCH} \
    --stream ${BUILD_STREAM} \
    --base ${DEBIAN_BASE} \
    --wheels ${WHEELS} \
    --user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
    --version ${IMAGE_VERSION} \
    --latest --latest-prefix ${LATEST_PREFIX} \
    ${PUSH_ARGS}

If I want to build using the wheels tarball from my build, instead:

WHEELS=http://${HOSTNAME}:8088/${MY_WORKSPACE}/std/build-wheels-${OS}-${BUILD_STREAM}/stx-${OS}-${BUILD_STREAM}-wheels.tar

Note: To specify a local wheels tarball, loci needs to be able to access it via wget from a Docker container. This could mean changes to your http server and iptables rules to allow “external” access, to allow access from the Docker containers.

## Note: Verify that lighttpd is not bound to "localhost"
vi /etc/lighttpd/lighttpd.conf
# server.bind = "localhost"
systemctl restart lighttpd

## Note: You may need to add an iptables rule to allow the Docker
## containers to access the http server on your host. For example:
iptables -I INPUT 6 -i docker0 -p tcp --dport ${HOST_PORT} -m state --state NEW,ESTABLISHED -j ACCEPT

If you only want to build specify images, the build-stx-images.sh provides –only and –skip options (ie. –only stx-nova).

Testing image on running system¶

Now that you’ve built an image, you may be wondering how to test it on an already running system. First, locate the specific image of interest you just built

## Note:  You may need to be root to run Docker commands on your build system.  If so, "sudo -s"
docker images
# Locate the image of interest you just built in the output, should be at or near the top of the list, then
# save the image of interest as a compressed tarball.  It could be rather large.
docker save <image id> | gzip -9 >container_filename.tgz
# scp container_filename.tgz to the active controller, then
# ssh to active controller, then run the following instructions there:
# become root:
sudo -s
# load the transferred container compressed tarball into the image list
zcat container_filename.tgz | docker load
# find the newly loaded container image in the list and make note of it
docker images

Now that you have the container image loaded, proceed to use it for test/debug as detailed in the use container for debugging FAQ. The next step is to tag the container before pushing it, so scroll down to the point in the FAQ instructions where you are directed to tag the container and continue from there.

Incremental image updates ¶

The update-stx-image.sh utility (introduced by https://review.opendev.org/661641) provides designers with a mechanism to add minor updates to an existing image without rebuilding the entire image. This results in a small customization layer appended to the existing image, which reduces storage requirements over a full rebuild of the entire image. The tool allows for updates via:

installing or updating Python modules, via:
- wheel file
- Python module source directory
- Python module source git repository
software packages (currently supporting Debian packages)
customization script: a bash script that the updater will run that can make necessary modifications in the image that can’t be handled by updating software

Specifying Python module source¶

The –module-src command-line option (or MODULE_SRC in an update directives file) allows a designer to specify python module source from either a directory or git repository. If specifying a git repository, you can also specify a branch or tag to be fetched, as well as optionally hardcode a version number. For example:

`` –module-src \ “```https://github.com/starlingx-staging/stx-nova|stx/stein.1 <https://github.com/starlingx-staging/stx-nova%7Cstx/stein.1>`__”

This will clone the stx-nova repo and fetch/checkout the stx/stein.1 branch, installing or updating the python module in the image.

Customization script¶

You can optionally provide a customization script to make changes to the image that cannot be handled by updating software, using the –customize command-line option (or CUSTOMIZATION_SCRIPT in an update directives file). You can also provide supporting files with the –extra command-line option (or EXTRA_FILES in an update directives file), which will be accessible to the customization script in the /image-update/extras directory within the update container.

Update directives file¶

You can optionally specify an updates directives file with the –file command-line option to provide the various update directives. Options that can be set from the update directives file include:

FROM= # Specify the base image (equivalent to --from)
IMAGE_UPDATE_VER= # Specify image update version (equivalent to --version)
CUSTOMIZATION_SCRIPT= # Specify customization script (equivalent to --customize)
WHEELS= # Specify one or more wheel files (equivalent to --wheel)
DIST_PACKAGES= # Specify one or more software packages (equivalent to --pkg)
MODULE_SRC= # Specify one or more python module source locations (equivalent to --module-src)
EXTRA_FILES= # Specify one or more extra files to be accessible to customization script (equivalent to --extra)

Examples¶

## Upstream image:
STX_MIRROR_DOCKER_URL="https://mirror.starlingx.windriver.com/mirror/starlingx/master/debian/bullseye/amd64/monolithic/latest_docker_image_build/outputs/docker-images"
DOCKER_USER=mydockerid # My docker ID
DOCKER_REGISTRY=docker.io # The docker registry to use for pushing. This can be a private registry, like 192.168.2.10:9001

# For this example, I've setup a directory with files I'm using for updates,
# under ${MY_WORKSPACE}/updates:
#
# updates/pip-packages/modules/aodh - a git clone of upstream aodh, where I've made changes
# updates/pip-packages/wheels - a directory with multiple wheel files
# updates/dist-packages - a directory with multiple Debian package files
#
# Additionally, there's a basic updates/customize.sh bash script that creates
# a couple of files in /etc:
#
# #!/bin/bash
#
# # Sample
# echo example-update > /etc/example-update.txt
# find /image-update > /etc/example-update-dirlst.txt
#

# Get the latest versioned stx-aodh image tag
STX_AODH=$(curl ${STX_MIRROR_DOCKER_URL}/images-debian-${OS_CODENAME}-${OS_ARCH}-stable-versioned.lst 2>/dev/null | grep stx-aodh:)
echo ${STX_AODH}

BASE_VERSION=$(echo ${STX_AODH} | sed 's/.*://')


# For the purposes of the first few examples, each update builds on the previous one.
# So the --from argument points to the image built in the previous step (or .0 in the first case)

# Build the first update using the customization script and a couple of random files
time bash -x ${MY_REPO}/build-tools/build-docker-images/update-stx-image.sh \
    --user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
    --from ${STX_AODH} \
    --customize ${MY_WORKSPACE}/updates/customize.sh \
    --extra ${MY_REPO}/build-tools/build-docker-images/build-stx-base.sh \
    --extra ${MY_REPO}/build-tools/build-wheels/build-base-wheels.sh \
    --push \
    --update-id example.1

# To see the content of one of the files created by the customization script:
# $ docker run --rm \
#     ${DOCKER_REGISTRY}/${DOCKER_USER}/stx-aodh:${BASE_VERSION%.0}.1 \
#     cat /etc/example-update-dirlst.txt
# /image-update
# /image-update/internal-update-stx-image.sh
# /image-update/customize.sh
# /image-update/extras
# /image-update/extras/build-base-wheels.sh
# /image-update/extras/build-stx-base.sh

# To see the size of the customization layer we just added to the image
# $ docker history ${DOCKER_REGISTRY}/${DOCKER_USER}/stx-aodh:${BASE_VERSION%.0}.1 | head -2
# IMAGE                                 CREATED             CREATED BY                                      SIZE                COMMENT
# 8735dde77f9c                          3 minutes ago       bash -x -c  bash -x /image-update/internal-u…   201B


# Install/update wheels from the directory we've setup:
time bash -x ${MY_REPO}/build-tools/build-docker-images/update-stx-image.sh \
    --user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
    --from ${DOCKER_REGISTRY}/${DOCKER_USER}/stx-aodh:${BASE_VERSION%.0}.1 \
    --wheel "${MY_WORKSPACE}/updates/pip-packages/wheels/*.whl" \
    --push \
    --update-id example.2

# We can now do a diff of the piplst files from the updates and verify the change (which can also be seen in output of the tool):
# $ diff std/update-images/example.1/stx-aodh-${BASE_VERSION%.0}.1.piplst std/update-images/example.2/stx-aodh-${BASE_VERSION%.0}.2.piplst
# 14a15
# > cgcs-patch==1.0
# 130a132
# > tsconfig==1.0.0


# Update the aodh module from the dir we cloned and modified:
time bash -x ${MY_REPO}/build-tools/build-docker-images/update-stx-image.sh \
    --user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
    --from ${DOCKER_REGISTRY}/${DOCKER_USER}/stx-aodh:${BASE_VERSION%.0}.2 \
    --module-src ${MY_WORKSPACE}/updates/pip-packages/modules/aodh \
    --push \
    --update-id example.3

# We can now do a diff of the piplst files from the updates and verify the change (which can also be seen in output of the tool):
# $ diff std/update-images/example.2/stx-aodh-${BASE_VERSION%.0}.2.piplst std/update-images/example.3/stx-aodh-${BASE_VERSION%.0}.3.piplst
# 3c3
# < aodh==8.0.1.dev4
# ---
# > aodh==8.1.0.dev28


# Install/update RPMs from the dir we setup:
time bash -x ${MY_REPO}/build-tools/build-docker-images/update-stx-image.sh \
    --user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
    --from ${DOCKER_REGISTRY}/${DOCKER_USER}/stx-aodh:${BASE_VERSION%.0}.3 \
    --pkg "${MY_WORKSPACE}/updates/dist-packages/*.deb" \
    --push \
    --update-id example.4

# We can now do a diff of the piplst files from the updates and verify the change (which can also be seen in output of the tool):
# $ diff std/update-images/example.3/stx-aodh-${BASE_VERSION%.0}.3.dpkglst std/update-images/example.4/stx-aodh-${BASE_VERSION%.0}.4.dpkglst


# All of the above can also be done in a single command:
time bash -x ${MY_REPO}/build-tools/build-docker-images/update-stx-image.sh \
    --user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
    --from ${STX_AODH} \
    --wheel "${MY_WORKSPACE}/updates/pip-packages/wheels/*.whl" \
    --pkg "${MY_WORKSPACE}/updates/dist-packages/*.deb" \
    --module-src ${MY_WORKSPACE}/updates/pip-packages/modules/aodh \
    --customize ${MY_WORKSPACE}/updates/customize.sh \
    --extra ${MY_REPO}/build-tools/build-docker-images/build-stx-base.sh \
    --extra ${MY_REPO}/build-tools/build-wheels/build-base-wheels.sh



# Update stx-nova with the latest update from the stx-nova staging repo

STX_NOVA=$(curl ${STX_MIRROR_DOCKER_URL}/images-centos-stable-versioned.lst 2>/dev/null | grep stx-nova:)
echo ${STX_NOVA}

time bash -x ${MY_REPO}/build-tools/build-docker-images/update-stx-image.sh \
    --user ${DOCKER_USER} --registry ${DOCKER_REGISTRY} \
    --from ${STX_NOVA} \
    --module-src "https://github.com/starlingx-staging/stx-nova.git|stx/stein.1" \
    --update-id example.nova

build-stx-images.sh options¶

--os OS: Base OS (e.g., debian).
--os-codename CODENAME: Debian release codename (e.g., bullseye, trixie).
--os-arch ARCH: Target architecture (e.g., amd64). Autodetected via dpkg --print-architecture if not specified.
--os-label-legacy LABEL: Explicit legacy label string for backward-compatible tags. If specified, legacy tags of the form ${LABEL}-${BUILD_STREAM}-${VERSION} are created in addition to the new-format tags. For debian-bullseye-amd64 builds, legacy tags are created automatically (using debian as the label) without needing this option.
--stream STREAM: Build stream (stable or dev).
--version VERSION: Image version string.
--base IMAGE: Base image reference for building application images.
--wheels URL: URL or path to the wheels tarball.
--only IMAGE: Build only the specified image(s).
--skip IMAGE: Skip the specified image(s).
--prefix PREFIX: Prefix for image tags (default: ${OS}).
--latest: Create latest tags in addition to versioned tags.
--latest-prefix PREFIX: Prefix for latest tags.
--push: Push images to the registry after building.
--clean: Remove local images after pushing.
--user USER: Docker registry user/namespace.
--registry REGISTRY: Docker registry URL.

Building helm charts ¶

After building packages and Docker images, the next step is to build the application helm chart tarballs. The build-helm-charts.sh tool in stx-root/build-tools extracts helm charts from built .deb packages, optionally substitutes Docker image references from image record files, and produces application tarballs suitable for upload to a StarlingX system via system application-upload.

The tool requires the MY_WORKSPACE and MY_REPO environment variables to be set, and looks for built packages under ${MY_WORKSPACE}/std/.

Prerequisites¶

Packages must be built first (build-pkgs), since helm charts are extracted from .deb packages found under ${MY_WORKSPACE}/std/.
For primary application charts (e.g., stx-openstack), Docker images should be built first so that image record files are available for tag substitution.

Image record files¶

When Docker images are built with build-stx-images.sh, image record files are produced in ${MY_WORKSPACE}/std/build-images/. These files list the full image references (registry/repository/image:tag) and are used by build-helm-charts.sh to substitute image tags into helm chart manifests.

With the OS_CODENAME and OS_ARCH arguments, image record files follow the naming convention:

images-${OS}-${OS_CODENAME}-${OS_ARCH}-${BUILD_STREAM}-${BUILD_TAG}.lst

For example:

images-debian-bullseye-amd64-stable-latest.lst
images-debian-bullseye-amd64-stable-versioned.lst

Complete build examples¶

The following examples show how to build all helm charts for both non-openstack (platform) and openstack builds. These commands are run inside stx shell, where OS, OS_CODENAME, OS_ARCH, and BUILD_STREAM are already set from the [builder] section of stx.conf.

OS_LABEL=${OS}-${OS_CODENAME}-${OS_ARCH}

Non-openstack (platform) build¶

A platform build uses the default.xml manifest. Platform helm chart packages are built directly by specifying the app name and package name. The app name is derived by stripping the -helm suffix from the package name (e.g. package stx-platform-helm → app stx-platform).

Platform apps are built without image tag substitution from image record files. For each app, run:

$MY_REPO/build-tools/build-helm-charts.sh \
    --os ${OS} \
    --app <app-name> \
    --package <package-name> \
    --verbose

This produces:

${MY_WORKSPACE}/std/build-helm/stx/<app-name>-<version>.tgz

Note

The Jenkins pipeline automates helm chart discovery via the find_helm_apps() function in stx-jenkins-pipelines/scripts/build-helm-charts.sh, which scans for debian_helm.inc files and matches package names against glob patterns from build.conf (typically stx-*-helm for EXTRA_HELM_CHART_PACKAGES). Currently only the openstack-armada-app repo provides a debian_helm.inc file.

OpenStack build¶

An openstack build uses the stx-openstack.xml or flock.xml manifest, which includes the openstack-armada-app repo. The primary helm chart package is stx-openstack-helm-fluxcd, which requires image tag substitution from image record files produced by the Docker image build.

Build with both latest and versioned image tags:

# Primary chart with latest image tags
$MY_REPO/build-tools/build-helm-charts.sh \
    --os ${OS} \
    --app stx-openstack \
    --package stx-openstack-helm-fluxcd \
    --label "${OS_LABEL}-${BUILD_STREAM}-latest" \
    --image-record ${MY_WORKSPACE}/std/build-images/images-${OS_LABEL}-${BUILD_STREAM}-latest.lst \
    --ignore "helm-toolkit-[0-9]*.tgz" \
    --verbose

# Primary chart with versioned image tags
$MY_REPO/build-tools/build-helm-charts.sh \
    --os ${OS} \
    --app stx-openstack \
    --package stx-openstack-helm-fluxcd \
    --label "${OS_LABEL}-${BUILD_STREAM}-versioned" \
    --image-record ${MY_WORKSPACE}/std/build-images/images-${OS_LABEL}-${BUILD_STREAM}-versioned.lst \
    --ignore "helm-toolkit-[0-9]*.tgz" \
    --verbose

This produces tarballs such as:

${MY_WORKSPACE}/std/build-helm/stx/stx-openstack-<version>-debian-bullseye-amd64-stable-latest.tgz
${MY_WORKSPACE}/std/build-helm/stx/stx-openstack-<version>-debian-bullseye-amd64-stable-versioned.tgz

Then build any extra (non-openstack) charts without image substitution. These are specified via EXTRA_HELM_CHART_PACKAGES in build.conf and matched against package names in debian_helm.inc files:

$MY_REPO/build-tools/build-helm-charts.sh \
    --os ${OS} \
    --app <app-name> \
    --package <package-name> \
    --verbose

Note

The --image-record file contains full Docker image references (e.g., docker.io/starlingx/stx-aodh:master-debian-bullseye-amd64-stable-latest). The helm_chart_modify.py script matches by image name and replaces the entire reference in the chart’s values.yaml, regardless of what default tag was hardcoded in the source.

App discovery in Jenkins¶

The Jenkins pipeline automates the above by:

Reading debian_helm.inc files from repos under cgcs-root/ (currently only openstack-armada-app provides one). Each file lists the deb package names containing helm charts for that app.
Matching package names against glob patterns from build.conf:
- HELM_CHART_PACKAGES (e.g., stx-openstack-helm-fluxcd) — built with image tag substitution for each stream/tag combination.
- EXTRA_HELM_CHART_PACKAGES (e.g., stx-*-helm) — built without image tag substitution.
For primary charts, iterating over build streams (stable, dev) and build tags (latest, versioned), looking for matching image record files in ${MY_WORKSPACE}/std/build-images/.

build-helm-charts.sh options¶

--os OS: Base OS. Currently only debian is supported.
-a, --app NAME: Application name. Defaults to stx-openstack.
--package PACKAGE_NAME,...: Comma-separated list of top-level deb package(s) containing the helm chart(s). Defaults to <app-name>-helm.
-i, --image-record FILENAME: Path or URL to image record file(s). Multiple files can be specified with a comma-separated list or repeated arguments. Files are processed in order; later entries take higher priority for duplicate images.
-l, --label LABEL: Label appended to the application version in the tarball name.
-B, --app-version VERSION: Override the application tarball version.
--ignore CHART_PATTERN,...: Helm chart file name pattern(s) to exclude from the tarball.
-p, --patch-dependency PATCH,...: Patch dependencies to record in the tarball metadata.
--verbose: Enable verbose output.

Note

The build-helm-charts.sh script also invokes build-extra-helm-charts.sh at the end of each run, which searches for *.helm.build files in the source repos and executes their custom build commands.

Image tag substitution in helm chart package builds ¶

For helm chart packages that are built as .deb packages (via build-pkgs), the build system propagates image tag variables from stx.conf all the way into the sbuild chroot environment. This allows debian/rules files to perform sed-based substitution of image tags in helm chart values.yaml files at package build time.

Variable propagation flow¶

The variables flow through the build system as follows:

stx.conf
  → stx_control.py (reads [builder] keys, substitutes @BUILDER_*@ tokens)
    → localrc.sample → stx-localrc (exports as environment variables)
      → debbuilder.py (reads from stx-localrc)
        → debbuilder.conf (sbuild $build_environment)
          → sbuild chroot (available in debian/rules)

Available variables in sbuild¶

The following environment variables are available inside the sbuild chroot when building packages with build-pkgs:

Variable	Description	Default
`OS`	Operating system identifier	`debian`
`OS_CODENAME`	Debian release codename	`bullseye`
`OS_ARCH`	Target architecture	`amd64`
`BUILD_STREAM`	Build stream (stable or dev)	`stable`
`IMAGE_PREFIX`	Prefix for image tags (typically the branch name)	`master`
`IMAGE_SUFFIX`	Suffix for image tags (typically latest or a timestamp)	`latest`
`PLATFORM_REGISTRY`	Docker registry for platform images	`docker.io/starlingx`
`OSTREE_OSNAME`	OSTree OS name (from `[project]` section)	`debian`

Note

OSTREE_OSNAME is a pre-existing variable from the [project] section of stx.conf, not the [builder] section. It is included here because it is also propagated into the sbuild environment.

insecure_docker_reg is not propagated into the sbuild environment. It only configures the Docker daemon in the build containers to allow push/pull access to insecure (non-TLS) registries.

These variables are configured in the [builder] section of stx.conf:

[builder]
os_id = debian
os_codename = bullseye
os_arch = amd64
build_stream = stable
image_prefix = master
image_suffix = latest
platform_registry = docker.io/starlingx

To change these values for your build environment:

stx config --add builder.build_stream stable
stx config --add builder.image_prefix master
stx config --add builder.image_suffix latest
stx config --add builder.platform_registry docker.io/starlingx

Using a private registry¶

If you are running your own Docker registry for development, set platform_registry to point to it:

stx config --add builder.platform_registry 192.168.0.1:9001/myuser

If you need to push or pull images from a Docker registry that does not use TLS (or uses a self-signed certificate), set insecure_docker_reg to its host:port so that the Docker daemon inside the build containers will accept connections to it:

stx config --add builder.insecure_docker_reg 192.168.0.1:9001

insecure_docker_reg is empty by default. It can also be overridden by setting the STX_INSECURE_DOCKER_REGISTRIES environment variable before sourcing import-stx.

Note

After changing stx.conf, restart the build containers for the new values to take effect:

stx control stop
stx control start

Pre-existing build environments are automatically upgraded with default values for the new keys when stx config --upgrade is run (or implicitly on container start).

Example: using variables in debian/rules¶

A helm chart package can use these variables in its debian/rules file to substitute image tags into values.yaml at build time:

#!/usr/bin/make -f

export DH_VERBOSE = 1

IMAGE_TAG = $(IMAGE_PREFIX)-$(OS)-$(OS_CODENAME)-$(OS_ARCH)-$(BUILD_STREAM)-$(IMAGE_SUFFIX)

override_dh_auto_build:
    # Substitute image tags in helm chart values
    sed -i "s|@IMAGE_TAG@|$(IMAGE_TAG)|g" helm-charts/myapp/values.yaml
    sed -i "s|@PLATFORM_REGISTRY@|$(PLATFORM_REGISTRY)|g" helm-charts/myapp/values.yaml

%:
    dh $@

This produces image references like docker.io/starlingx/stx-myapp:master-debian-bullseye-amd64-stable-latest in the packaged helm chart.