StarlingX Build Guide

This section describes the steps for building an ISO image from a StarlingX R3.0 and earlier release.

Requirements

Hardware requirements

A workstation computer with:

  • Processor: x86_64 is the only supported architecture

  • Memory: At least 32GB RAM

  • Hard Disk: 500GB HDD

  • Network: Network adapter with active Internet connection

Software requirements

A workstation computer with:

Development environment setup

This section describes how to set up a StarlingX development system on a workstation computer. After completing these steps, you can build a StarlingX ISO image on the following Linux distribution:

  • Ubuntu 16.04 LTS 64-bit

Update your operating system

Before proceeding with the build, ensure your Ubuntu distribution is up to date. You first need to update the local database list of available packages:

sudo apt-get update

Installation requirements and dependencies

  1. Set up <user>.

    Make sure you are a non-root user with sudo privileges enabled when you build the StarlingX ISO.

    Use either your existing user or create a separate <user>:

    sudo useradd -s /bin/bash -d /home/<user> -m -G sudo <user>
sudo passwd <user>
sudo su - <user>

  2. Set up Git.

    Install the required Git packages on the Ubuntu host system:

    sudo apt-get install make git curl

    Set up your identity in git using your actual name and email address:

    git config --global user.name "Name LastName"
git config --global user.email "Email Address"

  3. Install the required Docker CE packages in the Ubuntu host system.

    See Get Docker CE for Ubuntu for more information.

    Make sure to log out and log in to add your <user> to the Docker group:

    sudo usermod -aG docker <user>

  4. Install the Android Repo Tool in the Ubuntu host system.

    Follow the steps in the Installing Repo section.

Install public SSH key

Follow these instructions on GitHub to Generate a Public SSH Key. Then upload your public key to your GitHub and Gerrit account profiles:

Install tools project

  1. Under your $HOME directory, clone the <tools> project:

    cd $HOME
git clone https://opendev.org/starlingx/tools.git

    The command above fetches the tools project from the latest master. Use the commands below to switch to the tools project for STX3.0 and STX2.0 respectively.

    git checkout remotes/origin/r/stx.3.0
git checkout remotes/origin/r/stx.2.0

  2. Navigate to the <$HOME/tools> project directory:

    cd $HOME/tools/

Prepare the base Docker image

StarlingX base Docker image handles all steps related to StarlingX ISO creation. This section describes how to customize the base Docker image building process.

Configuration values

You can customize values for the StarlingX base Docker image using a text-based configuration file named localrc:

  • HOST_PREFIX points to the directory that hosts the ‘designer’ subdirectory for source code, the ‘loadbuild’ subdirectory for the build environment, generated RPMs, and the ISO image. Best practices dictate creating the workspace directory in your $HOME directory.

  • HOST_MIRROR_DIR points to the directory that hosts the CentOS mirror repository.

localrc configuration file

Create your localrc configuration file. Make sure to set the project and the user name. For example:

# tbuilder localrc
MYUNAME=<your user name>
PROJECT=<project name>
HOST_PREFIX=$HOME/starlingx/workspace
HOST_MIRROR_DIR=$HOME/starlingx/mirror

Build the base Docker image

Once the localrc configuration file has been customized, it is time to build the base Docker image.

  1. If necessary, you might have to set http/https proxy in your Dockerfile before building the docker image:

    ENV http_proxy " http://your.actual_http_proxy.com:your_port "
ENV https_proxy " https://your.actual_https_proxy.com:your_port "
ENV ftp_proxy " http://your.actual_ftp_proxy.com:your_port "
ENV no_proxy "127.0.0.1"
RUN echo " proxy=http://your-proxy.com:port " >> /etc/yum.conf

  2. The tb.sh script automates the base Docker image build:

    ./tb.sh create

Build the CentOS mirror repository

The creation of the StarlingX ISO relies on a repository of RPM binaries, RPM sources, and tar compressed files. This section describes how to build this CentOS mirror repository.

Run building Docker container

  1. Navigate to the $HOME/tools/ project directory:

    cd $HOME/tools/

  2. Verify environment variables:

    bash tb.sh env

  3. Run the building Docker container:

    bash tb.sh run

  4. Execute the building Docker container:

    bash tb.sh exec

Download source code repositories

  1. Inside the building Docker container, start the internal environment:

    eval $(ssh-agent)
ssh-add

  2. Use the repo tool to create a local clone of the manifest Git repository based on the “master” branch:

    cd $MY_REPO_ROOT_DIR
repo init -u https://opendev.org/starlingx/manifest -m default.xml

    Optionally, specify a specific branch to clone, for example the R2.0 release branch:

    cd $MY_REPO_ROOT_DIR
repo init -u https://opendev.org/starlingx/manifest -m default.xml -b r/stx.2.0

  3. Synchronize the repository:

    repo sync -j`nproc`

Download packages

  1. Inside the Docker container, enter the following commands to download the required packages to populate the CentOS mirror repository:

    cd  $MY_REPO_ROOT_DIR/stx-tools/centos-mirror-tools && bash download_mirror.sh

  2. Monitor the download of packages until it is complete. When the download is complete, the following message appears:

    step #5: done successfully
sudo rm -rf /tmp/stx_mirror_vyPozw
IMPORTANT: The following 3 files are just bootstrap versions. Based on them, the workable images
for StarlingX could be generated by running "update-pxe-network-installer" command after "build-iso"
    - ./output/stx/CentOS/Binary/LiveOS/squashfs.img
    - ./output/stx/CentOS/Binary/images/pxeboot/initrd.img
    - ./output/stx/CentOS/Binary/images/pxeboot/vmlinuz totally 17 files are downloaded!

Verify packages

  1. Verify no missing or failed packages exist:

    cat logs/*_missing_*.log
cat logs/*_failmoved_*.log

  2. In case missing or failed packages do exist, which is usually caused by network instability (or timeout), you need to download the packages manually. Doing so assures you get all RPMs listed in rpms_3rdparties.lst/rpms_centos.lst/rpms_centos3rdparties.lst.

Packages structure

The following is a general overview of the packages structure resulting from downloading the packages:

/localdisk/designer/<user>/<project>/stx-tools/centos-mirror-tools/output
.
└── stx
    └── CentOS
        ├── Binary
        │   ├── EFI
        │   │   └── BOOT
        │   │       └── fonts
        │   ├── images
        │   │   └── pxeboot
        │   ├── isolinux
        │   ├── LiveOS
        │   ├── noarch
        │   └── x86_64
        ├── downloads
        │   ├── integrity
        │   │   ├── evm
        │   │   └── ima
        │   └── puppet
        │       └── packstack
        │           └── puppet
        │               └── modules
        └── Source

Copy CentOS mirror repository

Exit from the building Docker container. Run the following commands:

  1. Change the mirror folder owner to the current user and create CentOS folder using the commands below:

    sudo chown $USER: $HOME/starlingx/mirror
mkdir -p $HOME/starlingx/mirror/CentOS/
chmod -R ug+w $HOME/starlingx/mirror

  2. Copy the built CentOS mirror repository $HOME/starlingx/mirror/ workspace directory:

    cp -r $HOME/starlingx/workspace/localdisk/designer/<user>/<project>/stx-tools/centos-mirror-tools/output/stx $HOME/starlingx/mirror/CentOS/

Create StarlingX packages

  1. Login to the container using the command below:

    cd $HOME/tools/
./tb.sh exec

  2. Create a tarballs repository:

    ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/

    Alternatively, you can run the “populate_downloads.sh” script to copy the tarballs instead of using a symlink:

    populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/

    Outside the container

  3. Exit from the container. On the host machine, create mirror binaries:

    mkdir -p $HOME/starlingx/mirror/CentOS/stx-installer
cp $HOME/starlingx/mirror/CentOS/stx/CentOS/Binary/images/pxeboot/initrd.img $HOME/starlingx/mirror/CentOS/stx-installer/initrd.img
cp $HOME/starlingx/mirror/CentOS/stx/CentOS/Binary/images/pxeboot/vmlinuz $HOME/starlingx/mirror/CentOS/stx-installer/vmlinuz
cp $HOME/starlingx/mirror/CentOS/stx/CentOS/Binary/LiveOS/squashfs.img $HOME/starlingx/mirror/CentOS/stx-installer/squashfs.img

Build packages

  1. Enter the StarlingX container using below command:

    cd $HOME/tools/
./tb.sh exec

  2. Temporal! Build-Pkgs Errors. Be prepared to have some missing / corrupted rpm and tarball packages generated during Build the CentOS Mirror Repository, which will cause the next step to fail. If that step does fail, manually download those missing / corrupted packages.

  3. Update the symbolic links:

    cd $MY_REPO_ROOT_DIR/stx-tools/toCOPY
bash generate-centos-repo.sh /import/mirrors/CentOS/stx/CentOS/

  4. Build the packages:

    build-pkgs

  5. Optional! Generate local-repo:

    While this step is optional, it improves performance on subsequent builds. The local-repo has the dependency information that sequences the build order. To generate or update the information, you need to execute the following command after building modified or new packages.

    generate-local-repo.sh

Build StarlingX ISO

Build the image:

build-iso

Build installer

To get your StarlingX ISO ready to use, you must create the initialization files used to boot the ISO, additional controllers, and worker nodes.

NOTE: You only need this procedure during your first build and every time you upgrade the kernel.

After running “build-iso”, run:

build-pkgs --installer

This builds rpm and anaconda packages. Then run:

update-pxe-network-installer

The update-pxe-network-installer covers the steps detailed in $MY_REPO/stx/metal/installer/initrd/README. This script creates three files on /localdisk/loadbuild/<user>/<project>/pxe-network-installer/output.

new-initrd.img
new-squashfs.img
new-vmlinuz

Rename the files, as the file system is read only in the container, exit from the container and follow the commands below to rename the files:

cd $HOME/starlingx/workspace/localdisk/loadbuild/<user>/<project>/pxe-network-installer/output
sudo mv new-initrd.img initrd.img
sudo mv new-squashfs.img squashfs.img
sudo mv new-vmlinuz vmlinuz

Two ways exist for using these files:

  1. Store the files in the /import/mirrors/CentOS/stx-installer/ folder for future use. Follow the commands below to store files:

    cp -r $HOME/starlingx/workspace/localdisk/loadbuild/<user>/<project>/pxe-network-installer/output/* $HOME/starlingx/mirror/CentOS/stx-installer/

  2. Store the files in an arbitrary location and modify the $MY_REPO/stx/metal/installer/pxe-network-installer/centos/build_srpm.data file to point to these files.

Enter the StarlingX container, recreate the pxe-network-installer package, and rebuild the image using the commands below:

cd $HOME/tools/
./tb.sh exec
build-pkgs --clean pxe-network-installer
build-pkgs pxe-network-installer
build-iso

Your ISO image should be able to boot.

Additional notes

  • In order to get the first boot working, this complete procedure needs to be done. However, once the init files are created, these can be stored in a shared location where different developers can make use of them. Updating these files is not a frequent task and should be done whenever the kernel is upgraded.

  • StarlingX is in active development. Consequently, it is possible that a future version will change to a more generic solution.

Build StarlingX OpenStack application

Use the following command:

$MY_REPO_ROOT_DIR/cgcs-root/build-tools/build-helm-charts.sh

Build avoidance

The foundational principle of build avoidance is that it is faster to download the rpms than it is to build them. This typically true when the host for reference builds and the consumer are close to each other and share a high speed link. It is not practical for mirror.starlingx.windriver.com to serve as a provider of reference builds for the world. The real goal is for a corporate office to have a provider of reference builds to the designers within their corporate network.

Purpose

Build avoidance can greatly reduce build times after using repo to synchronize a local repository with an upstream source (i.e. repo sync). Build avoidance works well for designers working within a regional office. Starting from a new workspace, build-pkgs typically requires three or more hours to complete. Build avoidance can reduce this step to approximately 20 minutes.

Limitations

  • Little or no benefit for designers who refresh a pre-existing workspace at least daily (e.g. download_mirror.sh, repo sync, generate-centos-repo.sh, build-pkgs, build-iso). In these cases, an incremental build (i.e. reuse of same workspace without a build-pkgs --clean) is often just as efficient.

  • Not likely to be useful to solo designers, or teleworkers that wish to compile on using their home computers. Build avoidance downloads build artifacts from a reference build, and WAN speeds are generally too slow.

Reference build overview

  • A server in the regional office performs regular (e.g. daily) automated builds using existing methods. These builds are called reference builds.

  • The builds are time-stamped and preserved for some time (i.e. a number of weeks).

  • A build CONTEXT, which is a file produced by build-pkgs at location $MY_WORKSPACE/CONTEXT, is captured. It is a bash script that can cd to each and every Git and check out the SHA that contributed to the build.

  • For each package built, a file captures the md5sums of all the source code inputs required to build that package. These files are also produced by build-pkgs at location $MY_WORKSPACE//rpmbuild/SOURCES//srpm_reference.md5.

  • All these build products are accessible locally (e.g. a regional office) using rsync.

    Note

    Other protocols can be added later.

On the reference builds side:

  • Build contexts of all builds are collected into a common directory.

  • Context files are prefixed by the build time stamp allowing chronological traversal of the files.

On the consumer side:

  • The set of available reference build context are downloaded.

  • Traverse the set of available build contexts from newest to oldest.

    • If all SHA of all gits in a candidate reference build are also present in the local git context, stop traversal and use this reference build.

    • If selected reference build is newer than the last (if any) reference build that was downloaded, then download the selected build context, else do nothing.

Prerequisites

  • Reference build server data file

    • Data file describing your reference build server is required in the location $MY_REPO/local-build-data/build_avoidance_source. (This file is not supplied by the StarlingX gits.)

    • Required fields and hypothetical values for the data file include:

      BUILD_AVOIDANCE_DATE_FORMAT="%Y%m%d"
BUILD_AVOIDANCE_TIME_FORMAT="%H%M%S"
BUILD_AVOIDANCE_DATE_TIME_DELIM="T"
BUILD_AVOIDANCE_DATE_TIME_POSTFIX="Z"
BUILD_AVOIDANCE_DATE_UTC=0

BUILD_AVOIDANCE_FILE_TRANSFER="rsync"

BUILD_AVOIDANCE_USR="jenkins"
BUILD_AVOIDANCE_HOST="my-builder.my-company.com"
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/master"

  • Reference build server requirements

    • The reference build server should build regularly, e.g. daily.

    • The MY_WORKSPACE variable set prior to a reference build follows the format:

      TIMESTAMP=$(date +${BUILD_AVOIDANCE_DATE_FORMAT}${BUILD_AVOIDANCE_DATE_TIME_DELIM}${BUILD_AVOIDANCE_TIME_FORMAT}${BUILD_AVOIDANCE_DATE_TIME_POSTFIX})
export MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${TIMESTAMP}

    • Builds should be preserved for a useful period of time. e.g. at least two weeks.

    • The reference build server is configured to accept rsync requirements. It serves files under the BUILD_AVOIDANCE_DIR directory, which is /localdisk/loadbuild/jenkins/master in this example.

Download a selected reference build

The list of artifacts to download is captured in the datafile $MY_REPO/build-data/build_avoidance_source.

The following paths are relative to $MY_WORKSPACE/$BUILD_TYPE

BUILD_AVOIDANCE_SRPM_DIRECTORIES="inputs srpm_assemble rpmbuild/SRPMS rpmbuild/SOURCES"
BUILD_AVOIDANCE_SRPM_FILES=""
BUILD_AVOIDANCE_RPM_DIRECTORIES="results rpmbuild/RPMS rpmbuild/SPECS repo/local-repo/dependancy-cache"
BUILD_AVOIDANCE_RPM_FILES=".platform_release"

Details of the files and directories downloaded include:

  • inputs = Working directory used to assemble srpms from git or tarball

  • srpm_assemble = Working directory used to assemble srpms from upstream srpms

  • rpmbuild/SRPMS = Assembled stx src.rpms to build

  • rpmbuild/SOURCES = Additional per package metadata data collected to support build avoidance

  • rpmbuild/SOURCES/<package-name>/srpm_reference.md5 = md5sums of all files that go into building the STX src.rpm

  • results = Per package build logs and artifacts generated by mockchain

  • rpmbuild/RPMS = Build RPMs

  • rpmbuild/SPECS = Spec files of build RPMs

  • repo/local-repo/dependancy-cache = build-pkgs data summarizing:

    • The ‘Requires’ of RPMs

    • The ‘BuildRequires’ of src.rpms

    • Which RPMs are derived from which src.rpms

  • .platform_release = Platform release value

On the reference builds side, the only extra step to support build avoidance is to generate rpmbuild/SOURCES/<package-name>/srpm_reference.md5 files.

On the consumer side, for each build type:

  • For each file or subdirectory listed in $MY_REPO/build-data/build_avoidance_source, rsync the file or directory with options to preserve the file time stamp.

Build tool operations

The build tools automatically perform the tasks described below. There are no required configuration steps for setting up reference builds and no actions for consuming reference builds.

For each build type and for each package, build src.rpms:

  • Generate a list of input files for the current package.

  • Generate a srpm_reference.md5 file for the current inputs.

  • Compare srpm_reference.md5 files for current and reference builds. If differences are found (list of files, or md5sum of those files), then rebuild this src.rpm.

For each build type, for each package, and for the list of RPMs built by src.rpm:

  • If rpm is missing, must rebuild package.

  • If rpm is wrong version, must rebuild package.

  • If rpm older than src.rpm, must rebuild package.

    Note

    Assumes reference build and consumer are on NTP time, and any drift is well below the download time for the reference build.

Designer actions

  • Request a build avoidance build. Recommended after you have synchronized the repository using repo sync as shown below:

    repo sync
generate-centos-repo.sh
populate_downloads.sh
build-pkgs --build-avoidance

  • Use combinations of additional arguments, environment variables, and a configuration file unique to the regional office to specify an URL to the reference builds.

  • Using a configuration file to specify the location of your reference build:

    mkdir -p $MY_REPO/local-build-data

cat <<- EOF > $MY_REPO/local-build-data/build_avoidance_source
# Optional, these are already the default values.
BUILD_AVOIDANCE_DATE_FORMAT="%Y%m%d"
BUILD_AVOIDANCE_TIME_FORMAT="%H%M%S"
BUILD_AVOIDANCE_DATE_TIME_DELIM="T"
BUILD_AVOIDANCE_DATE_TIME_POSTFIX="Z"
BUILD_AVOIDANCE_DATE_UTC=1
BUILD_AVOIDANCE_FILE_TRANSFER="rsync"

# Required, unique values for each regional office
BUILD_AVOIDANCE_USR="jenkins"
BUILD_AVOIDANCE_HOST="stx-builder.mycompany.com"
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
EOF

  • Using command-line arguments to specify the location of your reference build:

    build-pkgs --build-avoidance --build-avoidance-dir /localdisk/loadbuild/jenkins/StarlingX_Reference_Build --build-avoidance-host stx-builder.mycompany.com --build-avoidance-user jenkins

  • You must accept the host key before your build attempt to prevent rsync failures on a yes/no prompt. You only have to do this once.

    grep -q $BUILD_AVOIDANCE_HOST $HOME/.ssh/known_hosts
if [ $? != 0 ]; then
ssh-keyscan $BUILD_AVOIDANCE_HOST >> $HOME/.ssh/known_hosts
fi

  • build-pkgs does the following:

    • From newest to oldest, scans the CONTEXTs of the various reference builds. Selects the first (i.e. most recent) context that satisfies the following requirement: every Git the SHA specifies in the CONTEXT is present.

    • The selected context might be slightly out of date, but not by more than a day. This assumes daily reference builds are run.

    • If the context has not been previously downloaded, then download it now. This means you need to download select portions of the reference build workspace into the designer’s workspace. This includes all the SRPMS, RPMS, MD5SUMS, and miscellaneous supporting files. Downloading these files usually takes about 10 minutes over an office LAN.

    • The designer could have additional commits or uncommitted changes not present in the reference builds. Affected packages are identified by the differing md5sum values. In these cases, the packages are rebuilt. Rebuilds usually take five or more minutes, depending on the packages that have changed.

  • What if no valid reference build is found? Then build-pkgs will fall back to a regular build.

Reference builds

  • The regional office implements an automated build that pulls the latest StarlingX software and builds it on a regular basis (e.g. daily builds). Jenkins, cron, or similar tools can trigger these builds.

  • Each build is saved to a unique directory, and preserved for a time that is reflective of how long a designer might be expected to work on a private branch without synchronizing with the master branch. This takes about two weeks.

  • We recommend that the MY_WORKSPACE directory for the build has a common root directory, and a leaf directory that is a sortable time stamp. The suggested format is YYYYMMDDThhmmss.

    sudo apt-get update
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ')
MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${BUILD_TIMESTAMP}

  • Designers can access all build products over the internal network of the regional office. The current prototype employs rsync. Other protocols that can efficiently share, copy, or transfer large directories of content can be added as needed.

Advanced usage

Can the reference build itself use build avoidance? Yes, it can. Can it reference itself? Yes, it can. However, in both these cases, caution is advised. To protect against any possible ‘divergence from reality’, you should limit how many steps you remove a build avoidance build from a full build.

Suppose we want to implement a self-referencing daily build in an environment where a full build already occurs every Saturday. To protect ourselves from a build failure on Saturday, we also want a limit of seven days since the last full build. Your build script might look like this:

...
BUILD_AVOIDANCE_DIR="/localdisk/loadbuild/jenkins/StarlingX_Reference_Build"
BUILD_AVOIDANCE_HOST="stx-builder.mycompany.com"
FULL_BUILD_DAY="Saturday"
MAX_AGE_DAYS=7

LAST_FULL_BUILD_LINK="$BUILD_AVOIDANCE_DIR/latest_full_build"
LAST_FULL_BUILD_DAY=""
NOW_DAY=$(date -u "+%A")
BUILD_TIMESTAMP=$(date -u '+%Y%m%dT%H%M%SZ')
MY_WORKSPACE=${BUILD_AVOIDANCE_DIR}/${BUILD_TIMESTAMP}

# update software
repo init -u ${BUILD_REPO_URL} -b ${BUILD_BRANCH}
repo sync --force-sync
$MY_REPO_ROOT_DIR/tools/toCOPY/generate-centos-repo.sh
$MY_REPO_ROOT_DIR/tools/toCOPY/populate_downloads.sh

# User can optionally define BUILD_METHOD equal to one of 'FULL', 'AVOIDANCE', or 'AUTO'
# Sanitize BUILD_METHOD
if [ "$BUILD_METHOD" != "FULL" ] && [ "$BUILD_METHOD" != "AVOIDANCE" ]; then
    BUILD_METHOD="AUTO"
fi

# First build test
if [ "$BUILD_METHOD" != "FULL" ] && [ ! -L $LAST_FULL_BUILD_LINK ]; then
    echo "latest_full_build symlink missing, forcing full build"
    BUILD_METHOD="FULL"
fi

# Build day test
if [ "$BUILD_METHOD" == "AUTO" ] && [ "$NOW_DAY" == "$FULL_BUILD_DAY" ]; then
    echo "Today is $FULL_BUILD_DAY, forcing full build"
    BUILD_METHOD="FULL"
fi

# Build age test
if [ "$BUILD_METHOD" != "FULL" ]; then
    LAST_FULL_BUILD_DATE=$(basename $(readlink $LAST_FULL_BUILD_LINK) | cut -d '_' -f 1)
    LAST_FULL_BUILD_DAY=$(date -d $LAST_FULL_BUILD_DATE "+%A")
    AGE_SECS=$(( $(date "+%s") - $(date -d $LAST_FULL_BUILD_DATE "+%s") ))
    AGE_DAYS=$(( $AGE_SECS/60/60/24 ))
    if [ $AGE_DAYS -ge $MAX_AGE_DAYS ]; then
        echo "Haven't had a full build in $AGE_DAYS days, forcing full build"
        BUILD_METHOD="FULL"
    fi
    BUILD_METHOD="AVOIDANCE"
fi

#Build it
if [ "$BUILD_METHOD" == "FULL" ]; then
    build-pkgs --no-build-avoidance
else
    build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER
fi
if [ $? -ne 0 ]; then
    echo "Build failed in build-pkgs"
    exit 1
fi

build-iso
if [ $? -ne 0 ]; then
    echo "Build failed in build-iso"
    exit 1
fi

if [ "$BUILD_METHOD" == "FULL" ]; then
    # A successful full build.  Set last full build symlink.
    if [ -L $LAST_FULL_BUILD_LINK ]; then
        rm -rf $LAST_FULL_BUILD_LINK
    fi
    ln -sf $MY_WORKSPACE $LAST_FULL_BUILD_LINK
fi
...

To use the full build day as your avoidance build reference point, modify the build-pkgs commands above to use --build-avoidance-day, as shown in the following two examples:

build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER --build-avoidance-day $FULL_BUILD_DAY

# Here is another example with a bit more shuffling of the above script.

build-pkgs --build-avoidance --build-avoidance-dir $BUILD_AVOIDANCE_DIR --build-avoidance-host $BUILD_AVOIDANCE_HOST --build-avoidance-user $USER --build-avoidance-day $LAST_FULL_BUILD_DAY

The advantage is that our build is never more than one step removed from a full build. This assumes the full build was successful.

The disadvantage is that by the end of the week, the reference build is getting rather old. During active weeks, build times could approach build times for full builds.