Layered Build Guide¶
This section describes the steps for layered build and building an ISO image from a StarlingX R4.0 release onwards.
For more detailed explanation of layered build, please refer Layered Build Reference.
Requirements¶
Layered build has the same requirements as StarlingX R3.0 build. Click the heading above for details.
Development environment setup¶
Layered build uses the same development environment setup as StarlingX R3.0 build. Click the heading above for details.
Build compiler layer¶
Prepare the base Docker image¶
Follow Prepare the base Docker image for creating the base StarlingX image with changes only in localrc file as below. Since, we are building compiler layer, create and modify only localrc file. Make sure to set the project and layer as compiler and user name as desired. For example:
# tbuilder localrc
MYUNAME=<your user name>
PROJECT=compiler
HOST_PREFIX=$HOME/starlingx/workspace
HOST_MIRROR_DIR=$HOME/starlingx/mirror
LAYER=compiler
Build the CentOS mirror repository¶
The creation of the StarlingX ISO relies on a repository of RPM binaries, RPM sources, and tar compressed files. This section describes how to build this CentOS mirror repository.
Run building Docker container¶
Follow the building docker container for more details. At the end of this procedure, you will be inside the Starlingx container.
Quick command summary is as below:
cd $HOME/tools/
bash tb.sh env
bash tb.sh run
bash tb.sh exec
Download source code repositories¶
Inside the building Docker container, start the internal environment:
eval $(ssh-agent) ssh-add
Use the repo tool to create a local clone of the manifest git repository based on the “master” branch for compiler layer.
cd $MY_REPO_ROOT_DIR repo init -u https://opendev.org/starlingx/manifest -b master -m compiler.xml
Synchronize code for compiler layer using command below:
repo sync -j`nproc`
Download packages¶
Inside the Docker container, enter the following commands to download the required packages to populate the CentOS mirror repository:
cd $MY_REPO_ROOT_DIR/stx-tools/centos-mirror-tools && bash download_mirror.sh
Monitor the download of packages until it is complete. When the download is complete successfully with no missing packages, the following message appears:
step #5: done successfully sudo rm -rf /tmp/stx_mirror_4d2URF IMPORTANT: The following 3 files are just bootstrap versions. Based on them, the workable images for StarlingX could be generated by running "update-pxe-network-installer" command after "build-iso" - ./output/stx/CentOS/Binary/LiveOS/squashfs.img - ./output/stx/CentOS/Binary/images/pxeboot/initrd.img - ./output/stx/CentOS/Binary/images/pxeboot/vmlinuz Success
Verify packages¶
Verify no missing or failed packages exist:
cat logs/*_missing_*.log cat logs/*_failmoved_*.log
In case missing or failed packages do exist, which is usually caused by network instability (or timeout), you need to download the packages manually or re-run the download command given in previous step. Doing so assures you get all RPMs listed in centos_srpms_3rdparties.lst, centos_srpms_centos.lst, centos_srpms_centos3rdparties.lst.
Packages structure¶
The following is a general overview of the packages structure resulting from downloading the packages for compiler layer:
/localdisk/designer/<user>/<project>/stx-tools/centos-mirror-tools/output
.
└── stx
└── CentOS
├── Binary
│ ├── noarch
│ └── x86_64
├── downloads
│ └── puppet
├── layer_build_info
├── layer_image_inc
├── layer_pkg_lists
├── layer_wheels_inc
└── Source
Copy CentOS mirror repository¶
Exit from the building Docker container. Run the following commands:
Navigate to CentOS mirror directory mirror/CentOS under your starlingx workspace directory:
cd $HOME/starlingx/mirror/CentOS/
Copy the built CentOS mirror repository $HOME/starlingx/mirror/ workspace directory. Make sure to insert the <user>:
cp -r $HOME/starlingx/workspace/localdisk/designer/<user>/compiler/stx-tools/centos-mirror-tools/output/stx .
Create a tarball repository¶
Enter the StarlingX container using below command:
cd $HOME/tools/ ./tb.sh exec
Copy downloaded CentOS tarballs into StarlingX repo
ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/
Alternatively, you can run the “populate_downloads.sh” script to copy the tarballs instead of using a symlink:
bash populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/
Build packages¶
Temporal! Build-Pkgs Errors. Be prepared to have some missing / corrupted rpm and tarball packages generated during Build the CentOS mirror repository, which will cause the next step to fail. If that step does fail, manually download those missing / corrupted packages.
Update the symbolic links:
bash generate-centos-repo.sh /import/mirrors/CentOS/stx/CentOS/
Build the packages:
build-pkgs build-pkgs --installer
Build distro layer¶
Build the CentOS mirror repository¶
The creation of the StarlingX ISO relies on a repository of RPM binaries, RPM sources, and tar compressed files. This section describes how to build this CentOS mirror repository.
If you were building compiler layer, exit from container and enter the tools directory using below commands:
cd $HOME/tools/
Since, we are building distro layer, modify localrc file. Make sure to set the project and layer as distro. For example:
# tbuilder localrc
MYUNAME=<your user name>
PROJECT=distro
HOST_PREFIX=$HOME/starlingx/workspace
HOST_MIRROR_DIR=$HOME/starlingx/mirror
LAYER=distro
Run building Docker container¶
Follow the building docker container for more details. At the end of this procedure, you will be inside the Starlingx container.
Quick command summary is as below:
cd $HOME/tools/
bash tb.sh env
bash tb.sh run #Need not give, If the container was created earlier.
bash tb.sh exec
Download source code repositories¶
Inside the building Docker container, start the internal environment:
eval $(ssh-agent) ssh-add
Use the repo tool to create a local clone of the manifest git repository based on the “master” branch for compiler layer.
cd $MY_REPO_ROOT_DIR repo init -u https://opendev.org/starlingx/manifest -b master -m distro.xml
Synchronize code for compiler layer using command below:
repo sync -j`nproc`
Download packages¶
Inside the Docker container, enter the following commands to download the required packages to populate the CentOS mirror repository:
cd $MY_REPO_ROOT_DIR/stx-tools/centos-mirror-tools && bash download_mirror.sh
Monitor the download of packages until it is complete. When the download is complete, the following message appears:
step #5: done successfully sudo rm -rf /tmp/stx_mirror_UIQ675 IMPORTANT: The following 3 files are just bootstrap versions. Based on them, the workable images for StarlingX could be generated by running "update-pxe-network-installer" command after "build-iso" - ./output/stx/CentOS/Binary/LiveOS/squashfs.img - ./output/stx/CentOS/Binary/images/pxeboot/initrd.img - ./output/stx/CentOS/Binary/images/pxeboot/vmlinuz Success
Verify packages¶
Verify no missing or failed packages exist:
cat logs/*_missing_*.log cat logs/*_failmoved_*.log
In case missing or failed packages do exist, which is usually caused by network instability (or timeout), you need to download the packages manually. Doing so assures you get all RPMs listed in centos_srpms_3rdparties.lst, centos_srpms_centos.lst, centos_srpms_centos3rdparties.lst.
Packages structure¶
The following is a general overview of the packages structure resulting from downloading the packages for distro layer:
/localdisk/designer/<user>/distro/stx-tools/centos-mirror-tools/output
.
└── stx
└── CentOS
├── Binary
│ ├── EFI
│ │ └── BOOT
│ │ └── fonts
│ ├── images
│ │ └── pxeboot
│ ├── isolinux
│ ├── LiveOS
│ ├── noarch
│ └── x86_64
├── downloads
│ └── puppet
│ └── packstack
│ └── puppet
│ └── modules
├── layer_build_info
├── layer_image_inc
├── layer_pkg_lists
├── layer_repos
│ └── compiler
│ └── std
│ ├── repodata
│ └── repodata.upstream
├── layer_wheels_inc
└── Source
Copy CentOS mirror repository¶
Exit from the building Docker container. Run the following commands:
Navigate to CentOS mirror directory mirror/CentOS under your starlingx workspace directory:
cd $HOME/starlingx/mirror/CentOS/
Copy the built CentOS mirror repository $HOME/starlingx/mirror/ workspace directory. Make sure to insert the <user>:
cp -r $HOME/starlingx/workspace/localdisk/designer/<user>/distro/stx-tools/centos-mirror-tools/output/stx .
Create a tarball repository¶
Enter the StarlingX container using below command:
cd $HOME/tools/ ./tb.sh exec
Copy downloaded CentOS tarballs into StarlingX repo
ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/
Alternatively, you can run the “populate_downloads.sh” script to copy the tarballs instead of using a symlink:
populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/
Build packages¶
Temporal! Build-Pkgs Errors. Be prepared to have some missing / corrupted rpm and tarball packages generated during Build the CentOS mirror repository, which will cause the next step to fail. If that step does fail, manually download those missing / corrupted packages.
Update the symbolic links:
bash generate-centos-repo.sh /import/mirrors/CentOS/stx/CentOS/
This step creates the repo directory, following is the output on the console for a successful repo directory creation for the <user> stx:
Copying comps.xml file. Createing yum repodata. Directory walk started Directory walk done - 51 packages Temporary output repo path: /localdisk/designer/stx/distro/cgcs-root/local-repo/Source/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Directory walk started Directory walk done - 0 packages Temporary output repo path: /localdisk/designer/stx/distro/cgcs-root/local-repo/rt/Source/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Directory walk started Directory walk done - 1450 packages Temporary output repo path: /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Directory walk started Directory walk done - 0 packages Temporary output repo path: /localdisk/designer/stx/distro/cgcs-root/local-repo/rt/Binary/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Copying mock.cfg.proto file. Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/EFI/BOOT/BOOTX64.EFI Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/EFI/BOOT/fonts/unicode.pf2 Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/EFI/BOOT/grub.cfg Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/EFI/BOOT/grubx64.efi Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/images/efiboot.img Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/images/pxeboot/initrd.img Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/images/pxeboot/vmlinuz Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/boot.msg Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/grub.conf Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/initrd.img Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/isolinux.bin Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/isolinux.cfg Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/memtest Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/splash.png Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/vesamenu.c32 Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/isolinux/vmlinuz Creating symlink for /localdisk/designer/stx/distro/cgcs-root/local-repo/Binary/LiveOS/squashfs.img Creating folder EFI Creating folder EFI/BOOT Creating folder EFI/BOOT/fonts Creating folder images Creating folder images/pxeboot Creating folder isolinux Creating folder LiveOS Done creating repo directory
Build the packages:
build-pkgs build-pkgs --installer
Build flock layer¶
Build the CentOS mirror repository¶
The creation of the StarlingX ISO relies on a repository of RPM binaries, RPM sources, and tar compressed files. This section describes how to build this CentOS mirror repository.
If you were building distro layer, exit from container and enter the tools directory using below commands:
cd $HOME/tools/
Since, we are building flock layer, modify localrc file. Make sure to set the project and layer as flock. For example:
# tbuilder localrc
MYUNAME=<your user name>
PROJECT=flock
HOST_PREFIX=$HOME/starlingx/workspace
HOST_MIRROR_DIR=$HOME/starlingx/mirror
LAYER=flock
Run building Docker container¶
Follow the building docker container for more details. At the end of this procedure, you will be inside the Starlingx container.
Quick command summary is as below:
cd $HOME/tools/
bash tb.sh env
bash tb.sh run #Need not give, If the container was created earlier.
bash tb.sh exec
Download source code repositories¶
Inside the building Docker container, start the internal environment:
eval $(ssh-agent) ssh-add
Use the repo tool to create a local clone of the manifest git repository based on the “master” branch for flock layer.
cd $MY_REPO_ROOT_DIR repo init -u https://opendev.org/starlingx/manifest -b master -m flock.xml
Synchronize code for compiler layer using command below:
repo sync -j`nproc`
Download packages¶
Inside the Docker container, enter the following commands to download the required packages to populate the CentOS mirror repository:
cd $MY_REPO_ROOT_DIR/stx-tools/centos-mirror-tools && bash download_mirror.sh
Monitor the download of packages until it is complete. When the download is complete, the following message appears:
step #5: done successfully sudo rm -rf /tmp/stx_mirror_ievdiA IMPORTANT: The following 3 files are just bootstrap versions. Based on them, the workable images for StarlingX could be generated by running "update-pxe-network-installer" command after "build-iso" - ./output/stx/CentOS/Binary/LiveOS/squashfs.img - ./output/stx/CentOS/Binary/images/pxeboot/initrd.img - ./output/stx/CentOS/Binary/images/pxeboot/vmlinuz
Verify packages¶
When the download is not successful, the following message appears:
IMPORTANT: The following 3 files are just bootstrap versions. Based
on them, the workable images for StarlingX could be generated by
running "update-pxe-network-installer" command after "build-iso"
- ./output/stx/CentOS/Binary/LiveOS/squashfs.img
- ./output/stx/CentOS/Binary/images/pxeboot/initrd.img
- ./output/stx/CentOS/Binary/images/pxeboot/vmlinuz
Warning: Not all download steps succeeded. You are likely missing files.
You can verify and list missing or failed packages using below command:
cat logs/*_missing_*.log
cat logs/*_failmoved_*.log
In case missing or failed packages do exist, which is usually caused by network instability (or timeout), you need to download the packages manually. Doing so assures you get all RPMs listed in centos_srpms_3rdparties.lst, centos_srpms_centos.lst, centos_srpms_centos3rdparties.lst.
Packages structure¶
The following is a general overview of the packages structure resulting from downloading the packages for flock layer:
/localdisk/designer/<user>/flock/stx-tools/centos-mirror-tools/output
└── stx
└── CentOS
├── Binary
│ ├── EFI
│ │ └── BOOT
│ │ └── fonts
│ ├── images
│ │ └── pxeboot
│ ├── isolinux
│ ├── LiveOS
│ ├── noarch
│ └── x86_64
├── downloads
│ └── puppet
├── layer_build_info
├── layer_image_inc
├── layer_pkg_lists
├── layer_repos
│ ├── compiler
│ │ └── std
│ │ ├── repodata
│ │ └── repodata.upstream
│ └── distro
│ ├── installer
│ │ ├── repodata
│ │ └── repodata.upstream
│ ├── rt
│ │ ├── repodata
│ │ └── repodata.upstream
│ └── std
│ ├── repodata
│ └── repodata.upstream
├── layer_wheels_inc
└── Source
Copy CentOS mirror repository¶
Exit from the building Docker container. Run the following commands:
Navigate to CentOS mirror directory mirror/CentOS under your starlingx workspace directory:
cd $HOME/starlingx/mirror/CentOS/
Copy the built CentOS mirror repository $HOME/starlingx/mirror/ workspace directory:
cp -r $HOME/starlingx/workspace/localdisk/designer/<user>/flock/stx-tools/centos-mirror-tools/output/stx .
Create a tarball repository¶
Enter the StarlingX container using below command:
cd $HOME/tools/ ./tb.sh exec
Copy downloaded CentOS tarballs into StarlingX repo
ln -s /import/mirrors/CentOS/stx/CentOS/downloads/ $MY_REPO/stx/
Alternatively, you can run the “populate_downloads.sh” script to copy the tarballs instead of using a symlink:
populate_downloads.sh /import/mirrors/CentOS/stx/CentOS/
Exit from the container. On the host machine, create mirror binaries:
mkdir -p $HOME/starlingx/mirror/CentOS/stx-installer cp $HOME/starlingx/mirror/CentOS/stx/CentOS/Binary/images/pxeboot/initrd.img $HOME/starlingx/mirror/CentOS/stx-installer/initrd.img cp $HOME/starlingx/mirror/CentOS/stx/CentOS/Binary/images/pxeboot/vmlinuz $HOME/starlingx/mirror/CentOS/stx-installer/vmlinuz cp $HOME/starlingx/mirror/CentOS/stx/CentOS/Binary/LiveOS/squashfs.img $HOME/starlingx/mirror/CentOS/stx-installer/squashfs.img
Build packages¶
Enter the StarlingX container using below command:
cd $HOME/tools/ ./tb.sh exec
Temporal! Build-Pkgs Errors. Be prepared to have some missing / corrupted rpm and tarball packages generated during Build the CentOS mirror repository, which will cause the next step to fail. If that step does fail, manually download those missing / corrupted packages.
Update the symbolic links:
bash generate-centos-repo.sh /import/mirrors/CentOS/stx/CentOS/
Following is the output:
Copying comps.xml file. Createing yum repodata. Directory walk started Directory walk done - 1 packages Temporary output repo path: /localdisk/designer/stx/flock/cgcs-root/local-repo/Source/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Directory walk started Directory walk done - 0 packages Temporary output repo path: /localdisk/designer/stx/flock/cgcs-root/local-repo/rt/Source/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Directory walk started Directory walk done - 1892 packages Temporary output repo path: /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Directory walk started Directory walk done - 40 packages Temporary output repo path: /localdisk/designer/stx/flock/cgcs-root/local-repo/rt/Binary/.repodata/ Preparing sqlite DBs Pool started (with 5 workers) Pool finished Copying mock.cfg.proto file. Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/EFI/BOOT/BOOTX64.EFI Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/EFI/BOOT/fonts/unicode.pf2 Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/EFI/BOOT/grub.cfg Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/EFI/BOOT/grubx64.efi Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/images/efiboot.img Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/images/pxeboot/initrd.img Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/images/pxeboot/vmlinuz Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/boot.msg Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/grub.conf Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/initrd.img Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/isolinux.bin Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/isolinux.cfg Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/memtest Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/splash.png Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/vesamenu.c32 Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/isolinux/vmlinuz Creating symlink for /localdisk/designer/stx/flock/cgcs-root/local-repo/Binary/LiveOS/squashfs.img Creating folder EFI Creating folder EFI/BOOT Creating folder EFI/BOOT/fonts Creating folder images Creating folder images/pxeboot Creating folder isolinux Creating folder LiveOS Done creating repo directory
Build the packages:
build-pkgs
Build StarlingX ISO¶
Build the image:
build-iso
Build installer¶
Layered build has the same procedure for build installer as StarlingX R3.0 build except for the changes in path of files as below. Click the heading above for details.
The steps covered by the script update-pxe-network-installer is detailed in $MY_REPO/stx/stx-metal/installer/initrd/README. This script creates three files on /localdisk/loadbuild/stx/flock/pxe-network-installer/output.
The path for build_srpm.data is $MY_REPO/stx/metal/installer/pxe-network-installer/centos/.