StarlingX: STX Tool

STX Tool - Script consolidation for build management, build and more

Storyboard Story: https://storyboard.openstack.org/#!/story/2008862

Create a new ‘stx’ tool which would consolidate many of the tools currently available in the starlingx/tools repository. These tools are used to manage the STX Build container, manipulate the package repository mirror, create platform builds, etc..

By consolidating many tools into a single tool with many functions we can offer many benefits

  • Rolled up help text

  • Improved tab completion

  • Consistency from sub-command to sub-command

  • Facilitate code re-use

  • Provide a familiar interface as found in git or kubectl for example

When combined with the Debian Build spec the ‘stx’ tool will help ensure that the more complex build architecture is abstracted from the developer.

Problem description

Navigating the StarlingX tools repository today requires former knowledge of the many available tools or heavy reliance on documentation resources. This is often not a friendly environment for new users who are just starting to build StarlingX for the first time.

The tool authors have done a good job keeping coding standards, variable use, naming conventions, etc. consistent, but as new developers contribute to the project this consistency becomes harder to enforce. It is also hard to share functionality between scripts to encourage code reuse.

If this spec is not accepted, in addition to major updates to support the transition to Debian, most of the tools would have to be made conditional on if CentOS or Debian operations would be desired. If this spec is accepted most of the current tools can be left in place and as-is throughout the Debian transition, allowing continued use to build CentOS builds alongside the Debian transition work.

Use Cases

This change would affect most of the use cases described in the StarlingX Build Guide[1]. Such as, but not limited to:

  • Build packages

  • Build ISOs

  • Build installer

  • Build StarlingX OpenStack Application

This would impact the StarlingX Developer. All key operations performed by developers will have equivalents available in the ‘stx’ tool. As part of this spec all StarlingX resources will be updated to reflect the changes, but will not extend to developer’s own resources. Any wrapper scripts written by developers, notes, documents, etc. written for their personal use will have to be updated by them.

After this spec is implemented the developer will have access to a command structure matching that of common tools such as git and kubectl. This type of command structure brings with it many benefits that would hopefully make it easier for new developers to start building and contributing to the StarlingX project.

[1][https://docs.starlingx.io/developer_resources/build_guide.html]

Proposed change

A new ‘stx’ tool is available in the starlingx/tools repository. This tool will be written in a language such as Python(3) to facilitate the development of modules which can be reused by the various sub-commands.

The ‘stx’ tool will be developed as part of the transition to Debian, allowing the existing tools to remain untouched and available for CentOS based builds throughout the OS transition (and possibly into the future).

Running the ‘stx’ tool with no arguments will display help text describing the available sub-commands. This behavior matches what you see with git or kubectl.

Running the ‘stx’ tool with a single argument, a sub-command name, will display help text for that specific sub-command. This will include things such as command line options, details about inputs or outputs, etc.. It may be possible for sub-commands to have sub-commands of their own.

The ‘stx’ tool will have a sub-command to manage its own configuration. Users can choose to edit the configuration file on their own, but using ‘stx config’ will ensure valid values are used and provide other useful feedback. Again, this should be similar to the experience found with ‘git config’, though it should not be required to be as complex.

A review of existing tools in the starlingx/tools repository will be done to ensure equivalent operations are available in the new ‘stx’ tool, with adjustments made to reflect the Debian build requirements. Determinations can be made to not include an existing tool in the ‘stx’ tool for technical or practical reasons, these should be the minority.

The ‘stx’ tool will not wrap other tools such as git or repo unless it makes sense to do so. For instance if a call to repo is made as a step in the middle of a build we do not want to force a user to run a ‘stx’ tool command, run a repo command and then run a second ‘stx’ command to complete a build, in this case the ‘stx’ tool would make the call to repo. Discretion will be made in this area to ensure the right tool is used for the job.

The ‘stx’ tool will have sub-commands for work such as, but not limited to:

  • controlling build container(s) state (stx control)

  • building (stx build)

  • configuration get/set (stx config)

  • repository management (stx repomgmt)

  • status information (stx status)

  • version information (stx versions)

(name for demonstration purposes only, subject to change)

NOTE: “repository management” or “repomgmt” in the above refers to the handling of a Debian binary package and source package repository, as described by Debian Repository Format [2]. This context will be provided by help text and documentation to provide disambiguation from the ‘repo tool’ or ‘git repository’.

[2][https://wiki.debian.org/DebianRepository/Format]

Alternatives

We could reuse the existing scripts, make them conditional on CentOS vs Debian and implement the Debian analogues. On the one hand this would be one less change for developers to absorb as we transition to Debian, but on the other it does not address the issues described here.

Additionally to reusing the scripts with Debian modifications we could review the organization of the tools repository, such as consolidating scripts in a ‘bin’ directory, having a docs directory which map to individual commands… This would address some of the issues described but would leave others open, such as code reuse.

Data model impact

None

REST API impact

None

Security impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Developer impact will not be insignificant. Many/most of the commands they are using today will be dropped in favor of using commands in the new ‘stx’ tool. Any personal resources they have developed to assist in their workflows which make use of tools which will be impacted by this change and will be required to be updated by them. Unfortunately this impact is hard to measure as these resources fall outside of the project scope.

New and existing developers should find the new ‘stx’ tool easy to use and familiar if they are used to this command structure. Even if they are not familiar with this command structure, the rolled up help text, easy tab completion and other benefits should translate to them quickly being proficient with the tool.

Developers who work on the tools themselves will hopefully benefit from code reuse making changes and expanded functionality easier to implement.

Upgrade impact

None

Implementation

Assignee(s)

Primary assignee:

Mark Asselstine - markawr

Other contributors:

  • Yue Tao - wrytao

  • Hongxu Jia - hjia

  • Xiao Zhang - xzhang1

  • Peng Yan - yanpenghit

  • Haiqing Bai - hqbai

Repos Impacted

starlingx/tools

Work Items

See storyboard. Additional items will be added after spec approval.

Dependencies

This spec is related to the Debian Transition spec and the Debian Build spec. This spec does not depend on the Debian Build spec and can be accepted and implemented regardless of the outcome of that spec.

Essentially the existing tools must be re-worked to support the already accepted Debian Transition spec. If this re-work were to be done with the introduction of a ‘stx’ tool, as described in this spec, it would simplify the tools work needed to support the updated Debian Build.

Testing

As the scope of this spec is restricted to the building of StarlingX it does not introduce any additional runtime testing requirements. As this change is proposed to take place alongside the move to Debian, full runtime testing is expected related to that spec.

Every command, sub-command, option, input and output of the new tool should have test code submitted alongside code implementing functionality. Since this work is taking place alongside the Debian Transition work it will not be possible to test existing and new outcomes to ensure a match. Because of this existing tool’s outcomes will be reviewed and the equivalent ‘stx’ tool operation outcomes will be checked for parity.

In cases where new commands and sub-commands are created, where no existing tool equivalent exists, requirements will be developed and outcomes will be tested against these requirements.

Documentation Impact

Revisit much of the StarlingX Build Guide https://docs.starlingx.io/developer_resources/build_guide.html

Revisit much of the StarlingX Layered Build Guide https://docs.starlingx.io/developer_resources/layered_build_guide.html

Possible updates to the Build StarlingX Docker Images document https://docs.starlingx.io/developer_resources/build_docker_image.html

NOTE: updates to these documents are already required to be updated for the Debian transition, regardless of the changes suggested in this spec, though the scope with these changes will be larger.

References

History

Revisions

Release Name

Description

STX-6.0

Introduced