Automated Application Package Updater¶

Storyboard URL:: https://storyboard.openstack.org/#!/story/2011392

Introduction¶

This toolset automates the upgrade of StarlingX applications by seamlessly incorporating revised upstream packages into StarlingX. It replaces the historically challenging, manual, and error-prone process of revising StarlingX applications with automated tools that consistently and accurately integrate the latest upstream changes. The update process is streamlined using two key functionalities alongside additional supporting functions, enabling StarlingX packages to be updated at a faster and more reliable cadence—improving both quality and development/release velocity. The two key areas of improved functionality are:

YAML Markup:
Enables precise modification of YAML files and YAML-encoded Helm templates using a dedicated markup configuration.
Helm Template Encoding/Decoding:
Converts Helm templates into valid YAML syntax and vice versa, allowing semantic modifications to be applied directly. During encoding, multi-line Helm blocks are transformed into Base64 encoded placeholders. When decoding, these placeholders are converted back into the original multi-line Helm blocks.

In addition, the toolset performs all necessary actions to complete a commit for the upgrade. Supporting modules handle tasks such as synchronizing code with upstream, generating commit messages based on version updates and configuration details, creating patch files via git diff, staging the resulting patches, and integrating with Gerrit for code reviews.

Problem Description¶

The current process for updating upstream packages presents several challenges:

Generation of Updated Patch Files for Each Release:
Although many required changes are similar between releases, variations in upstream syntax force the manual recreation of new patch files based on the patches used in the prior release.
Manual and Error-Prone Process:
Patch creation is the most challenging aspect of the upgrades. Developers must manually update upstream source code using previous patch files as reference, which is both time-consuming and error-prone. In addition, updates for supplementary artifacts—such as changelogs, metadata files, and other configuration documents—must also be performed manually.
Cumulative Complexity:
The sequential layering of patch files makes it difficult for engineers to accurately reason about the overall modifications. This complexity hinders the ability to organize and verify changes, as deprecated or legacy modifications can be overlooked.
Need for Consolidation:
There is a critical need for an automated, integrated approach that consolidates patch file creation and updates to ancillary files. Such an approach would simplify the reasoning process for engineers and ensure consistency and accuracy throughout the upgrade process.

Use Cases¶

Developers:
- Upstream Target Selection:
  Developers select an upstream target version (typically the latest tagged release) and a branch as the basis for modifications.
- Local Code Pull and Patch Application:
  The tool pulls the upstream code locally and applies YAML-based modifications using the provided markup configuration to Helm templates as needed on a per-application basis.
- Code Review Generation:
  The tool generates a structured code review that details the applied changes.
- Additional Configuration:
  Updates for other file types (e.g., changelogs, metadata files, etc.) are to be finalized, but the expectation is that available tools and utilities can be leveraged for this purpose.
DevOps/Operators:
- Interface Options:
  The tool can be executed via the command line or a web-based UI, allowing users to easily select target versions and branches.
- Consistent and Transparent Updates:
  The automated process ensures that YAML modifications are applied consistently and that changes standardize the syntax style, simplifying code review diffs and reducing errors.
- Potential Integration:
  The generated code review can be integrated into CI/CD pipelines to facilitate subsequent testing and review processes.
CI/CD Integration:
- Automated Code Review Output:
  The tool produces a code review that captures the modifications applied to the upstream code.
- Optional Pipeline Integration:
  This code review output can be integrated into CI/CD pipelines to trigger further artifact generation and testing.
- Minimal Built-In Validation:
  While the tool performs basic validation, additional testing processes—aligned with both manual and automated workflows—ensure that changes are robust before deployment.

Proposed Change¶

The proposed change aims to automate the upgrade process for StarlingX applications by generating complete code commits that incorporate all necessary modifications. This is achieved by combining core capabilities and supporting modules to handle every aspect of the upgrade.

Core Capabilities:
- Helm Template Encoding/Decoding:
  - Purpose: Convert Helm templates into valid YAML syntax and vice versa.
  - Functionality: Enables the toolset to interpret and manipulate Helm charts as YAML files, which is essential for applying semantic modifications required for an upgrade.
- YAML Markup and Transformation:
  - Purpose: Modify YAML files that include encoded Helm templates.
  - Functionality:
    
    Parses the YAML while recognizing the underlying Helm encoding.
    
    Applies schema-driven transformations (add, update, delete) to generate updated patch files.
Supporting Modules:
- Code Synchronization and Commit Module:
  Handles pulling upstream code, managing Git interactions, and interfacing with Gerrit. It generates commit messages based on version updates and configuration details, creates patch files by performing a git diff on the updated YAML (with applied markup and decoded Helm blocks), and stages the resulting patches.
- File Operations Module:
  Manages copying generated files to designated staging locations.
- Workflow Orchestration Module:
  Coordinates the overall process by invoking the encoding/decoding, YAML markup, file operations, and code synchronization modules sequentially, ensuring a streamlined upgrade workflow.
- Webserver and UI Module:
  Provides a web-based interface that enables users to select upstream target versions, branches, and configure updates. This module also allows users to trigger the orchestrated upgrade process and ultimately generate the corresponding code review. This service is designed to run in a Docker container.
- Package Configuration Module:
  Allows configuration of additional package changes such as image versions, changelogs, and upstream package URLs.

The overall process, using these components, results in an automated generation of a code commit that incorporates all necessary changes for the upgrade. This commit can then be reviewed (via Gerrit) and integrated into the CI/CD pipeline.

Data Model Impact¶

Markup Language Grammar: A new YAML schema file will define the grammar of the markup language, specifying the types of changes (e.g., add, update, delete) applicable to Helm templates and other YAML files.
Versioning: The schema file will use semantic versioning to ensure backward compatibility and support future enhancements.
Configuration File Updates: There are no changes to other configuration files; the impact is confined to the introduction and modification of the configuration files managed by the toolset.

REST API Impact¶

The toolset primarily functions as a set of standalone scripts.
The toolselt will be hosted and run in a Docker container, facilitating deployment and scalability.
There are no anticipated changes to existing upstream APIs, as this functionality is encapsulated within the toolset.

Security Impact¶

There is no significant additional security impact introduced by this toolset.
Standard security practices, including code reviews and rigorous testing, will be maintained as part of the development and deployment processes.
Existing security policies and best practices for handling YAML files, Docker containers, and web services will continue to be enforced.

Other End User Impact¶

The toolset primarily targets automation of upstream updates and does not directly alter end-user interfaces.
Automated and standardized updates result in clearer code review diffs, aiding developers and administrators in understanding applied changes.
When deployed as a web service, the UI provides an accessible interface for monitoring update processes and reviewing generated code changes.

Performance Impact¶

The toolset is designed to operate as part of pre-deployment automation and is not expected to introduce performance overhead.
Operations such as pulling code, applying YAML modifications, and generating code reviews are lightweight.
Performance optimization efforts will focus on ensuring responsiveness of the web UI and efficient YAML processing.

Other Deployer Impact¶

Deployers can integrate the toolset into existing workflows with minimal infrastructure changes.
The Docker-based deployment model facilitates seamless integration into containerized environments.
The consistent and automated update mechanism reduces manual intervention, streamlining deployment and maintenance processes.

Developer Impact¶

The toolset standardizes the process for applying upstream modifications, reducing manual effort and error potential.
Developers benefit from increased visibility into changes via structured code reviews and automated patch generation.
A modular design supports easy testing, maintenance, and future extensibility of the toolset.

Upgrade Impact¶

Automation reduces manual overhead during upgrades, ensuring modifications are applied consistently across upstream releases.

Implementation¶

The implementation will proceed in the following phases:

Design & Prototyping:
- API Design for Core and Supporting Modules:
Develop API interfaces for the core modules, including:
Orchestrator/Actuator Module:

read_config(config_file: str) -> dict
Reads a YAML-based configuration file specifying which modules to invoke.

execute_tasks(config: dict) -> None
Invokes the specified modules to execute upgrade tasks.

Configuration Module:

load_orchestrator_config(config_path: str) -> dict
Loads configuration parameters for the orchestrator.

Upstream/Local Repo Version Checker:

check_versions(local_repo_dir: str, upstream_repo: str) -> List[str]
Returns a list of available upstream versions based on the local repository and upstream registry.

Git Operations Module:

sync_code(repo_path: str) -> None
Synchronizes the local repository with upstream.

create_commit_message(versions: dict, config_info: dict) -> str
Generates a commit message based primarily on the version updates and additional configuration details.

create_patch_files(yaml_output: str) -> List[str]
Creates patch files by performing a git diff on the updated YAML (with applied markup and decoded Helm blocks) and stages the resulting patches.

Gerrit Operations Module:

create_review(commit_message: str, patch_files: List[str]) -> str
Creates a Gerrit review and returns the review ID.

set_reviewers(review_id: str, reviewers: List[str]) -> None
Sets reviewers for the created review.

mark_as_wip(review_id: str) -> None
Marks the review as work-in-progress.

File Operations Module:

move_files(source: str, destination: str) -> None
Moves generated files to designated staging areas.

Helm/YAML Updater Module:

update_yaml(file_path: str, changes: dict) -> None
Applies schema-driven transformations to update YAML files.

encode_helm_template(file_path: str) -> str
Encodes a Helm template into YAML.

decode_helm_template(yaml_content: str) -> str
Decodes YAML content back into a Helm template.

Web Service Module:

Expose REST endpoints (e.g., /select, /update, /config, /review) to:

Select upstream target versions and branches.

Trigger the orchestrator module.

Retrieve the generated code review output.
- Develop data flow and API interaction diagrams for each module.
- Prototype the core encoding/decoding and YAML markup functionalities with sample inputs.
Development:
- Implement the core components (Helm template encoding/decoding and YAML markup transformation) and integrate the designed APIs.
- Develop supporting modules for code synchronization (using Git and Gerrit commands), file operations (for staging outputs), workflow orchestration (sequentially invoking modules), and the web service (providing a unified UI and REST API).
- Integrate API calls across modules as specified in the design.
- Include hooks for both manual and automated testing.
Testing & Validation:
- Conduct unit tests for each module, including tests for all API endpoints.
- Perform integration testing (from upstream code pull to code review generation) to verify that the intended changes and reviews are correctly produced.
- Validate the automated generation of code reviews and YAML transformations.
- Support both manual and automated testing processes.
Documentation & Review:
- Create comprehensive documentation, including an API reference and data flow diagrams.
- Developer guides will define the API for each module.
- Follow the current review process for peer reviews, with provisions for further automation in testing and integration reviews.

Command Line Tool Interface¶

The tool provides a command-line interface to perform encoding and decoding operations on YAML files containing Helm templates. The help section of the tool outlines the usage as follows:

Usage: python3 helm_block_yaml.py [-v|–verbose] <encode|decode> input.yaml output.yaml

Where: - encode: Transforms multi-line Helm blocks into Base64 encoded placeholders within the YAML. - decode: Converts the encoded placeholders back into their original multi-line Helm block format. - The -v or –verbose flag enables verbose debug output.

Extended Command Line Options¶

In addition to the basic encode/decode operations, the tool supports extended options for updating YAML files:

–input-dirs: Specifies one or more directories containing YAML files to be processed.
–output-dir: Designates the directory where updated YAML files will be saved.
–markup-file: Specifies the YAML markup file that defines the transformation rules.
–prefix: (Optional) A prefix for output file names.
–exclude-types: (Optional) A list of resource types to exclude from processing (e.g., “Namespace”, “Pod”).
–single-output: (Optional) If provided, all updated YAML documents are output to a single file, separated by document markers.

Assignee(s)¶

Primary Assignee: gleb.aronsky@windriver.com Other Contributors: - Wey Yi Guy – wey-yi.guy@windriver.com - Sandhya Kalisetty – sandhya.kalisetty@windriver.com - Joshua Kraitberg – joshua.kraitberg@windriver.com

Repos Impacted¶

starlingx/utilities
Other application repositories will have commits with auto-generated code

Work Items¶

Define and document the markup language grammar.
Develop and test Helm template encoding/decoding functionality.
Implement the YAML markup transformation module.
Integrate code synchronization and file operations modules.
Develop the web server and UI components.
Build supporting modules for Gerrit and Git interactions.
Prepare integration tests and comprehensive documentation.

Dependencies¶

Python 3.9+
PyYAML and ruamel.yaml library for YAML processing
Helm CLI (for compatibility checks and validations)
Docker (for containerized deployment)
Standard logging and configuration libraries

Testing¶

Unit Testing:
Ensure all functions in the encoding/decoding and YAML markup modules operate as expected using frameworks such as pytest or unittest.
Integration Testing:
Test the complete workflow—from upstream code pull to code review generation (not a full CI/CD pipeline)—to verify that the intended changes and reviews are correctly produced.
Edge Case Testing:
Validate the toolset’s behavior with malformed YAML inputs, unexpected Helm template structures, and error conditions.

Examples¶

Below are simplified examples that illustrate the transformation process.

Original YAML¶

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 3              # Original replica count
  template:
    spec:
      containers:
        - name: my-container
          image: "myimage:old"  # Original image reference

Markup YAML¶

Deployment:
  direct_paths:
    spec.replicas: helm:.Values.newReplicas      # Update replicas using helm variable
    spec.template.spec.containers[0].image: helm:.Values.newImage # Update container image using helm variable

Final Output YAML¶

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: {{ .Values.newReplicas }}  # Replica count updated via markup
  template:
    spec:
      containers:
        - name: my-container
          image: {{ .Values.newImage }}  # Container image updated via markup

Encoded Helm YAML Example with Markup Update¶

Original YAML with Helm Block:

apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-config
data:
  config: |-
    {{- if .Values.enableFeature }}
    feature: enabled
    timeout: 30                 # Original timeout value
    {{- else }}
    feature: disabled
    timeout: 60
    {{- end }}

Markup YAML:

ConfigMap:
  helm_block_updates:
    - description: "Update timeout value"
      search_path: "timeout:"                   # Identify lines with timeout
      search_conditions:
        - key: "timeout"
          contains: "30"                     # Apply update when timeout is '30'
      update:
        key: "timeout"
        new_value: "45"                      # New timeout value

Encoded YAML (After Applying Markup):

apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-config
data:
  config: "__HELM_BLOCK__2:BASE64_ENCODED_UPDATED_DATA__END_HELM_BLOCK__"

Explanation: The helm block in ‘data.config’ is now encoded. ‘2’ indicates the stored indent. ‘BASE64_ENCODED_UPDATED_DATA’ is the Base64 encoding of the helm block after the timeout value has been updated via markup.

Decoded YAML (Final Output):

apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-config
data:
  config: |-
    {{- if .Values.enableFeature }}
    feature: enabled
    timeout: 45                 # Timeout value updated via markup
    {{- else }}
    feature: disabled
    timeout: 60
    {{- end }}

Explanation: The encoded helm block is decoded back into its original multi-line format. The timeout value is now ‘45’ as specified in the markup YAML.

Application Flow¶

The overall workflow of the application is as follows:

Read Configuration: The application reads a YAML configuration file that defines which modules to invoke and provides parameters such as repository paths and update settings.
Repository Check: It examines both the local repository and the upstream repository/registry to determine the current state.
Fetch Upstream Versions: The tool queries the upstream repository to retrieve a list of available versions (typically the latest tagged releases).
User Selection: Available upstream versions are presented to the user via a command-line interface or web UI. The user selects the desired target version and branch.
Pull Upstream Code: The selected upstream code is pulled locally, forming the basis for the update.
Process Helm Templates: The Helm/YAML updater module encodes/decodes Helm templates as needed, ensuring that any embedded Helm blocks are properly processed before further transformations.
Apply YAML Markup: The YAML markup module applies schema-driven transformations (add, update, delete) to update the YAML files based on the provided configuration.
Process Other Application Files: The tool processes supplementary artifacts—such as changelogs, metadata files, and additional configuration documents—applying necessary updates as defined in the configuration.
Generate Patch Files: The Git Operations module computes differences between the updated and original code and generates patch files via git diff.
Stage Files: The File Operations module moves the generated patch files and updated artifacts to designated staging areas.
Create Code Review: Finally, the Gerrit Operations module creates a code review with a commit message (constructed based on version updates and configuration details) that is ready for integration and further testing.

Overview:

[Read Config]
↓

[Repo Check]
↓

[Fetch Upstream Versions]
↓

[User Selection]
↓

[Pull Upstream Code]
↓

[Process Helm Templates]
↓

[Apply YAML Markup]
↓

[Process Other Application Files]
↓

[Generate Patch Files]
↓

[Stage Files]
↓

[Create Code Review]

Documentation Impact¶

Developer guides will be updated to include detailed instructions for using the toolset.
End-user documentation will outline configuration and operation procedures for both the command-line and web UI interfaces.
The documentation will define the API for each module, including interfaces for orchestration, Git/Gerrit interactions, YAML markup, and Helm template encoding/decoding.
CI/CD pipeline documentation will be revised to incorporate the automated code review output.

References¶

History¶

Revision 1.0: Initial creation of the spec with detailed module APIs, data flow diagrams, and example transformations.

Revisions¶

Revision 1.0: Initial creation.