Containerization: Helm Chart Override Generation¶
Storyboard: #2003909
This spec is part of the larger effort to containerize the StarlingX infrastructure. Please see the background information regarding this effort on the StarlingX wiki.
For managing the life-cycle of containerized services, StarlingX will be using two open source technologies: Helm, the Kubernetes package manager and Armada, a tool for managing multiple Helm charts with dependencies.
This specification will outline the use of Helm in a StarlingX containerized platform and the need to generate system compatible Helm chart overrides.
Armada integration and use will be discussed in an additional Armada integration story.
Problem description¶
Each StarlingX native service that will be containerized shall be described by a Helm chart.
The Helm chart for a given service will be built from a chart provided by one of the following open source projects: openstack-helm (for openstack services), openstack-helm-infra (for openstack support services), external-storage (for volume provisioners), and StarlingX (nova-api proxy, fault manager, etc)
The current set of native services that will be containerized are: AODH, Ceilometer, Cinder, Fault Manager, Glance, Gnocchi, Heat, Horizon, Ironic, Keystone, Libvirt, Magnum, Murano, Neutron, Nova, Nova-API Proxy, Panko, and Rabbitmq.
To support these services, the following charts and corresponding services are also required: Ingress, MariaDB, Memcached, and an RBD volume provisioner.
A chart comes with a default set of values that will describe the behavior of the service installed within the Kubernetes cluster. Helm provides a mechanism to override these default values so the service can be updated based on the desired service configuration.
In the current StarlingX code-base, native services are configured via puppet modules with supplied hiera data derived from the contents of the system inventory database. The values in the database reflect the current physical components of installation along with the configuration choices of the Deployer.
In order to migrate the native services to containerized services, the following problems need to be addressed:
Need a sysinv based framework to examine the system inventory database and generate chart specific system overrides for each native service.
The framework should be easily extensible for adding chart support.
Need an API for the Deployer to easily override or complement the system generated overrides for a specific chart
Need an API to generate the final overrides for a specific chart or set of charts. The final generated set of overrides should be overrides with the following superseding priority: default values (base) <- system generated <- Deployer provided.
It should be noted that this does not preclude any Helm chart from being installed on the system and used with standard Helm mechanisms for managing the life-cycle of a chart release.
Use Cases¶
The Deployer needs to be able manage and generate the helm overrides for a given Helm chart. The following API actions need to be enabled
List system supported helm charts.
Show overrides for a chart.
Delete overrides for a chart.
Update helm chart user overrides.
Proposed change¶
In order to properly configure a Helm chart for a containerized service the following is proposed:
Provide a plugin framework as part of sysinv to allow adding support for new charts as required.
Provide a basic plugin for each chart in the openstack-helm project.
Provide specific plugin logic to generate chart overrides for each of the currently supported StarlingX native services.
With the plugin framework and chart plugins deployed as part of sysinv, a mechanism is required to generate the overrides for a specific chart. For this the following is proposed:
Implement sysinv API/CLI for managing system and Deployer provided overrides:
helm-override-list - List system helm charts.
helm-override-show - Show overrides for a chart.
helm-override-delete - Delete overrides for a chart.
helm-override-update - Update helm chart user overrides.
Additional options will be provided to the helm-overrides-show CLI to extract the combined overrides in a yaml form to feed into Helm or Armada.
–format [ helm | armada ]
Specifying armada will add the Armada compliant header to the yaml output.
–file=<filename>
An optional parameter, provided with the use of the –format option will generate a yaml file containing the overrides. If this option is not specified, then the resulting yaml file will be delivered to stdout.
Alternatives¶
An alternative approach would be to leave the override configuration for each charts as an exercise up to the Deployer using standard Helm and/or Armada mechanisms. This approach would increase initial system provisioning times and would lead to a hit-or-miss integration strategy to find the correct overrides for a give configuration.
Data model impact¶
To support managing chart overrides, a new table will be added to the sysinv DB, with the following columns:
created_at, updated_at, deleted_at: standard columns
id: primary key
name: chart name
namespace: chart namespace
user_overrides: store deltas as compared to system overrides
unique constraint is a combination of name and namespace
As usual, a new data migration will be added to create this table in the sysinv DB.
REST API impact¶
This impacts the sysinv REST API:
The new resource /helm_charts is added and the GET method would return all the charts that provide system overrides along with their namespaces.
URLS:
/v1/helm_charts/
Request Methods:
GET
JSON response example:
{"charts": [ {"name": "ingress", "namespaces": ["kube-system", "openstack"]}, {"name": "rbd-provisioner", "namespaces": ["kube-system"]}, {"name": "openvswitch", "namespaces": ["openstack"]}, {"name": "rabbitmq", "namespaces": ["openstack"]}, {"name": "libvirt", "namespaces": ["openstack"]}, {"name": "heat", "namespaces": ["openstack"]}, {"name": "keystone", "namespaces": ["openstack"]}, {"name": "nova", "namespaces": ["openstack"]}, {"name": "horizon", "namespaces": ["openstack"]}, {"name": "cinder", "namespaces": ["openstack"]}, {"name": "glance", "namespaces": ["openstack"]}, {"name": "mariadb", "namespaces": ["openstack"]}, {"name": "memcached", "namespaces": ["openstack"]}, {"name": "neutron", "namespaces": ["openstack"]}]}
By providing the GET method a specific chart name along with key/value pair specifying a namespace, the values of a chart can be retrieved
URLS:
/v1/helm_charts/{name}?namespace={namespace}
Request Methods:
GET
Example:
Request: GET /v1/helm_charts/rabbitmq?namespace=openstack
JSON response example:
{"namespace": "openstack", "name": "rabbitmq", "system_overrides": "pod:\n replicas: {server: 1}\n", "user_overrides": "", "combined_overrides": "pod:\n replicas:\n server: 1\n"}
Using the PATCH method will allow the Deployer to update override values
URLS:
/v1/helm_charts/{name}?namespace={namespace}
Request Methods:
PATCH
Example:
Request: PATH /v1/helm_charts/rabbitmq?namespace=openstack
JSON request example:
{"flag": "reuse", "values": {"files": [], "set": ["pod.replicas=2"]}
JSON response example:
{"user_overrides": "pod:\n replicas: 2\n", "namespace": "openstack", "name": "rabbitmq"}
Using the DELETE method will allow the Deployer to delete all Deployer provided overrides. Once deleted, the system generated override will still be applied for the chart.
URLS:
/v1/helm_charts/{name}?namespace={namespace}
Request Methods:
DELETE
Example:
Request: DELETE /v1/helm_charts/rabbitmq?namespace=openstack
Security impact¶
Passwords may be provided in the overrides. Considerations need to be made when displaying/setting these override values. One potential solution is to prevent overriding these values via the API.
Other end user impact¶
As mentioned in the Proposed change section, new CLI commands will be provided and will behave as follows:
helm-override-list - List system helm charts.:
$ system helm-override-list +-----------------+--------------------------------+ | chart name | overrides namespaces | +-----------------+--------------------------------+ | ceilometer | [u'openstack'] | | cinder | [u'openstack'] | | glance | [u'openstack'] | | gnocci | [u'openstack'] | | heat | [u'openstack'] | | horizon | [u'openstack'] | | ingress | [u'kube-system', u'openstack'] | | keystone | [u'openstack'] | | libvirt | [u'openstack'] | | mariadb | [u'openstack'] | | memcached | [u'openstack'] | | neutron | [u'openstack'] | | nova | [u'openstack'] | | openvswitch | [u'openstack'] | | rabbitmq | [u'openstack'] | | rbd-provisioner | [u'kube-system'] | +-----------------+--------------------------------+
helm-override-update - Update helm chart user overrides.:
$ system helm-override-update rabbitmq openstack \ --reuse-values \ --set pod.replicas=2 +----------------+---------------+ | Property | Value | +----------------+---------------+ | name | rabbitmq | | namespace | openstack | | user_overrides | pod: | | | replicas: 2 | | | | +----------------+---------------+
helm-override-show - Show overrides for a chart.:
$ system helm-override-show rabbitmq openstack +--------------------+-------------------------+ | Property | Value | +--------------------+-------------------------+ | combined_overrides | pod: | | | replicas: 2 | | | | | name | rabbitmq | | namespace | openstack | | system_overrides | pod: | | | replicas: {server: 1} | | | | | user_overrides | pod: | | | replicas: 2 | | | | +--------------------+-------------------------+helm-override-delete - Delete overrides for a chart.:
$ system helm-override-delete rabbitmq openstack Deleted chart overrides for rabbitmq:openstack
Performance Impact¶
Minimal impact to the system controllers is to be expected. The proposed changes to generate and manage the chart overrides will access the sysinv database and execute helm commands to determined the combined overrides.
Other deployer impact¶
The API and ability to extract the combined overrides will enable Armada to be integrated as a mechanism to manage and launch all the current bare metal native service as a kubetnetes application.
Developer impact¶
Developers working in StarlingX will need to use the API changes provided here to view, adjust, and generate chart overrides for containerized service deployment via Helm.
Upgrade impact¶
None. This is the first StarlingX release with Helm support. No previous release supports this feature and no upgrade support is required.
Implementation¶
As the system inventory database contains all the pertinent information about the installed system, changes will be required in stx-config/sysinv to generate the overrides for the Helm charts and provide the APIs to access them.
The implementation will duplicate the plugin framework and associated constructs used for extracting and generating the puppet hiera for existing native services.
Once fully implemented, the puppet manifests for the native services and the plugins that generated the hiera data for those manifests will be removed. In their stead, Helm chart overrides will be generated and provided Armada for launching the containerized services.
Assignee(s)¶
Primary assignee:
Robert Church (rchurch)
Other contributors:
Chris Friesen (cbf123)
Gerry Kopec (gerry-kopec)
Joseph Richard (josephrichard)
Tyler Smith (tyler.smith)
Angie Wang (angiewang)
Irina Mihai (irina.mihai.wrs)
Ovidiu Poncea (ovidiu.poncea)
Al Bailey (albailey1974)
Lachlan Plant (lachlan.plant)
Repos Impacted¶
stx-config
Work Items¶
stx-config/sysinv:
Provide a plugin framework as part of sysinv to allow adding support for new charts as required.
Provide a basic plugin for each chart in the openstack-helm project.
Provide specific plugin logic to generate chart overrides for each of the currently supported StarlingX native services.
Implement API changes to support listing supported charts that provide system overrides.
Implement API changes to support showing the system, Deployer, and combined overrides.
Implement API changes to support adding and modifying Deployer provided overrides.
Implement API changes to support deleting Deployer provided overrides.
stx-config/cgts-client:
Implement sysinv CLI for managing system and Deployer provided overrides by calling the sysinv API:
helm-override-list - List system helm charts.
helm-override-show - Show overrides for a chart.
helm-override-delete - Delete overrides for a chart.
helm-override-update - Update helm chart user overrides.
Add helm-override-show options for generating the final view of overrides
Add: –format [ helm | armada ]
Add: –file=<filename>
Dependencies¶
This requires new functionality being developed under the following stories:
Testing¶
The following testing will be performed in association with these proposed changes:
The sysinv REST API will be exercised using curl to verify/validate it’s operation. It will be used for documenting API access.
The cgts-client commands will be exercised along with any supported options.
The resulting combined overrides yaml files will be used with Helm and Armada to ensure compatibility.
Documentation Impact¶
This story affects the following StarlingX documentation:
Installation and configuration of containerized services.
sysinv REST API documentation.
Specific details of the documentation changes will be addressed once the implementation is complete.
References¶
References are provided throughout this document at the point when terms or items are introduced. No additional references are needed at this time.
History¶
Release Name |
Description |
|---|---|
2019.03 |
Introduced |
