Update default certificate configuration to use cert-manager on install

Storyboard: https://storyboard.openstack.org/#!/story/2009811

This story updates the default certificate configuration to use cert-manager on fresh StarlingX installation. The story covers installs for both standalone and distributed cloud systems.

Key Objectives:

  • Enable HTTPS by default on all core platform APIs

  • Use cert-manager for managing all default platform certificates, in order to simplify certificate management (e.g. auto-renewals)

  • Use a common/single auto-generated (via cert-manager) local root CA for signing all default certificates, such that external clients need to only trust a single additional Root CA for accessing all StarlingX APIs securely

  • Use the same naming and certificate hierarchy in system controller and subclouds

Note: This story is a continuation of https://storyboard.openstack.org/#!/story/2007361 which introduced the use of cert-manager for managing REST API, registry, and oidc certificates.

In stx.6.0:

  • Only the K8S API and registry.local API is configured as HTTPS by default; the StarlingX REST APIs and Horizon are configured as HTTP by default

  • The default certificate configurations do not use cert-manager on fresh installs. The user has to configure the system explicitly to use cert-manager to create/manage platform certificates.

Problem description

The now recommended cert-manager based certificate management is not used for platform certificates from initial setup. That results in certificates, such as the registry.local certificate, not taking advantage of features like auto renewal, which are offered by cert-manager.

By default, these platform certificates (StarlingX REST API/GUI and registry.local certificates) are auto generated, self-signed certificates. That means they are static and need to be re-configured manually via system certificate-install when they approach expiration or manually reconfigured to use cert-manager [2] to be auto renewable.

Additionally, this initial setup, does not enable HTTPS on all interfaces. Interfaces such as StarlingX REST API / GUI are left as HTTP only and need to be manually switched, with system modify –https_enabled true [5], to be secured with HTTPS.

Use Cases

  • System bootstrap

Proposed change

This story addresses this issue by updating system bootstrap to:

  • Create a system-local-ca ‘tls’ type kubernetes secret to be the top of the platform certificate chain

  • Create a cert-manager ClusterIssuer that uses system-local-ca for signing all platform certificates

This system-local-ca will be auto-populated with the kubernetes Root CA certificate and key [1], such that the default platform certificate configuration will use a single Root CA for all Platform Certificates (i.e. k8s, starlingx, registry). Note that the kubernetes Root CA is either auto-generated by the system at startup, or specified by user as a bootstrap playbook override. [4]

Also at system bootstrap, the platform certificates (StarlingX REST API/GUI and registry.local certificates) will be created using cert-manager Certificates, with the system-local-ca specified as the Issuer to sign them.

Also, from system bootstrap, HTTPS will be enabled by default for communication across platform rest APIS. Interfaces such as StarlingX REST API / GUI will be HTTPS by default.

More important, this change encourages users to use and take advantage of cert-manager which is now recommended for certificate management in the platform.

Alternatives

Have system-local-ca and the kubernetes Root CA be different ICA certificates and another Root CA at the very top of both.

This option was considered but discarded for now as it needs careful consideration. Also the hierarchy proposed here with cert-manager can be easily adapted to have another ClusterIssuer at the top, if we decide to evolve in that direction.

Data model impact

None

REST API impact

No impact to REST API schema specification. However, connections are now secured by HTTPS.

Security impact

  • Enhanced security by having platform certificates managed by cert-manager by default

  • Enhanced security as cert-manager makes it easier to have certificates with shorter durations

  • Enhanced security by enabling platform certificates auto renewal

  • Enhanced security by having https enabled by default

  • Simpler external clients connection as they need to only trust a single additional Root CA for accessing all StarlingX APIs securely

Other end user impact

The default horizon URL [6] will change to HTTPS protocol and port: https://<oam-floating-ip-address>:8443

Performance Impact

Most of the changes for this feature are in the system bootstrap with little impact on performance. After bootstrap, no performance impact is expected.

Other deployer impact

None

Developer impact

Third party automated scripts connecting to StarlingX REST APIs may need to be updated in order to use the HTTPS connection.

Upgrade impact

None

Implementation

Assignee(s)

Primary assignee:

  • Rei Oliveira (rjosemat)

Repos Impacted

Impacted repo from this spec:

  • config

  • stx-ansible

Work Items

  • Make cert-mon aware of secrets created before its startup

  • Create system-local-ca issuer and platform certificates as part of ansible bootstrap playbook

  • Auto-addition of Root CA values to sucloud bootstrap overrides based on system-controller’s (dcmanager subcloud add)

  • Add support to auto generate Root CA to migrate-platform-certificate playbook - Story 2007361, task 44036 [3]

  • Developer testing - bootstrap / CA update on different system configurations

System Upgrade

None

Dependencies

None

Testing

The feature must be tested in the following StarlingX configurations:

  • Standalone

  • Distributed Cloud

The test can be performed on hardware or virtual environments.

Testing must consist of:

  • System installation / boostrap

  • Certificate renewals

  • Root CA updates after bootstrap

Documentation Impact

This story affects: * The StarlingX installation / bootstrap documentation. * Horizon access documentation[6]_ will change to HTTPS protocol and port: https://<oam-floating-ip-address>:8443

References

History

Revisions

Release Name

Description

stx-7.0

Introduced