Certificate Management with cert-manager

Storyboard: https://storyboard.openstack.org/#!/story/2007360

This story introduces cert-manager for certificate life-cycle management on the StarlingX platform.

Problem Description

Various StarlingX platform certificates are currently stored on the host filesystem, requiring manual creation, updates and renewal monitoring. Cert-manager allows for automated certificate issuance via various sources in a cloud-native Kubernetes environment. Cert-manager also has built-in support for monitoring expiry dates & configurable auto-renewal process.

Cert-manager supports self-signed certificates, generating certification signing requests (CSRs) and validating certificates via Certificate Authorities (CA) using the ACME protocol, storage/retrieval of certificates in a Vault etc. Cert-manager is an open source project from Jetstack. More info can be found here: https://cert-manager.io/docs/

Use Cases

  • End user wants the platform & hosted application’s certificates managed.

  • End user wants certificates auto-renewed with no manual intervention.

  • End user wants a flexibility of various CAs for certificates used by platform and hosted applications.

Proposed change

  • Packaging & installation

StarlingX will contain the Armada manifest and cert-manager helm charts, which will be uploaded and applied to pull the container images during installation from a public registry. A new Armada app will be introduced as part of this proposed change.

The cert-manager app will be auto-installed at the end of bootstrap process using Ansible playbook. A new role will be introduced at the end of the current bootstrap process (post the ‘bringup-essential-services’ stage – which installs Kubernetes, Helm, and essential networking apps). The new role will be called ‘bringup-bootstrap-applications’, which will install cert-manager as part of this work item. In future, other bootstrap-time applications may be appended to this list.

The user will be provided the ability of overriding the default settings of bootstrap applications via the localhost.yml file. This override details may be embedded in the yaml file directly or linked to a separate yaml.

  • Ingress controller

Cert-manager requires an ingress controller to serve HTTP-01 challenges from a CA. An ingress controller Armada application will be included as part of the StarlingX platform that will be automatically uploaded and applied to serve such requests. During the bootstrap process, ingress controller will be serialized and started before Cert-manager.

Note that an Nginx ingress controller is already present as part of StarlingX packaged with the Stx-Openstack application. Proposal here is to remove it from Openstack app and make it part of the StarlingX platform instead.

  • Cert-manager integration for platform certificates

Issuers and certificates will be created via cert-manager CRDs on the system. Depending on the certificate issuance methodology, there may be some delay involved in getting the CSR signed through an external authority (between a few seconds to a few minutes). The methodology between containerized and non-containerized applications will differ at this point where containerized apps will read the certificate (stored in a Kubernetes secret) directly via a volume mount inside the app. For non-containerized apps (such as haproxy, docker etc) the certificate will be copied from the Kubernetes secret to /etc/ssl/certs folder, app is restarted and an audit timer is setup to check for certificate renewal. When certificate is auto-renewed by cert-manager, the audit timer will replace the existing certificate & restartthe target application.

Alternatives

Cert-manager is the only tool that provides the full flexibility needed for full lifecycle certificate management in a cloud-native Kubernetes environment. While tools such Hashicorp Vault have some functionality for certificate management, Cert-manager provides greater flexibility of usage such as LetsEncrypt CA, Vault CA, Venafi CA etc.

Data model impact

Current certificate installation is performed via the CLI command ‘system certificate install -mode ssl <pem.file>’.

With the introduction of cert-manager, issuers will be defined in manifest files and certificates created will not need a manual installation by user.

REST API impact

None

Security impact

This feature introduces automation of certification management on the platform. Certificates handled by cert-manager are stored in Kubernetes Secret by default. Some of these certificates will be copied to the host filesystem to support current deployment model of non-containerized (host native) applications.

Other end user impact

None

Performance Impact

None

Other deployer impact

Ansible playbook will be updated to install cert-manager during the bootstrap process.

Developer impact

None

Upgrade impact

None

Implementation

Assignee(s)

Primary assignee:

  • Sabeel Ansari

Repos Impacted

  • cert-manager-armada-app

  • ansible-playbooks

Work Items

  • Create new repo for the new application ‘nginx-ingress-controller’, define required manifests & import helm charts for app

  • Create new repo for the new application ‘cert-manager’, define required manifests & import helm charts for app

  • Add new ansible role ‘bringup-bootstrap-applications’

  • Define Issuers, ClusterIssuers, Certificates for migrating platform app certificates.

  • For non-containerized apps, extract certificate & keys, copy to filesystem and setup auditing mechanism to refresh certificates.

Dependencies

  • Kubernetes cluster

Testing

  • The user should be able to have cert-manager manage certificates without any manual intervention.

  • User overrides should be available for various parameters such as different issuers, CAs, certificate overrides etc.

  • In case of communication failure with external CA, cert-manager should gracefully handle certificate management (self-signed certificates?).

  • All platform certificates should be managed by cert-manager.

Documentation Impact

Documentation to be updated with user-override configuration parameters and usability of cert-manager in StarlingX.

References

History

Revisions :header-rows: 1

Release Name

Description

stx-4.0

Introduced