Etcd Certificates¶
Etcd is the consistent and highly-available key value store used as Kubernetes backing store for all cluster data. Interaction with etcd service is by secure HTTPs where x509 certificates are required.
Etcd certificates in StarlingX includes:
Etcd Root CA certificate
Etcd server certificate
Etcd client certificate
kube-apiserver
’s etcd client certificate
etcd Root CA certificate
This is the etcd Root CA certificate. It signs etcd server and client
certificates, and kube-apiserver
etcd client certificate. This is also the
CA certificate used to verify various server and client certificates signed
by etcd Root CA certificate.
Etcd Root CA certificate and corresponding private keys are stored in file systems:
/etc/etcd/ca.crt
/etc/etcd/ca.key
etcd server certificate
This is the etcd server’s serving certificate. Services such as
kube-apiserver
that access etcd verify this serving certificate with etcd
Root CA certificate.
Etcd server certificate and corresponding private keys are stored in file systems:
/etc/etcd/etcd-server.crt
/etc/etcd/etcd-server.key
etcd client certificate
This is a client certificate generated from etcd Root CA. It can be used by clients to identify itself when connecting to etcd by HTTPS.
Etcd client certificate and corresponding private key are stored in file system:
/etc/etcd/etcd-client.crt
/etc/etcd/etcd-client.key
kube-apiserver’s etcd client certificate
This is the kube-apiserver
client certificate generated from etcd Root CA. It
is used by kube-apiserver
to identify itself when connecting to etcd by HTTPS.
Kube-apiserver’s etcd client certificate and corresponding private keys are stored in file systems:
/etc/kubernetes/pki/apiserver-etcd-client.crt
/etc/kubernetes/pki/apiserver-etcd-client.key
Install custom etcd Root CA certificate¶
By default, etcd Root CA certificate and corresponding private key are generated during system bootstrap and have 10 years validation.
The other etcd certificates are generated from the Root CA certificate during system bootstrap once the Root CA certificate is generated and available. The following generated certificates have 1 year validation:
etcd-client
etcd-server
kube-apiserver-etcd-client
To provide a Root CA for etcd, add the etcd_root_ca_key
and
etcd_root_ca_cert
overrides to localhost.yml
before bootstrap.
For example:
etcd_root_ca_key: /home/sysadmin/<my-root-ca-key.pem>
etcd_root_ca_cert: /home/sysadmin/<my-root-ca-cert.pem>
Note
The values must be absolute file paths.
Both key and cert must be provided (or omitted).
The certificate should be valid for 5-10 years as currently there is no mechanism to update this certificate.
Update/Renew etcd certificates¶
Updating etcd Root CA certificate is a complex process, because it is not only the Root CA certificate need to be updated, but also all the other etcd certificates signed by it need to be regenerated and updated. The update of the etcd Root CA is currently not supported.
The other child certificates generated from the etcd Root CA are monitored by a cronjob, which runs every day at midnight to check if any of these certificates’ expiry date is approaching and renew them if the expiry date is within 15 days.
If the renewal fails, a 250.003 alarm will be raised:
Kubernetes certificates have been renewed but not all services have been updated.
For this alarm, controller nodes need to lock/unlock for the services to take the new certificates.
Kubernetes certificates renewal failed.
For this alarm, the Kubernetes certificates need to be renewed manually, during which services need to restart.
If this alarm is raised, the administrator should follow the recommended action for the specific alarm.