Local OIDC IdP Proxy Dex

oidc-auth-apps is a system-level application that serves as an OIDC IdP for authenticating users accessing the Kubernetes API and optionally the StarlingX API.

oidc-auth-apps uses the open-source Dex (dexidp.io) for OIDC authentication. Dex acts as an intermediary (or proxy) to other IdPs through connectors, allowing it to delegate OIDC authentication requests to external IdPs using other authentication protocols, such as LDAP (for local OpenLDAP and WAD) and OIDC.

The procedure below shows examples for configuring oidc-auth-apps (Dex) to proxy requests to StarlingX’s local LDAP IdP, a remote WAD IdP, and/or a remote OIDC IdP.

At install time, the oidc-auth-apps application is deployed and configured by default to use the local LDAP server. The only exception is the DC environments, where this LDAP server runs only on the System Controller’s controllers. It is not present in the subcloud’s controllers.

Use this section to re-configure oidc-auth-apps, for example, to change to use an external IdP (example: WAD or OIDC) rather than or in addition to the local LDAP server.

Prerequisites

Ensure that kube‑apiserver uses oidc-auth-apps for OIDC token validation (this is the default). If the default configuration has been modified, update the kube‑apiserver OIDC parameters before proceeding. For more information on configuring the Kubernetes kube-apiserver, see Configure Kubernetes for OIDC Token Validation while Bootstrapping the System or Configure Kubernetes for OIDC Token Validation after Bootstrapping the System.

Configure Frontend Dex OIDC

Perform the following steps to configure frontend Dex OIDC:

Procedure

1. Create certificates for OIDC servers.

   Certificates used by oidc-auth-apps can be managed by Cert-Manager. Doing so will automatically renew the certificates before they expire. The system-local-ca ClusterIssuer (see System Local CA Issuer) will be used to issue this certificate.

   Note

   If a signing CA is not a well-known trusted CA, ensure that the system trusts the CA by specifying it either during the bootstrap phase of system installation, by specifying ssl_ca_cert: <certificate_file> in the ansible bootstrap overrides localhost.yml file, or by using the system ca-certificate-install command. See System Trusted CA Certificates.

   Important

   The namespace for oidc-auth-apps resources must be kube-system.

   1. Create the OIDC client and IdP server certificate and private key pair.

      ~(keystone_admin)]$ cat <<EOF > oidc-auth-apps-certificate.yaml
      ---
      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: oidc-auth-apps-certificate
        namespace: kube-system
      spec:
        secretName: oidc-auth-apps-certificate
        duration: 2160h # 90 days
        renewBefore: 360h # 15 days
        issuerRef:
          name: system-local-ca
          kind: ClusterIssuer
        commonName: <OAM_floating_IP_address>
        subject:
          organizations:
            - ABC-Company
          organizationalUnits:
            - StarlingX-system-oidc-auth-apps
        ipAddresses:
        - <OAM_floating_IP_address>
      
      EOF
      

      Note

      The Certificate usage of Cert-manager Documentation (https://cert-manager.io/docs/usage/certificate/) states that one should “Take care when setting the renewBefore field to be very close to the duration as this can lead to a renewal loop, where the Certificate is always in the renewal period.”

      Considering the statement above, you must not set renewBefore to a value very close to the duration value, such as a renewBefore of 29 days and a duration of 30 days. Instead, you could set values such as renewBefore=15 days and duration=30 days to avoid renewal loops.

   2. Apply the configuration.

      ~(keystone_admin)]$ kubectl apply -f oidc-auth-apps-certificate.yaml
      

   3. Verify the configuration.

      ~(keystone_admin)]$ kubectl get certificate oidc-auth-apps-certificate –n kube-system
      

   4. Configure the OIDC-client with both the OIDC Client and IdP Server Certificate and its associated trusted CA certificate.

      Configure the certificate of the root CA that signed the OIDC client certificate and the IdP server certificate. In this example, it will be ca.crt of oidc-auth-apps-certificate created above.

      ~(keystone_admin)]$ cat <<EOF > stx-oidc-client.yaml
      tlsName: oidc-auth-apps-certificate
      config:
        # The OIDC-client container mounts the dex-ca-cert secret at /home, therefore
        # issuer_root_ca: /home/<filename-only-of-generic-secret>
        issuer_root_ca: /home/ca.crt
        issuer_root_ca_secret: oidc-auth-apps-certificate
      EOF
      
      ~(keystone_admin)]$ system helm-override-update oidc-auth-apps oidc-client kube-system --values stx-oidc-client.yaml
      

   5. Configure the secret observer to track changes.

      Change the cronSchedule according to your requirement. The cronSchedule controls how often the application checks to see if the certificate mounted on the Dex and the oidc-client pods had changed.

      Create a yaml configuration to modify the cronSchedule according to your requirement.

      The following example sets the schedule to every 15 minutes.

      ~(keystone_admin)]$ cat <<EOF > secret-observer-overrides.yaml
      cronSchedule: "*/15 * * * *"
      observedSecrets:
        - secretName: "oidc-auth-apps-certificate"
          filename: "ca.crt"
          deploymentToRestart: "stx-oidc-client"
        - secretName: "oidc-auth-apps-certificate"
          filename: "tls.crt"
          deploymentToRestart: "stx-oidc-client"
        - secretName: "oidc-auth-apps-certificate"
          filename: "tls.crt"
          deploymentToRestart: "oidc-dex"
      EOF
      

      Run the following command to update the overrides:

      ~(keystone_admin)]$ system helm-override-update oidc-auth-apps secret-observer kube-system --values secret-observer-overrides.yaml
      

2. Update the secret in the OIDC client.

   Add the following to the config: section of the stx-oidc-client.yaml Helm overrides file from the previous step:

   config:
     client_secret: <your-custom-secret-for-dex-client>
   

   Note

   This secret must match the secret in stx-oidc-client-app of the staticClients section of the dex-overrides.yaml file, in the sections on configuring backends.

   Update the stx-oidc-client.yaml overrides file again.

   ~(keystone_admin)]$ system helm-override-update oidc-auth-apps oidc-client kube-system --values /home/sysadmin/oidc-client-overrides.yaml --reuse-values
   

3. Re-apply oidc-auth-apps to apply the override changes:

   ~(keystone_admin)]$ system application-apply oidc-auth-apps
   

Configure Local LDAP Backend

Perform the following steps to configure local LDAP backend:

Procedure

1. Create a secret with the certificate of the CA that signed the certificate of the StarlingX local LDAP IdP.

   Use ca.crt from system-openldap-local-certificate.

   system-openldap-local-certificate will be used in the following step when configuring the Dex IdP server overrides.

2. Specify user overrides for the Dex IdP server.

   The dex-overrides.yaml file contains the desired Dex Helm chart overrides (that is, one or more connectors, in this case, StarlingX’s local LDAP, optional token expiry, and so on), and volume mounts for providing access to the required certificates. See example of the dex-overrides.yaml file for a local LDAP backend/connector below.

   For the complete list of Dex Helm chart values supported, see Dex Helm Chart Values. For the complete list of parameters of the Dex LDAP connector configuration, see Authentication Through LDAP. For list of default values for the Dex Helm chart, see Default Helm Overrides for OIDC Auth Apps application.

   The overall Dex documentation is available on dexidp.io. The configuration of Dex server version v2.44.0 is described on github (https://github.com/dexidp/dex/blob/v2.44.0/config.yaml.dist) with example config.dev.yaml (https://github.com/dexidp/dex/blob/v2.44.0/config.dev.yaml).

   The example below configures a token expiry of ten hours, an ldap type connector to StarlingX’s Local LDAP IdP using HTTPS / LDAPs using system-openldap-local-certificate, the required StarlingX’s local LDAP IdP’s administrator login information (that is, bindDN and bindPW), and example userSearch, and groupSearch clauses.

   The value of bindPW can be retrieved through keyring get ldap ldapadmin command run in the controller where the local LDAP server is running. In Distributed Cloud environment, the MGMT floating IP address to be used is the one from the System Controller.

   cat <<EOF > dex-overrides.yaml
   config:
     staticClients:
     - id: stx-oidc-client-app
       name: STX OIDC Client app
       secret: <your-custom-secret-for-dex-client>
       redirectURIs:
       - https://<OAM floating IP address>:30555/callback
     expiry:
       idTokens: "10h"
     connectors:
     - type: ldap
       name: LocalLDAP
       id: localldap-1
       config:
         host: <MGMT floating IP address>:636
         rootCA: /etc/ssl/certs/idpcert/ca.crt
         insecureNoSSL: false
         insecureSkipVerify: false
         bindDN: CN=ldapadmin,DC=cgcs,DC=local
         bindPW: [<password>]
         usernamePrompt: Username
         userSearch:
           baseDN: ou=People,dc=cgcs,dc=local
           filter: "(objectClass=posixAccount)"
           username: uid
           idAttr: DN
           emailAttr: uid
           nameAttr: gecos
         groupSearch:
           baseDN: ou=Group,dc=cgcs,dc=local
           filter: "(objectClass=posixGroup)"
           userMatchers:
           - userAttr: uid
             groupAttr: memberUid
           nameAttr: cn
   volumeMounts:
   - mountPath: /etc/ssl/certs/idpcert
     name: certdir
   - mountPath: /etc/dex/tls
     name: https-tls
   volumes:
   - name: certdir
     secret:
       secretName: system-openldap-local-certificate
   - name: https-tls
     secret:
       defaultMode: 420
       secretName: oidc-auth-apps-certificate
   

   ~(keystone_admin)]$ system helm-override-update oidc-auth-apps dex kube-system --values /home/sysadmin/dex-overrides.yaml
   

Configure Remote WAD Backend

Perform the following steps to configure remote WAD backend:

Procedure

1. Create the secret wad-ca-cert with the CA’s certificate that signed the Active Directory’s certificate using the following command:

   Note

   It is assumed below that you have obtained this certificate and have it in a local file wad-ca-cert.crt.

   ~(keystone_admin)]$ kubectl create secret generic wad-ca-cert --from-file=wad-ca-cert.crt -n kube-system
   

2. Specify user overrides for Dex IdP server:

   The dex-overrides.yaml file contains the desired Dex Helm chart overrides (that is, one or more connectors, in this case, a remote WAD), volume mounts for providing access to the required certificates. See example of the dex-overrides.yaml file for a WAD backend/connector:

   The example below configures a token expiry of ten hours, an ldap type connector to the remote WAD IdP using HTTPS / LDAPs, the required remote WAD IdP’s admin login information (that is, bindDN, and bindPW), and example userSearch, and groupSearch clauses.

   The following is the configuration for a WAD server:

   config:
     staticClients:
     - id: stx-oidc-client-app
       name: STX OIDC Client app
       redirectURIs: ['https://<OAM floating IP address>:30555/callback']
       secret: <your-custom-secret-for-dex-client>
     expiry:
       idTokens: "10h"
     connectors:
     - type: ldap
       name: WAD
       id: wad-1
       config:
         host: pv-windows-acti.windows-activedir.example.com:636
         rootCA: /etc/ssl/certs/idpcert/wad-ca-cert.crt
         insecureNoSSL: false
         insecureSkipVerify: false
         bindDN: cn=Administrator,cn=Users,dc=windows-activedir,dc=example,dc=com
         bindPW: [<password>]
         usernamePrompt: Username
         userSearch:
           baseDN: ou=Users,ou=Titanium,dc=windows-activedir,dc=example,dc=com
           filter: "(objectClass=user)"
           username: sAMAccountName
           idAttr: sAMAccountName
           emailAttr: sAMAccountName
           nameAttr: displayName
         groupSearch:
           baseDN: ou=Groups,ou=Titanium,dc=windows-activedir,dc=example,dc=com
           filter: "(objectClass=group)"
           userMatchers:
           - userAttr: DN
             groupAttr: member
           nameAttr: cn
   volumeMounts:
   - mountPath: /etc/ssl/certs/idpcert
     name: certdir
   - mountPath: /etc/dex/tls
     name: https-tls
   volumes:
   - name: certdir
     secret:
       secretName: wad-ca-cert
   - name: https-tls
     secret:
       defaultMode: 420
       secretName: oidc-auth-apps-certificate
   

   ~(keystone_admin)]$ system helm-override-update oidc-auth-apps dex kube-system --values /home/sysadmin/dex-overrides.yaml
   

Configure Remote OIDC Backend

You can use a Dex OIDC connector to configure an external OIDC IdP for Kubernetes authentication and optionally StarlingX authentication. This document uses the open-source IdP Keycloak as an example, but other OIDC providers work in similar manner (example: Azure active Directory or Google).

Modern OIDC IdPs typically enable additional security layers, such as MFA which are recommended for a secure configuration.

Prerequisites

Before configuring the OIDC connector/backend, ensure that you have the following:

• Administrative access to your StarlingX system

• Access to configure the external OIDC IdP

• The public certificate of the CA that signed the external OIDC IdP’s certificate

• Using the documentation provided by the external OIDC IdP (example: Keycloak) you are using, perform the following:

  1. Configure an OIDC client in your external IdP with the following client settings:

     • Client ID: Unique identifier of the client

     • Client Secret: Secure secret for client authentication

     • Redirect URI: Points to Dex callback in StarlingX: https://$ {STARLINGX_OAM_FLOATING_IP}:dex/callback

     Note

     These values will also be configured as Helm overrides to Dex later to establish the connection.

  2. Configure users and groups in the external OIDC IdP. These users and/or groups can be used in configured K8s RoleBindings to manage authorization/access control for the users and groups.

Configure the Dex OIDC Connector

Perform the following steps to configure the Dex OIDC connector:

Procedure

1. Import the external OIDC IdP CA certificate.

   Note

   It is assumed below that you have obtained this certificate and have it in the local file upstream-ca.crt.

   Create a Kubernetes secret containing the external IdP’s CA certificate:

   ~(keystone_admin)]$ kubectl -n kube-system create secret generic oidc-upstream-idp-ca --from-file upstream-ca.crt
   

2. Create a Helm overrides file for the oidc-auth-apps configuration.

   Populate the following variables:

   OAM_FLOATING_IP=128.224.49.105
   CLIENT_ID=api-client
   CLIENT_SECRET=97h72eC8v9P4hiKENR924lIG3vhZGL7j
   EXTERNAL_IDP_ISSUER_URL=https://idp.example.com/realms/stx-realm
   

   Where,

   OAM_FLOATING_IP

   The StarlingX system’s floating OAM IP address. Get this in the StarlingX system with system addrpool-list.

   CLIENT_ID

   The OIDC client identifier registered in the external IdP. Dex uses this value to identify itself during OIDC authorization and token exchange. Get this in the external IdP portal.

   CLIENT_SECRET

   The client secret associated with the OIDC client registered in the external IdP. Dex uses this secret to authenticate itself during the OAuth2 token exchange. Get this in the external IdP portal.

   EXTERNAL_IDP_ISSUER_URL

   The OIDC issuer URL of the external IdP. Dex uses this URL for OIDC discovery (/.well-known/openid-configuration) and to locate authorization, token, and JWKS endpoints. Get this in the external IdP portal.

   Then, create the override file:

   cat << EOF> external-idp-connector.yml
   config:
     connectors:
     - config:
         claimMapping:
           email: email
           name: name
           preferred_username: preferred_username
         clientID: ${CLIENT_ID}
         clientSecret: ${CLIENT_SECRET}
         getUserInfo: true
         issuer: ${EXTERNAL_IDP_ISSUER_URL}
         rootCA: /etc/ssl/certs/idpca/upstream-ca.crt
         redirectURI: https://${OAM_FLOATING_IP}:30556/dex/callback
         insecureEnableGroups: true
         scopes:
         - profile
         - email
         userIDKey: preferred_username
         userNameKey: preferred_username
       id: keycloak
       name: keycloak
       type: oidc
   staticClients:
   - id: stx-oidc-client-app
     name: STX OIDC Client app
     redirectURIs:
     - https://${OAM_FLOATING_IP}:30555/callback
     - http://localhost:8000
     - http://${OAM_FLOATING_IP}:8000
     secret: St8rlingX
   volumeMounts:
     - mountPath: /etc/dex/tls
       name: https-tls
     - mountPath: /etc/ssl/certs/idpca
       name: oidc-idp-ca
   volumes:
     - name: https-tls
       secret:
         defaultMode: 420
         secretName: oidc-auth-apps-certificate
     - name: oidc-idp-ca
       secret:
         secretName: oidc-upstream-idp-ca
   envVars:
    - name: SSL_CERT_FILE
      value: /etc/ssl/certs/idpca/upstream-ca.crt
   EOF
   

3. Apply the Helm overrides and update the application.

   ~(keystone_admin)]$ system helm-override-update oidc-auth-apps dex kube-system --values external-idp-connector.yml
   ~(keystone_admin)]$ system application-apply oidc-auth-apps
   

Postrequisites

Configure user/group authorization for K8s. For details, see Kubernetes API User Authorization.

OIDC Connector Configuration Parameters

The following table provides the full list of the config parameters under a Dex connector of type oidc.

Parameter	Description
id	Unique identifier for the connector within Dex
name	Human-readable label displayed on the Dex login screen
type	Connector type must be oidc for OIDC providers
claimMapping	Maps claim names from external IdP to Dex’s internal claims. For instance, the external IdP may use mail and may need to map to email in Dex.
clientID	OAuth2 client ID registered in the external OIDC provider
clientSecret	OAuth2 client secret for token exchange with the provider
getUserInfo	If true, Dex calls the IdP’s userinfo endpoint for additional claims
issuer	Issuer URL of the external OIDC IdP for discovery and JWKS
rootCA	Path to custom CA bundle for trusting the external IdP’s TLS certificate
redirectURI	Callback URL where the external IdP sends authorization responses
insecureEnable Groups	Enables group information extraction from non-standard structures
scopes	OAuth2/OIDC scopes requested during authentication
userIDKey	Claim used as the unique user identifier (stable and immutable)
userNameKey	Claim used as the human-readable username
volumes and volumeMounts	Mount certificate secret to pod file paths. Mount the oidc-auth-apps-certificate used to enable communication between oidc-auth-app services. Mount the oidc-upstream-idp-ca, created previously to the pod filepath.
envVars	Used to set SSL_CERT_FILE pod environment variable pointing to the external IdP CA inside the pod. This environment variable is required by the Golang http client.

Configure Multiple Backends

If both WAD and local LDAP servers are used at same time, use the connectors from WAD and local LDAP in the same connectors list using the volumes stated below:

volumeMounts:
- mountPath: /etc/ssl/certs/idpca
  name: oidc-idp-ca
- mountPath: /etc/ssl/certs/adcert
  name: certdir
- mountPath: /etc/dex/tls
  name: https-tls
volumes:
- name: certdir
  projected:
    sources:
    - secret:
      name: wad-ca-cert
    - secret:
      name: local-ldap-ca-cert
- name: https-tls
  secret:
    defaultMode: 420
    secretName: oidc-auth-apps-certificate
- name: oidc-idp-ca
  secret:
    secretName: oidc-upstream-idp-ca

If more than one WAD service is required for authenticating different users of StarlingX, multiple ldap type connectors can be configured, one for each WAD service.

If more than one userSearch plus groupSearch clauses are required for the same WAD service, multiple ldap type connectors with the same host information but different userSearch plus groupSearch clauses should be used.

Whenever you use multiple ldap type connectors, ensure that you use unique name: and id: parameters for each connector.

Note

Helm replaces arrays in overrides. If you are using multiple connectors, volumeMounts, or any array type of override, ensure that all the connectors and objects are combined into a single override file before applying.

Default Helm Overrides for OIDC Auth Apps application

For backward compatibility reasons, the default Helm overrides for Dex Helm are:

Note

It is recommended to create certificates using cert-manager and explicitly refer to the resulting certificate secrets in user-specified Helm overrides, as described in the procedure above.

image:
  repository: ghcr.io/dexidp/dex
  pullPolicy: IfNotPresent
  tag: v2.40.0
imagePullSecrets:
  - name: default-registry-key
env:
  name: KUBERNETES_POD_NAMESPACE
  value: kube-system
config:
  issuer: https://<OAM_IP>:30556/dex
  staticClients:
  - id: stx-oidc-client-app
    name: STX OIDC Client app
    secret: St8rlingX
    redirectURIs:
    - https://<OAM_IP>:30555/callback
  enablePasswordDB: false
  web:
    tlsCert: /etc/dex/tls/tls.crt
    tlsKey: /etc/dex/tls/tls.key
  storage:
    type: kubernetes
    config:
      inCluster: true
  oauth2:
    skipApprovalScreen: true
  logger:
    level: debug
service:
  type: NodePort
  ports:
    https:
      nodePort: 30556
https:
  enabled: true
grpc:
  enabled: false
nodeSelector:
  node-role.kubernetes.io/control-plane: ""
volumeMounts:
- mountPath: /etc/dex/tls/
  name: https-tls
volumes:
- name: https-tls
  secret:
    defaultMode: 420
    secretName: local-dex.tls
tolerations:
- key: "node-role.kubernetes.io/master"
  operator: "Exists"
  effect: "NoSchedule"
- key: "node-role.kubernetes.io/control-plane"
  operator: "Exists"
  effect: "NoSchedule"
podLabels:
  app: dex
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - dex
      topologyKey: kubernetes.io/hostname

The default Helm overrides for oidc-client are:

config:
  client_id: stx-oidc-client-app
  client_secret: St8rlingX
  issuer: https://<OAM_IP>:30556/dex
  issuer_root_ca: /home/dex-ca.pem
  issuer_root_ca_secret: dex-client-secret
  listen: https://0.0.0.0:5555
  redirect_uri: https://<OAM_IP>:30555/callback
  tlsCert: /etc/dex/tls/https/server/tls.crt
  tlsKey: /etc/dex/tls/https/server/tls.key
nodeSelector:
  node-role.kubernetes.io/control-plane: ""
service:
  type: NodePort
  port: 5555
  nodePort: 30555
replicas: <replicate count>
tolerations:
- key: "node-role.kubernetes.io/master"
  operator: "Exists"
  effect: "NoSchedule"
- key: "node-role.kubernetes.io/control-plane"
  operator: "Exists"
  effect: "NoSchedule"
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - stx-oidc-client
      topologyKey: kubernetes.io/hostname
helmv3Compatible: true

The default Helm overrides for secret-observer are:

namespace: "kube-system"
observedSecrets:
  - secretName: "dex-client-secret"
    filename: "dex-ca.pem"
    deploymentToRestart: "stx-oidc-client"
  - secretName: "local-dex.tls"
    filename: "tls.crt"
    deploymentToRestart: "stx-oidc-client"
  - secretName: "local-dex.tls"
    filename: "tls.crt"
    deploymentToRestart: "oidc-dex"
tolerations:
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Exists"
    effect: "NoSchedule"