Install Additional RBD Provisioners¶

You can launch additional dedicated RBD provisioners to support specific applications using dedicated pools, storage classes, and namespaces.

About this task

This can be useful if, for example, to allow an application to have control over its own persistent volume provisioner, that is, managing the Ceph pool, storage tier, allowed namespaces, and so on, without requiring the kubernetes admin to modify the default RBD provisioner service in the kube-system namespace.

This procedure uses standard Helm mechanisms to install a second RBD provisioner.

Procedure

  1. Capture a list of monitors and cluster ID.

    This will be stored in the environment variable <MON_LIST> and used in the following step.

    ~(keystone_admin)]$ MON_LIST=$(ceph mon dump 2>&1 | awk /^[0-2]:/'{print $2}' | grep -oP '(?<=v1:).*(?=/)' | awk -F' ' '{print "  - "$1}')
    

    This will be stored in the environment variable <CLUSTER_ID> and used in the following step.

    ~(keystone_admin)]$ CLUSTER_ID=$(ceph fsid)
    
  2. Create an overrides yaml file defining the new provisioner.

    In this example we will create the file /home/sysadmin/my-second-provisioner-overrides.yaml.

    ~(keystone_admin)]$ cat <<EOF > ~/my-second-provisioner-overrides.yaml
    classdefaults:
      monitors:
    ${MON_LIST}
    classes:
    - additionalNamespaces:
      - default
      - kube-public
      chunk_size: 64
      clusterID: ${CLUSTER_ID}
      crush_rule_name: storage_tier_ruleset
      name: 2nd-provisioner
      pool_name: another-pool
      replication: 1
      userId: 2nd-user-secret
      userSecretName: 2nd-user-secret
    csiConfig:
    - clusterID: ${CLUSTER_ID}
      monitors:
    ${MON_LIST}
    nodeplugin:
      fullnameOverride: 2nd-nodeplugin
    provisioner:
      fullnameOverride: 2nd-provisioner
      replicaCount: 1
    driverName: cool-rbd-provisioner.csi.ceph.com
    EOF
    

    Note

    The replicaCount parameter has the value 1 for SX and 2 for others.

  3. Gets the directory where the rbd-provisioner static overrides file is located.

    This will be stored in the environment variable <RBD_STATIC_OVERRIDES> and used in the following step.

    ~(keystone_admin)]$ SW_VERSION=$(system show | awk /software_version/'{print $4}')
    ~(keystone_admin)]$ INTEG_APPS_VERSION=$(system application-show platform-integ-apps | awk /app_version/'{print $4}')
    ~(keystone_admin)]$ RBD_STATIC_OVERRIDES=/opt/platform/fluxcd/$SW_VERSION/platform-integ-apps/$INTEG_APPS_VERSION/platform-integ-apps-fluxcd-manifests/rbd-provisioner/rbd-provisioner-static-overrides.yaml
    
  4. Install the chart.

    ~(keystone_admin)]$ helm upgrade --install my-2nd-provisioner stx-platform/ceph-csi-rbd --namespace=isolated-app --create-namespace --values=$RBD_STATIC_OVERRIDES --values=/home/sysadmin/my-second-provisioner-overrides.yaml
    Release "my-2nd-provisioner" does not exist. Installing it now.
    NAME: my-2nd-provisioner
    LAST DEPLOYED: Wed Dec 14 04:20:00 2022
    NAMESPACE: isolated-app
    STATUS: deployed
    ...
    

    Note

    Helm automatically created the namespace isolated-app while installing the chart.

  5. Confirm that my-2nd-provisioner has been deployed.

    ~(keystone_admin)]$ helm list -n isolated-app
    NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
    my-2nd-provisioner      isolated-app    1               2022-12-14 04:20:00.345212618 +0000 UTC deployed        ceph-csi-rbd-3.6.2      3.6.2
    
  6. Confirm that the 2nd-storage storage class was created.

    ~(keystone_admin)]$ kubectl get sc -A
    NAME                PROVISIONER                         RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
    2nd-provisioner     cool-rbd-provisioner.csi.ceph.com   Delete          Immediate           true                   6m4s
    cephfs              cephfs.csi.ceph.com                 Delete          Immediate           true                   10m
    general (default)   rbd.csi.ceph.com                    Delete          Immediate           true                   10m
    

    You can now create and mount PVCs from the new RBD provisioner’s 2nd-storage storage class, from within the isolated-app namespace.