Deployment Models and Services for Rook Ceph

The deployment model is the topology strategy that defines the storage backend capabilities of the deployment. The deployment model dictates how the storage solution will look like defining rules for the placement of the storage cluster elements.

Available Deployment Models

Each deployment model works with different deployment strategies and rules to fit different needs. Choose one of the following models according to the demands of your cluster:

Controller Model (default)

  • The OSDs must be added only in hosts with controller personality.

  • The replication factor can be configured up to size 3.

  • Can swap into Open Model.

Dedicated Model

  • The OSDs must be added only in hosts with worker personality.

  • The replication factor can be configured up to size 3.

  • Can swap into Open Model.

Open Model

  • The OSD placement does not have any limitation.

  • The replication factor does not have any limitation.

  • Can swap into controller or dedicated if the placement requisites are satisfied.

Replication Factor

The replication factor is the number of copies that each piece of data has spread across the cluster to provide redundancy.

You can change the replication of an existing Rook Ceph storage backend with the following command:

~(keystone_admin)$ system storage-backend-modify ceph-rook-store replication=<size>

Possible Replication Factors on Deployment Models for platforms.

Simplex Controller Model:

Default: 1 Max: 3

Simplex Open Model:

Default: 1 Max: Any

Duplex Controller Model:

Default: 2 Max: 3

Duplex Open Model:

Default: 1 Max: Any

Duplex+ or Standard Controller Model:

Default: 2 Max: 3

Duplex+ or Standard Dedicated Model:

Default: 2 Max: 3

Duplex+ or Standard Open Model:

Default: 2 Max: Any

Minimum Replication Factor

The minimum replication factor is the least number of copies that each piece of data have spread across the cluster to provide redundancy.

You can assign any number smaller than the replication factor to this parameter. The default value is replication - 1.

You can change the minimum replication of an existing Rook Ceph storage backend with the command:

~(keystone_admin)$ system storage-backend-modify ceph-rook-store min_replication=<size>

Monitor Count

Monitors (mons) are allocated on all the hosts that have a host-fs ceph with the monitor capability on it.

When the host has no OSD registered on the platform, you should add host-fs ceph in every node intended to house a monitor with the command:

~(keystone_admin)$ system host-fs-add <hostname> ceph=<size>

When there are OSDs registered on a host you should add the monitor function to the existing host-fs.

~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=osd,monitor

Possible Monitor Count on Deployment Models for Platforms

Simplex:

Min: 1 Max: 1

Duplex:

Min: 1 Recommended: 3 (using floating monitor) Max: 3 (using floating monitor)

Duplex+ or Standard:

Min: 1 Recommended: 3 Max: Any

Floating Monitor (only in Duplex)

A Floating monitor is possible and recommended on Duplex platforms. The monitor roams and is always allocated on the active controller.

To add the floating monitor:

Note

You should lock the inactive controller add controllerfs ceph-float to the platform.

~(keystone_admin)$ system host-lock controller-1 (with controller-0 as the active controller)
~(keystone_admin)$ system controllerfs-add ceph-float=<size>

Host-fs and controller-fs

To properly set the environment for Rook Ceph, some filesystems are needed.

Note

All changes in host-fs and controller-fs need a reapply on the application to properly propagate the modifications in the Rook ceph cluster.

Functions

The functions parameter contains the ceph cluster function of a given host. A host-fs can have monitor and osd functions, a controller-fs can only have the monitor function.

To modify the function of a host-fs the complete list of functions desired must be informed.

Examples:

#(only monitor)
~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=monitor

#(only osd)
~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=osd

#(no function)
~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=

#(only monitor)
~(keystone_admin)$ system controllerfs-modify ceph-float --functions=monitor

#(no function)
~(keystone_admin)$ system controllerfs-modify ceph-float --functions=

Services

Services are the storage types (or classes) that provides storage to each pod with some mount or storage space allocation.

Available Services

There are four possible services compatible with Rook Ceph. You can combine them following the rules:

block (default)

  • Not possible to be deployed together with ecblock.

  • Will enable the block service in rook, will use cephRBD.

ecblock

  • Not possible to be deployed together with block.

  • Will enable the ecblock service in rook, will use cephRBD.

filesystem (default)

  • Will enable the ceph filesystem and use cephFS.

object

  • Will enable the ceph object store (RGW).

Services Parameterization for the Open Model

In the ‘open’ deployment model, no specific configurations are enforced. Users are responsible for customizing settings based on their specific needs. To update configurations, a Helm override is required.

When applying a helm-override update, list-type values are completely replaced, not incrementally updated. For example, modifying cephFileSystems (or cephBlockPools, cephECBlockPools, cephObjectStores) via Helm override will overwrite the entire entry.

Here is an example of how to change a parameter, using failureDomain as an example, for Cephfs and RBD:

# Get the current crush rule information
ceph osd pool get kube-cephfs-data crush_rule

# Get the current default values
helm get values -n rook-ceph rook-ceph-cluster -o yaml | sed -n  '/^cephFileSystems:/,/^[[:alnum:]_-]*:/p;' | sed '$d' > cephfs_overrides.yaml

# Update the failure domain
sed -i 's/failureDomain: osd/failureDomain: host/g' cephfs_overrides.yaml

# Get the current user override values ("combined overrides" is what will be deployed):
system helm-override-show rook-ceph rook-ceph-cluster rook-ceph

# Set the new overrides
system helm-override-update rook-ceph rook-ceph-cluster rook-ceph --reuse-values --values cephfs_overrides.yaml

# Get the updated user override values
system helm-override-show rook-ceph rook-ceph-cluster rook-ceph

# Apply the application
system application-apply rook-ceph

# Confirm the current crush rule information:
ceph osd pool get kube-cephfs-data crush_rule