Rook Ceph OSD Management¶
This section describes the steps to add, remove, and replace OSDs in Rook Container-based Ceph.
Add OSDs¶
Procedure
Obtain the UUIDs of the target disk(s).
[sysadmin@controller-1 ~(keystone_admin)]$ system host-disk-list compute-0 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+--------------------+--------------------------------------------+ | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path | | | de | num | type | gib | gib | | | | +--------------------------------------+-----------+---------+---------+-------+------------+--------------+--------------------+--------------------------------------------+ | bc48b2ba-068d-4fc4-8c5a-f75cbd26eee4 | /dev/sda | 2048 | HDD | 292. | 0.0 | Undetermined | VB734fe32a- | /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0 | | | | | | 968 | | | 9839f8bf | | | | | | | | | | | | | 1aa1e19b-474a-46ef-9877-12afe2759046 | /dev/sdb | 2064 | HDD | 9.765 | 0.0 | Undetermined | VB29b38d6e- | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0 | | | | | | | | | 28b7e2ae | | | | | | | | | | | | | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/sdc | 2080 | HDD | 5.859 | 5.855 | Undetermined | VB816fb0b6-2ace762 | /dev/disk/by-path/pci-0000:00:0d.0-ata-3.0 | | | | | | | | | a | | | | | | | | | | | | | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/sdd | 2096 | HDD | 5.859 | 5.855 | Undetermined | VBc059f8a9-953642e | /dev/disk/by-path/pci-0000:00:0d.0-ata-4.0 | | | | | | | | | e | | | | | | | | | | | | | 68f4ec86-3d3c-41e0-8356-ef75275f9c88 | /dev/sde | 2112 | HDD | 5.859 | 5.855 | Undetermined | VB627293e5-cdeb561 | /dev/disk/by-path/pci-0000:00:0d.0-ata-5.0 | | | | | | | | | 4 | | | | | | | | | | | | | 65e2238c-a355-460e-ab08-8e5d6ccd97aa | /dev/sdf | 2128 | HDD | 5.859 | 5.855 | Undetermined | VB93e0fe15-45d4840 | /dev/disk/by-path/pci-0000:00:0d.0-ata-6.0 | | | | | | | | | d | | | | | | | | | | | | +--------------------------------------+-----------+---------+---------+-------+------------+--------------+--------------------+--------------------------------------------+
Add the target disk(s) as OSDs.
[sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-add compute-0 ed4c51a7-372f-4460-915d-43049cf23dd5 [sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-add compute-0 ab35233a-9952-4437-87ed-80cc4d6817d7
At this point, the OSDs will be in the configuring-with-app state.
[sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-list compute-0 +--------------------------------------+----------+-------+----------------------+--------------------------------------+-----------------------------+------------+----------+-----------+ | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_ | tier_name | | | | | | | | de | size_gib | | +--------------------------------------+----------+-------+----------------------+--------------------------------------+-----------------------------+------------+----------+-----------+ | 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397 | osd | None | configuring-with-app | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/disk/by-path/pci-0000: | /dev/sdd2 | 1 | storage | | | | | | | 00:0d.0-ata-4.0-part2 | | | | | | | | | | | | | | | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | None | configuring-with-app | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | | | | | | | 00:0d.0-ata-3.0-part2 | | | | | | | | | | | | | | +--------------------------------------+----------+-------+----------------------+--------------------------------------+-----------------------------+------------+----------+-----------+
Apply rook-ceph application and wait for the application to become applied.
[sysadmin@controller-1 ~(keystone_admin)]$ system application-apply rook-ceph
The rook-ceph application may take several minutes to apply the changes and provision the OSDs before reaching the applied state.
Use the system application-show rook-ceph command to monitor progress.
List the OSDs and verify that they are now in the configured state.
[sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-list compute-0 +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_size_g | tier_name | | | | | | | | de | ib | | +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ | 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397 | osd | 0 | configured | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/disk/by-path/pci-0000: | /dev/sdd2 | 1 | storage | | | | | | | 00:0d.0-ata-4.0-part2 | | | | | | | | | | | | | | | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | 1 | configured | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | | | | | | | 00:0d.0-ata-3.0-part2 | | | | | | | | | | | | | | +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+
Remove OSDs¶
Procedure
Obtain the UUIDs of the OSDs to be removed.
[sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-list compute-0 +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_size_g | tier_name | | | | | | | | de | ib | | +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ | 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397 | osd | 0 | configured | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/disk/by-path/pci-0000: | /dev/sdd2 | 1 | storage | | | | | | | 00:0d.0-ata-4.0-part2 | | | | | | | | | | | | | | | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | 1 | configured | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | | | | | | | 00:0d.0-ata-3.0-part2 | | | | | | | | | | | | | | +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+
Mark the OSDs for removal.
[sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-delete 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397
At this point, the OSDs will be in the deleting-with-app state.
[sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-list compute-0 +--------------------------------------+----------+-------+-------------------+--------------------------------------+-----------------------------+------------+----------+-----------+ | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_ | tier_name | | | | | | | | de | size_gib | | +--------------------------------------+----------+-------+-------------------+--------------------------------------+-----------------------------+------------+----------+-----------+ | 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397 | osd | 0 | deleting-with-app | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/disk/by-path/pci-0000: | /dev/sdd2 | 1 | storage | | | | | | | 00:0d.0-ata-4.0-part2 | | | | | | | | | | | | | | | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | 1 | configured | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | | | | | | | 00:0d.0-ata-3.0-part2 | | | | | | | | | | | | | | +--------------------------------------+----------+-------+-------------------+--------------------------------------+-----------------------------+------------+----------+-----------+
Apply rook-ceph application and wait for the application to become applied.
[sysadmin@controller-1 ~(keystone_admin)]$ system application-apply rook-ceph
The rook-ceph application may take several minutes to apply the changes and provision the OSDs before reaching the applied state.
Use the system application-show rook-ceph command to monitor progress.
Wait until the OSDs are removed.
The OSDs will be removed asynchronously after rook-ceph reaches the applied state. Then, once the OSDs entry disappears from system host-stor-list, the OSDs are successfully removed.
[sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-list 3 +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_size_g | tier_name | | | | | | | | de | ib | | +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | 1 | configured | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | | | | | | | 00:0d.0-ata-3.0-part2 | | | | | | | | | | | | | | +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+
Replace OSDs¶
Procedure
Repeat the following steps for each target host, one at a time so that it does not impact Ceph availability.
Follow the steps in the Remove OSDs section to remove the target OSDs from the host.
Note
If after several minutes the OSD entry is reverted to the configured state, then the OSD removal is considered unsafe and is aborted to avoid data loss.
You can proceed using system host-stor-delete --force <uuid>. This will skip data integrity checks and will forcefully remove the OSD. However, this may cause loss of data. Thus, use this only when you are sure that there is enough replicas of data. Otherwise, before using the –force option, always attempt the standard deletion first.
Lock the host.
[sysadmin@controller-1 ~(keystone_admin)]$ system host-lock <host>
After the host is locked, power off the host.
[sysadmin@controller-1 ~(keystone_admin)]$ system host-power-off <host>
Replace the disk(s) while the host is powered off.
Power on the host and wait for the host to become online.
Unlock the host.
[sysadmin@controller-1 ~(keystone_admin)]$ system host-unlock <host>
Wait for the host to become unlocked/available.
Follow the steps in Add OSDs to add the new replaced disk(s) as OSDs.
