Rook Ceph OSD Management¶
This section describes the steps to add, remove, and replace OSDs in Rook Container-based Ceph.
Add OSDs¶
Procedure
Obtain the UUIDs of the target disk(s).
$ system host-disk-list compute-0 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+--------------------+--------------------------------------------+ | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path | | | de | num | type | gib | gib | | | | +--------------------------------------+-----------+---------+---------+-------+------------+--------------+--------------------+--------------------------------------------+ | bc48b2ba-068d-4fc4-8c5a-f75cbd26eee4 | /dev/sda | 2048 | HDD | 292. | 0.0 | Undetermined | VB734fe32a- | /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0 | | | | | | 968 | | | 9839f8bf | | | | | | | | | | | | | 1aa1e19b-474a-46ef-9877-12afe2759046 | /dev/sdb | 2064 | HDD | 9.765 | 0.0 | Undetermined | VB29b38d6e- | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0 | | | | | | | | | 28b7e2ae | | | | | | | | | | | | | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/sdc | 2080 | HDD | 5.859 | 5.855 | Undetermined | VB816fb0b6-2ace762 | /dev/disk/by-path/pci-0000:00:0d.0-ata-3.0 | | | | | | | | | a | | | | | | | | | | | | | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/sdd | 2096 | HDD | 5.859 | 5.855 | Undetermined | VBc059f8a9-953642e | /dev/disk/by-path/pci-0000:00:0d.0-ata-4.0 | | | | | | | | | e | | | | | | | | | | | | | 68f4ec86-3d3c-41e0-8356-ef75275f9c88 | /dev/sde | 2112 | HDD | 5.859 | 5.855 | Undetermined | VB627293e5-cdeb561 | /dev/disk/by-path/pci-0000:00:0d.0-ata-5.0 | | | | | | | | | 4 | | | | | | | | | | | | | 65e2238c-a355-460e-ab08-8e5d6ccd97aa | /dev/sdf | 2128 | HDD | 5.859 | 5.855 | Undetermined | VB93e0fe15-45d4840 | /dev/disk/by-path/pci-0000:00:0d.0-ata-6.0 | | | | | | | | | d | | | | | | | | | | | | +--------------------------------------+-----------+---------+---------+-------+------------+--------------+--------------------+--------------------------------------------+
Add the target disk(s) as OSDs.
$ system host-stor-add compute-0 ed4c51a7-372f-4460-915d-43049cf23dd5 $ system host-stor-add compute-0 ab35233a-9952-4437-87ed-80cc4d6817d7
At this point, the OSDs will be in the configuring-with-app state.
$ system host-stor-list compute-0 +--------------------------------------+----------+-------+----------------------+--------------------------------------+-----------------------------+------------+----------+-----------+ | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_ | tier_name | | | | | | | | de | size_gib | | +--------------------------------------+----------+-------+----------------------+--------------------------------------+-----------------------------+------------+----------+-----------+ | 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397 | osd | None | configuring-with-app | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/disk/by-path/pci-0000: | /dev/sdd2 | 1 | storage | | | | | | | 00:0d.0-ata-4.0-part2 | | | | | | | | | | | | | | | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | None | configuring-with-app | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | | | | | | | 00:0d.0-ata-3.0-part2 | | | | | | | | | | | | | | +--------------------------------------+----------+-------+----------------------+--------------------------------------+-----------------------------+------------+----------+-----------+
Apply rook-ceph application and wait for the application to become applied.
$ system application-apply rook-ceph
The rook-ceph application may take several minutes to apply the changes and provision the OSDs before reaching the applied state.
Use the system application-show rook-ceph command to monitor progress.
List the OSDs and verify that they are now in the configured state.
$ system host-stor-list compute-0 +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_size_g | tier_name | | | | | | | | de | ib | | +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ | 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397 | osd | 0 | configured | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/disk/by-path/pci-0000: | /dev/sdd2 | 1 | storage | | | | | | | 00:0d.0-ata-4.0-part2 | | | | | | | | | | | | | | | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | 1 | configured | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | | | | | | | 00:0d.0-ata-3.0-part2 | | | | | | | | | | | | | | +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+
Remove OSDs¶
Important
It is recommended to delete the OSDs one at a time to avoid issues with data replication and data loss.
Procedure
Obtain the UUIDs of the OSDs to be removed.
$ system host-stor-list compute-0 +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_size_g | tier_name | | | | | | | | de | ib | | +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ | 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397 | osd | 0 | configured | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/disk/by-path/pci-0000: | /dev/sdd2 | 1 | storage | | | | | | | 00:0d.0-ata-4.0-part2 | | | | | | | | | | | | | | | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | 1 | configured | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | | | | | | | 00:0d.0-ata-3.0-part2 | | | | | | | | | | | | | | +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+
Mark the OSDs for removal.
$ system host-stor-delete 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397
At this point, the OSDs will be in the deleting-with-app state.
$ system host-stor-list compute-0 +--------------------------------------+----------+-------+-------------------+--------------------------------------+-----------------------------+------------+----------+-----------+ | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_ | tier_name | | | | | | | | de | size_gib | | +--------------------------------------+----------+-------+-------------------+--------------------------------------+-----------------------------+------------+----------+-----------+ | 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397 | osd | 0 | deleting-with-app | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/disk/by-path/pci-0000: | /dev/sdd2 | 1 | storage | | | | | | | 00:0d.0-ata-4.0-part2 | | | | | | | | | | | | | | | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | 1 | configured | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | | | | | | | 00:0d.0-ata-3.0-part2 | | | | | | | | | | | | | | +--------------------------------------+----------+-------+-------------------+--------------------------------------+-----------------------------+------------+----------+-----------+
Apply rook-ceph application and wait for the application to become applied.
$ system application-apply rook-ceph
The rook-ceph application may take several minutes to apply the changes and provision the OSDs before reaching the applied state.
Use the system application-show rook-ceph command to monitor progress.
Wait until the OSDs are removed.
The OSDs will be removed asynchronously after Rook Ceph reaches the applied state. Then, once the OSDs entry disappears from system host-stor-list, the OSDs are successfully removed.
$ system host-stor-list 3 +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_size_g | tier_name | | | | | | | | de | ib | | +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | 1 | configured | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | | | | | | | 00:0d.0-ata-3.0-part2 | | | | | | | | | | | | | | +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+
Replace OSDs¶
Procedure
Repeat the following steps for each target host, one at a time so that it does not impact Ceph availability.
Follow the steps in the Remove OSDs section to remove the target OSDs from the host.
Lock the host.
$ system host-lock <host>
After the host is locked, power off the host.
$ system host-power-off <host>
Replace the disk(s) while the host is powered off.
Power on the host and wait for the host to become online.
Unlock the host.
$ system host-unlock <host>
Wait for the host to become unlocked/available.
Follow the steps in Add OSDs to add the new replaced disk(s) as OSDs.
