Re: ceph-volume inventory reports available devices as unavailable

Ralph Soika <ralph.soika@xxxxxxxxx> · Wed, 14 Dec 2022 16:41:32 +0100

Hi,

in run into the same problem. After installing ceph quincy on different 
servers, some were able to detect the disks others not.

My servers are hosted at hetzner.de and as I did not found a solution, 
so I tried as long different servers until I found servers where ceph 
detected the disks correctly. The hetzner support did not help and 
explained it is a software issue. Of course - but it seems to be related 
to a special hardware configuration.

I narrowed down the problem and installed a blank Debian with with 
ceph/ceph-adm. And I only tested the "$ sudo ceph-adm ceph-volume inventoy".

See also my issue here:

https://tracker.ceph.com/issues/58189

https://serverfault.com/questions/1117213/why-is-ceph-is-not-detecting-ssd-device-on-a-new-node

Best regards

Ralph

Am 14.12.22 um 14:22 schrieb Frank Schilder:
Hi all,

we are using "ceph-volume inventory" for checking if a disk can host an OSD or not prior to running "ceph-volume lvm batch". Unfortunately, these two tools behave inconsistently. Our use case are SSDs with multiple OSDs per disk and re-deploying one of the OSDs on disk (the OSD was purged with "ceph-volume lvm zap --osd-id ID" and the left-over volume removed with "lvremove OSD-VG/OSD-LV").

Ceph-volume inventory reports a disk as unavailable even though it has space for the new OSD. On the other hand, ceph-volume lvm batch happily creates the OSD. Expected is that inventory says there is space for an OSD and reports the disk as available. Is there any way to get this to behave in a consistent way? I don't want to run lvm batch for testing and then try to figure out how to interpret the conflicting information.

Example outputs below (for octopus and pacific), each of these disks has 1 OSD deployed and space for another one. Thanks for any help!

[root@ceph-adm:ceph-19 ~]# ceph-volume inventory --format json-pretty /dev/sdt
{
     "available": false,
     "device_id": "KINGSTON_SEDC500M3840G_50026B72825B6A67",
     "lsm_data": {},
     "lvs": [
         {
             "block_uuid": "iZGHyl-oY3R-K6va-t6Ji-VxFg-8K0V-Pl978X",
             "cluster_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9",
             "cluster_name": "ceph",
             "name": "osd-data-4ebd70a7-d51f-4f1c-921e-23269eb050fe",
             "osd_fsid": "a4f41f0e-0cf5-4aab-a4bb-390a64cfb01a",
             "osd_id": "571",
             "osdspec_affinity": "",
             "type": "block"
         }
     ],
     "path": "/dev/sdt",
     "rejected_reasons": [
         "LVM detected",
         "locked"
     ],
     "sys_api": {
         "human_readable_size": "3.49 TB",
         "locked": 1,
         "model": "KINGSTON SEDC500",
         "nr_requests": "256",
         "partitions": {},
         "path": "/dev/sdt",
         "removable": "0",
         "rev": "J2.8",
         "ro": "0",
         "rotational": "0",
         "sas_address": "0x500056b317b777ca",
         "sas_device_handle": "0x001e",
         "scheduler_mode": "mq-deadline",
         "sectors": 0,
         "sectorsize": "512",
         "size": 3840755982336.0,
         "support_discard": "512",
         "vendor": "ATA"
     }
}

[root@ceph-adm:ceph-19 ~]# ceph-volume lvm batch --report --prepare --bluestore --no-systemd --crush-device-class rbd_data --osds-per-device 2 -- /dev/sdt
--> DEPRECATION NOTICE
--> You are using the legacy automatic disk sorting behavior
--> The Pacific release will change the default to --no-auto
--> passed data devices: 1 physical, 0 LVM
--> relative data size: 0.5

Total OSDs: 1

   Type            Path                                                    LV Size         % of device
----------------------------------------------------------------------------------------------------
   data            /dev/sdt                                                1.75 TB         50.00%

# docker run --rm -v /dev:/dev --privileged --entrypoint /usr/sbin/ceph-volume "quay.io/ceph/ceph:v16.2.10" inventory --format json-pretty /dev/sdq
{
     "available": false,
     "device_id": "",
     "lsm_data": {},
     "lvs": [
         {
             "block_uuid": "ZtEuec-S672-meb5-xIQP-D20n-FjsC-jN3tVN",
             "cluster_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9",
             "cluster_name": "ceph",
             "name": "osd-data-37e894ed-167f-4fcc-a506-dca8bfc6c83f",
             "osd_fsid": "eaf62795-7c24-48e4-9f64-c66f42df973a",
             "osd_id": "582",
             "osdspec_affinity": "",
             "type": "block"
         }
     ],
     "path": "/dev/sdq",
     "rejected_reasons": [
         "locked",
         "LVM detected"
     ],
     "sys_api": {
         "human_readable_size": "3.49 TB",
         "locked": 1,
         "model": "KINGSTON SEDC500",
         "nr_requests": "256",
         "partitions": {},
         "path": "/dev/sdq",
         "removable": "0",
         "rev": "J2.8",
         "ro": "0",
         "rotational": "0",
         "sas_address": "0x500056b397fe9ac5",
         "sas_device_handle": "0x001b",
         "scheduler_mode": "mq-deadline",
         "sectors": 0,
         "sectorsize": "512",
         "size": 3840755982336.0,
         "support_discard": "512",
         "vendor": "ATA"
     }
}

# docker run --rm -v /dev:/dev --privileged --entrypoint /usr/sbin/ceph-volume "quay.io/ceph/ceph:v16.2.10" lvm batch --report --prepare --bluestore --no-systemd --crush-device-class rbd_data --osds-per-device 2 -- /dev/sdq

Total OSDs: 1

   Type            Path                                                    LV Size         % of device
----------------------------------------------------------------------------------------------------
   data            /dev/sdq                                                1.75 TB         50.00%
--> DEPRECATION NOTICE
--> You are using the legacy automatic disk sorting behavior
--> The Pacific release will change the default to --no-auto
--> passed data devices: 1 physical, 0 LVM
--> relative data size: 0.5

Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx