Re: ceph-volume inventory reports available devices as unavailable

[Martin Buss <mbuss7004@xxxxxxxxx>]

Hi list admins, I accidentally posted my private address, can you please 
delete that post?

https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/JMFG73QMB3MJKHDMNPIKZHQOUUCJPJJN/

Thanks,

Martin

On 14.12.22 15:18, Eugen Block wrote:
> Hi,
>
> I haven't been dealing with ceph-volume too much lately, but I remember 
> seeing that when I have multiple DB devices on SSD and wanted to replace 
> only one failed drive. Although ceph-volume inventory reported the disk 
> as unavailable the actual create command was successful. But I don't 
> remember which versions were okay and which weren't, there were multiple 
> regressions in ceph-volume IIRC, it seems to be a very complex 
> structure. But apparently '... batch --report' is more reliable than 
> '... inventory'.
>
> Regards,
> Eugen
>
> Zitat von Frank Schilder <frans@xxxxxx>:
>
> > Hi all,
> >
> > we are using "ceph-volume inventory" for checking if a disk can host 
> > an OSD or not prior to running "ceph-volume lvm batch". Unfortunately, 
> > these two tools behave inconsistently. Our use case are SSDs with 
> > multiple OSDs per disk and re-deploying one of the OSDs on disk (the 
> > OSD was purged with "ceph-volume lvm zap --osd-id ID" and the 
> > left-over volume removed with "lvremove OSD-VG/OSD-LV").
> >
> > Ceph-volume inventory reports a disk as unavailable even though it has 
> > space for the new OSD. On the other hand, ceph-volume lvm batch 
> > happily creates the OSD. Expected is that inventory says there is 
> > space for an OSD and reports the disk as available. Is there any way 
> > to get this to behave in a consistent way? I don't want to run lvm 
> > batch for testing and then try to figure out how to interpret the 
> > conflicting information.
> >
> > Example outputs below (for octopus and pacific), each of these disks 
> > has 1 OSD deployed and space for another one. Thanks for any help!
> >
> > [root@ceph-adm:ceph-19 ~]# ceph-volume inventory --format json-pretty 
> > /dev/sdt
> > {
> >     "available": false,
> >     "device_id": "KINGSTON_SEDC500M3840G_50026B72825B6A67",
> >     "lsm_data": {},
> >     "lvs": [
> >         {
> >             "block_uuid": "iZGHyl-oY3R-K6va-t6Ji-VxFg-8K0V-Pl978X",
> >             "cluster_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9",
> >             "cluster_name": "ceph",
> >             "name": "osd-data-4ebd70a7-d51f-4f1c-921e-23269eb050fe",
> >             "osd_fsid": "a4f41f0e-0cf5-4aab-a4bb-390a64cfb01a",
> >             "osd_id": "571",
> >             "osdspec_affinity": "",
> >             "type": "block"
> >         }
> >     ],
> >     "path": "/dev/sdt",
> >     "rejected_reasons": [
> >         "LVM detected",
> >         "locked"
> >     ],
> >     "sys_api": {
> >         "human_readable_size": "3.49 TB",
> >         "locked": 1,
> >         "model": "KINGSTON SEDC500",
> >         "nr_requests": "256",
> >         "partitions": {},
> >         "path": "/dev/sdt",
> >         "removable": "0",
> >         "rev": "J2.8",
> >         "ro": "0",
> >         "rotational": "0",
> >         "sas_address": "0x500056b317b777ca",
> >         "sas_device_handle": "0x001e",
> >         "scheduler_mode": "mq-deadline",
> >         "sectors": 0,
> >         "sectorsize": "512",
> >         "size": 3840755982336.0,
> >         "support_discard": "512",
> >         "vendor": "ATA"
> >     }
> > }
> >
> > [root@ceph-adm:ceph-19 ~]# ceph-volume lvm batch --report --prepare 
> > --bluestore --no-systemd --crush-device-class rbd_data 
> > --osds-per-device 2 -- /dev/sdt
> > --> DEPRECATION NOTICE
> > --> You are using the legacy automatic disk sorting behavior
> > --> The Pacific release will change the default to --no-auto
> > --> passed data devices: 1 physical, 0 LVM
> > --> relative data size: 0.5
> >
> > Total OSDs: 1
> >
> >   Type            Path                                               
> >      LV Size         % of device
> > ----------------------------------------------------------------------------------------------------
> >   data            /dev/sdt                                           
> >      1.75 TB         50.00%
> >
> >
> >
> > # docker run --rm -v /dev:/dev --privileged --entrypoint 
> > /usr/sbin/ceph-volume "quay.io/ceph/ceph:v16.2.10" inventory --format 
> > json-pretty /dev/sdq
> > {
> >     "available": false,
> >     "device_id": "",
> >     "lsm_data": {},
> >     "lvs": [
> >         {
> >             "block_uuid": "ZtEuec-S672-meb5-xIQP-D20n-FjsC-jN3tVN",
> >             "cluster_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9",
> >             "cluster_name": "ceph",
> >             "name": "osd-data-37e894ed-167f-4fcc-a506-dca8bfc6c83f",
> >             "osd_fsid": "eaf62795-7c24-48e4-9f64-c66f42df973a",
> >             "osd_id": "582",
> >             "osdspec_affinity": "",
> >             "type": "block"
> >         }
> >     ],
> >     "path": "/dev/sdq",
> >     "rejected_reasons": [
> >         "locked",
> >         "LVM detected"
> >     ],
> >     "sys_api": {
> >         "human_readable_size": "3.49 TB",
> >         "locked": 1,
> >         "model": "KINGSTON SEDC500",
> >         "nr_requests": "256",
> >         "partitions": {},
> >         "path": "/dev/sdq",
> >         "removable": "0",
> >         "rev": "J2.8",
> >         "ro": "0",
> >         "rotational": "0",
> >         "sas_address": "0x500056b397fe9ac5",
> >         "sas_device_handle": "0x001b",
> >         "scheduler_mode": "mq-deadline",
> >         "sectors": 0,
> >         "sectorsize": "512",
> >         "size": 3840755982336.0,
> >         "support_discard": "512",
> >         "vendor": "ATA"
> >     }
> > }
> >
> > # docker run --rm -v /dev:/dev --privileged --entrypoint 
> > /usr/sbin/ceph-volume "quay.io/ceph/ceph:v16.2.10" lvm batch --report 
> > --prepare --bluestore --no-systemd --crush-device-class rbd_data 
> > --osds-per-device 2 -- /dev/sdq
> >
> > Total OSDs: 1
> >
> >   Type            Path                                               
> >      LV Size         % of device
> > ----------------------------------------------------------------------------------------------------
> >   data            /dev/sdq                                           
> >      1.75 TB         50.00%
> > --> DEPRECATION NOTICE
> > --> You are using the legacy automatic disk sorting behavior
> > --> The Pacific release will change the default to --no-auto
> > --> passed data devices: 1 physical, 0 LVM
> > --> relative data size: 0.5
> >
> > Best regards,
> > =================
> > Frank Schilder
> > AIT Risø Campus
> > Bygning 109, rum S14
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx