Hi Eugen,
thanks for that. I guess the sane insane logic could be that if "rejected_reasons": ["LVM detected", "locked"], the disk has at least 1 OSD (or something ceph-ish) already and lvm batch would do something non-trivial (report-json not empty), one should consider the disk as "available".
Shame that the deployment tools are so inconsistent. It would be much easier to repair things if there was an easy way to query what is possible, how much space on a drive could be used and for what, etc.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Eugen Block <eblock@xxxxxx>
Sent: 14 December 2022 15:18:11
To: ceph-users@xxxxxxx
Subject: Re: ceph-volume inventory reports available devices as unavailable
Hi,
I haven't been dealing with ceph-volume too much lately, but I
remember seeing that when I have multiple DB devices on SSD and wanted
to replace only one failed drive. Although ceph-volume inventory
reported the disk as unavailable the actual create command was
successful. But I don't remember which versions were okay and which
weren't, there were multiple regressions in ceph-volume IIRC, it seems
to be a very complex structure. But apparently '... batch --report' is
more reliable than '... inventory'.
Regards,
Eugen
Zitat von Frank Schilder <frans@xxxxxx>:
> Hi all,
>
> we are using "ceph-volume inventory" for checking if a disk can host
> an OSD or not prior to running "ceph-volume lvm batch".
> Unfortunately, these two tools behave inconsistently. Our use case
> are SSDs with multiple OSDs per disk and re-deploying one of the
> OSDs on disk (the OSD was purged with "ceph-volume lvm zap --osd-id
> ID" and the left-over volume removed with "lvremove OSD-VG/OSD-LV").
>
> Ceph-volume inventory reports a disk as unavailable even though it
> has space for the new OSD. On the other hand, ceph-volume lvm batch
> happily creates the OSD. Expected is that inventory says there is
> space for an OSD and reports the disk as available. Is there any way
> to get this to behave in a consistent way? I don't want to run lvm
> batch for testing and then try to figure out how to interpret the
> conflicting information.
>
> Example outputs below (for octopus and pacific), each of these disks
> has 1 OSD deployed and space for another one. Thanks for any help!
>
> [root@ceph-adm:ceph-19 ~]# ceph-volume inventory --format
> json-pretty /dev/sdt
> {
> "available": false,
> "device_id": "KINGSTON_SEDC500M3840G_50026B72825B6A67",
> "lsm_data": {},
> "lvs": [
> {
> "block_uuid": "iZGHyl-oY3R-K6va-t6Ji-VxFg-8K0V-Pl978X",
> "cluster_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9",
> "cluster_name": "ceph",
> "name": "osd-data-4ebd70a7-d51f-4f1c-921e-23269eb050fe",
> "osd_fsid": "a4f41f0e-0cf5-4aab-a4bb-390a64cfb01a",
> "osd_id": "571",
> "osdspec_affinity": "",
> "type": "block"
> }
> ],
> "path": "/dev/sdt",
> "rejected_reasons": [
> "LVM detected",
> "locked"
> ],
> "sys_api": {
> "human_readable_size": "3.49 TB",
> "locked": 1,
> "model": "KINGSTON SEDC500",
> "nr_requests": "256",
> "partitions": {},
> "path": "/dev/sdt",
> "removable": "0",
> "rev": "J2.8",
> "ro": "0",
> "rotational": "0",
> "sas_address": "0x500056b317b777ca",
> "sas_device_handle": "0x001e",
> "scheduler_mode": "mq-deadline",
> "sectors": 0,
> "sectorsize": "512",
> "size": 3840755982336.0,
> "support_discard": "512",
> "vendor": "ATA"
> }
> }
>
> [root@ceph-adm:ceph-19 ~]# ceph-volume lvm batch --report --prepare
> --bluestore --no-systemd --crush-device-class rbd_data
> --osds-per-device 2 -- /dev/sdt
> --> DEPRECATION NOTICE
> --> You are using the legacy automatic disk sorting behavior
> --> The Pacific release will change the default to --no-auto
> --> passed data devices: 1 physical, 0 LVM
> --> relative data size: 0.5
>
> Total OSDs: 1
>
> Type Path
> LV Size % of device
> ----------------------------------------------------------------------------------------------------
> data /dev/sdt
> 1.75 TB 50.00%
>
>
>
> # docker run --rm -v /dev:/dev --privileged --entrypoint
> /usr/sbin/ceph-volume "quay.io/ceph/ceph:v16.2.10" inventory
> --format json-pretty /dev/sdq
> {
> "available": false,
> "device_id": "",
> "lsm_data": {},
> "lvs": [
> {
> "block_uuid": "ZtEuec-S672-meb5-xIQP-D20n-FjsC-jN3tVN",
> "cluster_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9",
> "cluster_name": "ceph",
> "name": "osd-data-37e894ed-167f-4fcc-a506-dca8bfc6c83f",
> "osd_fsid": "eaf62795-7c24-48e4-9f64-c66f42df973a",
> "osd_id": "582",
> "osdspec_affinity": "",
> "type": "block"
> }
> ],
> "path": "/dev/sdq",
> "rejected_reasons": [
> "locked",
> "LVM detected"
> ],
> "sys_api": {
> "human_readable_size": "3.49 TB",
> "locked": 1,
> "model": "KINGSTON SEDC500",
> "nr_requests": "256",
> "partitions": {},
> "path": "/dev/sdq",
> "removable": "0",
> "rev": "J2.8",
> "ro": "0",
> "rotational": "0",
> "sas_address": "0x500056b397fe9ac5",
> "sas_device_handle": "0x001b",
> "scheduler_mode": "mq-deadline",
> "sectors": 0,
> "sectorsize": "512",
> "size": 3840755982336.0,
> "support_discard": "512",
> "vendor": "ATA"
> }
> }
>
> # docker run --rm -v /dev:/dev --privileged --entrypoint
> /usr/sbin/ceph-volume "quay.io/ceph/ceph:v16.2.10" lvm batch
> --report --prepare --bluestore --no-systemd --crush-device-class
> rbd_data --osds-per-device 2 -- /dev/sdq
>
> Total OSDs: 1
>
> Type Path
> LV Size % of device
> ----------------------------------------------------------------------------------------------------
> data /dev/sdq
> 1.75 TB 50.00%
> --> DEPRECATION NOTICE
> --> You are using the legacy automatic disk sorting behavior
> --> The Pacific release will change the default to --no-auto
> --> passed data devices: 1 physical, 0 LVM
> --> relative data size: 0.5
>
> Best regards,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
