Here's an example of an OSD with separate block.db:
---snip---
osd-db-0cd3d532-a03a-4a17-bc0c-ae3e8c710ace
ceph.block_device=/dev/ceph-63cf931a-f7a8-4ab8-bdd2-4c01c8e10f4d/osd-block-a30f85ab-b2b9-4f75-a1f9-48c2ef6db1c1,ceph.block_uuid=8wWfCT-4aQw-2VKR-I1Lf-a0e3-rRRA-v3pLqL,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=7bdffde0-623f-11eb-b3db-fa163e672db2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/ceph-ce9e4b47-1007-453c-b11e-46212474c130/osd-db-0cd3d532-a03a-4a17-bc0c-ae3e8c710ace,ceph.db_uuid=UTO5eF-WmfP-vn32-w7Cg-1dS0-RVIT-5egFoM,ceph.encrypted=0,ceph.osd_fsid=a30f85ab-b2b9-4f75-a1f9-48c2ef6db1c1,ceph.osd_id=6,ceph.osdspec_affinity=default,ceph.type=db,ceph.vdo=0
osd-block-a30f85ab-b2b9-4f75-a1f9-48c2ef6db1c1
ceph.block_device=/dev/ceph-63cf931a-f7a8-4ab8-bdd2-4c01c8e10f4d/osd-block-a30f85ab-b2b9-4f75-a1f9-48c2ef6db1c1,ceph.block_uuid=8wWfCT-4aQw-2VKR-I1Lf-a0e3-rRRA-v3pLqL,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=7bdffde0-623f-11eb-b3db-fa163e672db2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.db_device=/dev/ceph-ce9e4b47-1007-453c-b11e-46212474c130/osd-db-0cd3d532-a03a-4a17-bc0c-ae3e8c710ace,ceph.db_uuid=UTO5eF-WmfP-vn32-w7Cg-1dS0-RVIT-5egFoM,ceph.encrypted=0,ceph.osd_fsid=a30f85ab-b2b9-4f75-a1f9-48c2ef6db1c1,ceph.osd_id=6,ceph.osdspec_affinity=default,ceph.type=block,ceph.vdo=0
---snip---
As you can see the information is identical, so in order to wipe an
OSD entirely you also need to wipe the db device, which ceph-volume
can do for you:
ceph-volume lvm zap --destroy
/dev/ceph-ce9e4b47-1007-453c-b11e-46212474c130/osd-db-0cd3d532-a03a-4a17-bc0c-ae3e8c710ace
Zitat von Philip Brown <pbrown@xxxxxxxxxx>:
Where does it read it from?
does it keep it in the block.db lv, or the block dev lv, or both?
I removed the vg from the block dev and did wipefs, if I recall.
----- Original Message -----
From: "Eugen Block" <eblock@xxxxxx>
To: "Philip Brown" <pbrown@xxxxxxxxxx>
Cc: "ceph-users" <ceph-users@xxxxxxx>
Sent: Tuesday, April 6, 2021 9:06:50 AM
Subject: [BULK] Re: Re: which is definitive:
/var/lib/ceph symlinks or ceph-volume?
Did you recreate the OSD with the same UUID? I'm guessing ceph-volume
just reads LV tags and thinks there's a block.db for that OSD. Before
creating the OSD you should also wipe the LV containing the block.db
for the failed OSD.
Zitat von Philip Brown <pbrown@xxxxxxxxxx>:
It was up and running.
I should mention that the situation came about when I was testing
OSD rebuild.
So I had a previously autogenerated hybrid OSD, with SSD db, and HDD
block dev.
I then stopped the OSD, wiped HDD, .... did "ceph osd rm" and ceph auth rm.
and then rebuilt with ceph-volume.. with the wrong flags.
I belive I used:
ceph-volume lvm create --block /dev/xxx --journal /dev/blahblah
After that, ceph-volume lvm list
claimed it was using the ssd again as db device. but ls -l on
/var/lib/ceph/osd/XXXX only showed symlink for block dev.
This was on nautilus.
----- Original Message -----
From: "Eugen Block" <eblock@xxxxxx>
To: "ceph-users" <ceph-users@xxxxxxx>
Sent: Tuesday, April 6, 2021 5:20:40 AM
Subject: Re: which is definitive: /var/lib/ceph
symlinks or ceph-volume?
Is the OSD up and running?
Do you see IO on the dedicated block.db LV when running a 'ceph tell
osd.7 bench'? It sounds very strange, I haven't seen that.
Zitat von Philip Brown <pbrown@xxxxxxxxxx>:
I am in a situation where I see conflicting information.
On the one hand,
ls -l /var/lib/ceph/osd/ceph-7
shows a symlink for block device, but no block.db
On the other hand,
ceph-volume lvm list
claims that there is a separate db device registered for osd 7
how can I know which one is correct?
(This is currently ceph nautilus)
--
Philip Brown| Sr. Linux System Administrator | Medata, Inc.
5 Peters Canyon Rd Suite 250
Irvine CA 92606
Office 714.918.1310| Fax 714.918.1325
pbrown@xxxxxxxxxx| www.medata.com
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
