you mean, "are you sure the new one got created that way"? yes, because it created a new lv on the ssd for it but I admire your sysadmin paranoia so I'll double check just for you, with ceph-volume lvm list :-) yup. separate block and db devs. block is on HDD db is on ssd ----- Original Message ----- From: "Tony Liu" <tonyliu0592@xxxxxxxxxxx> To: "Philip Brown" <pbrown@xxxxxxxxxx>, "Eugen Block" <eblock@xxxxxx> Cc: "ceph-users" <ceph-users@xxxxxxx> Sent: Friday, March 19, 2021 4:09:55 PM Subject: Re: ceph octopus mysterious OSD crash Are you sure the OSD is with DB/WAL on SSD? Tony ________________________________________ From: Philip Brown <pbrown@xxxxxxxxxx> Sent: March 19, 2021 02:49 PM To: Eugen Block Cc: ceph-users Subject: Re: [BULK] Re: Re: ceph octopus mysterious OSD crash Wow. My expectations have been adjusted. Thank you for detailing your experience, so I had motivation to try again. Explicit steps I took: 1. went into "cephadm shell" and did a vgremove on the HDD 2. ceph-volume zap /dev/(hdd) 3. lvremove (the matching old lv). This meant that the VG on the SSD had 25% space available. At this point, "ceph-volume inventory" shows the HDD as "available=True", but the shared SSD as false. 4. on my actual admin node, "ceph orch apply osd -i osd.deployspec.yml" and after a few minutes... it DID actually pick up the disk and make the OSD. (I had prevously "ceph osd rm"'d the id. so it used the prior ID) SO... there's still the concern about why the thing mysteriosly crashed in the first place :-/ (on TWO osd's!) But at least I know how to rebuild a single disk. ----- Original Message ----- From: "Eugen Block" <eblock@xxxxxx> To: "Stefan Kooman" <stefan@xxxxxx> Cc: "ceph-users" <ceph-users@xxxxxxx>, "Philip Brown" <pbrown@xxxxxxxxxx> Sent: Friday, March 19, 2021 2:19:55 PM Subject: [BULK] Re: Re: ceph octopus mysterious OSD crash I am quite sure that this case is covered by cephadm already. A few months ago I tested it after a major rework of ceph-volume. I don’t have any links right now. But I had a lab environment with multiple OSDs per node with rocksDB on SSD and after wiping both HDD and DB LV cephadm automatically redeployed the OSD according to my drive group file. Zitat von Stefan Kooman <stefan@xxxxxx>: > On 3/19/21 7:47 PM, Philip Brown wrote: > > I see. > >> >> I dont think it works when 7/8 devices are already configured, and >> the SSD is already mostly sliced. > > OK. If it is a test cluster you might just blow it all away. By > doing this you are simulating a "SSD" failure taking down all HDDs > with it. It sure isn't pretty. I would say the situation you ended > up with is not a corner case by any means. I am afraid I would > really need to set up a test cluster with cephadm to help you > further at this point, besides the suggestion above. > > Gr. Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |