You may need to use `ceph-volume lvm migrate’ [1] instead of ceph-bluestore-tool. If I recall correctly, this is a pretty new feature, I’m not sure whether it is available to your version. If you use ceph-bluestore-tool, then you need to modify the LVM tags manually. Please refer to the previous threads, e.g. [2] and some more. [1]: https://docs.ceph.com/en/latest/man/8/ceph-volume/#migrate [2]: https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/VX23NQ66P3PPEX36T3PYYMHPLBSFLMYA/#JLNDFGXR4ZLY27DHD3RJTTZEDHRZJO4Q 发件人: Szabo, Istvan (Agoda)<mailto:Istvan.Szabo@xxxxxxxxx> 发送时间: 2021年9月28日 18:20 收件人: Eugen Block<mailto:eblock@xxxxxx>; ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> 主题: Re: is it possible to remove the db+wal from an external device (nvme) Gave a try of it, so all the 3 osds finally failed :/ Not sure what went wrong. Do the normal maintenance things, ceph osd set noout, ceph osd set norebalance, stop the osd and run this command: ceph-bluestore-tool bluefs-bdev-migrate --dev-target /var/lib/ceph/osd/ceph-0/block --devs-source /var/lib/ceph/osd/ceph-8/block.db --path /var/lib/ceph/osd/ceph-8/ Output: device removed:1 /var/lib/ceph/osd/ceph-8/block.db device added: 1 /dev/dm-2 When tried to start I got this in the log: osd.8 0 OSD:init: unable to mount object store ** ERROR: osd init failed: (13) Permission denied set uid:gid to 167:167 (ceph:ceph) ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable), process ceph-osd, pid 1512261 pidfile_write: ignore empty --pid-file >From the another 2 osds the block.db removed and I can start it back. I've zapped the db drive just to be removed from the device completely and after machine restart none of these 2 osds came back, I guess missing the db device. Is there any steps missing? 1.Noout+norebalance 2. Stop osd 3. migrate with the above command the block.db to the block. 4. do on the other osds which is sharing the same db device that want to remove. 5. zap the db device 6. start back the osds. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@xxxxxxxxx --------------------------------------------------- -----Original Message----- From: Eugen Block <eblock@xxxxxx> Sent: Monday, September 27, 2021 7:42 PM To: ceph-users@xxxxxxx Subject: Re: is it possible to remove the db+wal from an external device (nvme) Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ Hi, I think 'ceph-bluestore-tool bluefs-bdev-migrate' could be of use here. I haven't tried it in a production environment yet, only in virtual labs. Regards, Eugen Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@xxxxxxxxx>: > Hi, > > Seems like in our config the nvme device as a wal+db in front of the > ssd slowing down the ssds osds. > I'd like to avoid to rebuild all the osd-, is there a way somehow > migrate to the "slower device" the wal+db without reinstall? > > Ty > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an > email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |