Re: 回复: Re: is it possible to remove the db+wal from an external device (nvme)

[Igor Fedotov <ifedotov@xxxxxxx>]

Hi Eugen,

indeed this looks like an issue related to containerized deployment, 
"ceph-volume lvm migrate" expects osd folder to be under /var/lib/ceph/osd:

> stderr: 2021-09-29T06:56:24.787+0000 7fde05b96180 -1 
bluestore(/var/lib/ceph/osd/ceph-1) _lock_fsid failed to lock 
/var/lib/ceph/osd/ceph-1/fsid (is another ceph-osd still running?)(11) 
Resource temporarily unavailable

As a workaround you might want to try to create a symlink to your actual 
location before issuing the migrate command: /var/lib/ceph/osd -> 
/var/lib/ceph/152fd738-01bc-11ec-a7fd-fa163e672db2/

More complicated (and more general IMO) way would be to run the migrate 
command from within a container deployed similarly (i.e. with all the 
proper subfolder mappings) to ceph-osd one. Just speculating - not a big 
expert in containers and never tried that with properly deployed 
production cluster...


Thanks,

Igor

On 9/29/2021 10:07 AM, Eugen Block wrote:
> Hi,
>
> I just tried with 'ceph-volume lvm migrate' in Octopus but it doesn't 
> really work. I'm not sure if I'm missing something here, but I believe 
> it's again the already discussed containers issue. To be able to run 
> the command for an OSD the OSD has to be offline, but then you don't 
> have access to the block.db because the path is different from outside 
> the container:
>
> ---snip---
> [ceph: root@host1 /]# ceph-volume lvm migrate --osd-id 1 --osd-fsid 
> b4c772aa-07f8-483d-ae58-0ab97b8d0cc4 --from db --target 
> ceph-b1ddff4b-95e8-4b91-b451-a3ea35d16ec0/osd-block-b4c772aa-07f8-483d-ae58-0ab97b8d0cc4 
>
> --> Migrate to existing, Source: ['--devs-source', 
> '/var/lib/ceph/osd/ceph-1/block.db'] Target: 
> /var/lib/ceph/osd/ceph-1/block
>  stdout: inferring bluefs devices from bluestore path
>  stderr: 
> /home/abuild/rpmbuild/BUILD/ceph-15.2.14-84-gb6e5642e260/src/os/bluestore/BlueStore.cc: 
> In function 'int BlueStore::_mount_for_bluefs()' thread 7fde05b96180 
> time 2021-09-29T06:56:24.790161+0000
>  stderr: 
> /home/abuild/rpmbuild/BUILD/ceph-15.2.14-84-gb6e5642e260/src/os/bluestore/BlueStore.cc: 
> 6876: FAILED ceph_assert(r == 0)
>  stderr: 2021-09-29T06:56:24.787+0000 7fde05b96180 -1 
> bluestore(/var/lib/ceph/osd/ceph-1) _lock_fsid failed to lock 
> /var/lib/ceph/osd/ceph-1/fsid (is another ceph-osd still running?)(11) 
> Resource temporarily unavailable
>
>
> # path outside
> host1:~ # ll /var/lib/ceph/152fd738-01bc-11ec-a7fd-fa163e672db2/osd.1/
> insgesamt 60
> lrwxrwxrwx 1 ceph ceph   93 29. Sep 08:43 block -> 
> /dev/ceph-b1ddff4b-95e8-4b91-b451-a3ea35d16ec0/osd-block-b4c772aa-07f8-483d-ae58-0ab97b8d0cc4
> lrwxrwxrwx 1 ceph ceph   90 29. Sep 08:43 block.db -> 
> /dev/ceph-6f1b8f49-daf2-4631-a2ef-12e9452b01ea/osd-db-69b11aa0-af96-443e-8f03-5afa5272131f
> ---snip---
>
>
> But if I shutdown the OSD I can't access the block and block.db 
> devices. I'm not even sure how this is supposed to work with cephadm. 
> Maybe I'm misunderstanding, though. Or is there a way to provide the 
> offline block.db path to 'ceph-volume lvm migrate'?
>
>
>
> Zitat von 胡 玮文 <huww98@xxxxxxxxxxx>:
>
> > You may need to use `ceph-volume lvm migrate’ [1] instead of 
> > ceph-bluestore-tool. If I recall correctly, this is a pretty new 
> > feature, I’m not sure whether it is available to your version.
> >
> > If you use ceph-bluestore-tool, then you need to modify the LVM tags 
> > manually. Please refer to the previous threads, e.g. [2] and some more.
> >
> > [1]: https://docs.ceph.com/en/latest/man/8/ceph-volume/#migrate
> > [2]: 
> > https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/VX23NQ66P3PPEX36T3PYYMHPLBSFLMYA/#JLNDFGXR4ZLY27DHD3RJTTZEDHRZJO4Q
> >
> > 发件人: Szabo, Istvan (Agoda)<mailto:Istvan.Szabo@xxxxxxxxx>
> > 发送时间: 2021年9月28日 18:20
> > 收件人: Eugen Block<mailto:eblock@xxxxxx>; 
> > ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
> > 主题:  Re: is it possible to remove the db+wal from an 
> > external device (nvme)
> >
> > Gave a try of it, so all the 3 osds finally failed :/ Not sure what 
> > went wrong.
> >
> > Do the normal maintenance things, ceph osd set noout, ceph osd set 
> > norebalance, stop the osd and run this command:
> > ceph-bluestore-tool bluefs-bdev-migrate --dev-target 
> > /var/lib/ceph/osd/ceph-0/block --devs-source 
> > /var/lib/ceph/osd/ceph-8/block.db --path /var/lib/ceph/osd/ceph-8/
> > Output:
> > device removed:1 /var/lib/ceph/osd/ceph-8/block.db
> > device added: 1 /dev/dm-2
> >
> > When tried to start I got this in the log:
> > osd.8 0 OSD:init: unable to mount object store
> >  ** ERROR: osd init failed: (13) Permission denied
> > set uid:gid to 167:167 (ceph:ceph)
> > ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) 
> > octopus (stable), process ceph-osd, pid 1512261
> > pidfile_write: ignore empty --pid-file
> >
> > From the another 2 osds the block.db removed and I can start it back.
> > I've zapped the db drive just to be removed from the device 
> > completely and after machine restart none of these 2 osds came back, 
> > I guess missing the db device.
> >
> > Is there any steps missing?
> > 1.Noout+norebalance
> > 2. Stop osd
> > 3. migrate with the above command the block.db to the block.
> > 4. do on the other osds which is sharing the same db device that want 
> > to remove.
> > 5. zap the db device
> > 6. start back the osds.
> >
> > Istvan Szabo
> > Senior Infrastructure Engineer
> > ---------------------------------------------------
> > Agoda Services Co., Ltd.
> > e: istvan.szabo@xxxxxxxxx
> > ---------------------------------------------------
> >
> > -----Original Message-----
> > From: Eugen Block <eblock@xxxxxx>
> > Sent: Monday, September 27, 2021 7:42 PM
> > To: ceph-users@xxxxxxx
> > Subject:  Re: is it possible to remove the db+wal from an 
> > external device (nvme)
> >
> > Email received from the internet. If in doubt, don't click any link 
> > nor open any attachment !
> > ________________________________
> >
> > Hi,
> >
> > I think 'ceph-bluestore-tool bluefs-bdev-migrate' could be of use 
> > here. I haven't tried it in a production environment yet, only in 
> > virtual labs.
> >
> > Regards,
> > Eugen
> >
> >
> > Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@xxxxxxxxx>:
> >
> > > Hi,
> > >
> > > Seems like in our config the nvme device  as a wal+db in front of the
> > > ssd slowing down the ssds osds.
> > > I'd like to avoid to rebuild all the osd-, is there a way somehow
> > > migrate to the "slower device" the wal+db without reinstall?
> > >
> > > Ty
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an
> > > email to ceph-users-leave@xxxxxxx
> >
> >
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an 
> > email to ceph-users-leave@xxxxxxx
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx