Hi,
I just tried with 'ceph-volume lvm migrate' in Octopus but it doesn't
really work. I'm not sure if I'm missing something here, but I believe
it's again the already discussed containers issue. To be able to run
the command for an OSD the OSD has to be offline, but then you don't
have access to the block.db because the path is different from outside
the container:
---snip---
[ceph: root@host1 /]# ceph-volume lvm migrate --osd-id 1 --osd-fsid
b4c772aa-07f8-483d-ae58-0ab97b8d0cc4 --from db --target
ceph-b1ddff4b-95e8-4b91-b451-a3ea35d16ec0/osd-block-b4c772aa-07f8-483d-ae58-0ab97b8d0cc4
--> Migrate to existing, Source: ['--devs-source',
'/var/lib/ceph/osd/ceph-1/block.db'] Target:
/var/lib/ceph/osd/ceph-1/block
stdout: inferring bluefs devices from bluestore path
stderr:
/home/abuild/rpmbuild/BUILD/ceph-15.2.14-84-gb6e5642e260/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_mount_for_bluefs()' thread 7fde05b96180 time
2021-09-29T06:56:24.790161+0000
stderr:
/home/abuild/rpmbuild/BUILD/ceph-15.2.14-84-gb6e5642e260/src/os/bluestore/BlueStore.cc: 6876: FAILED ceph_assert(r ==
0)
stderr: 2021-09-29T06:56:24.787+0000 7fde05b96180 -1
bluestore(/var/lib/ceph/osd/ceph-1) _lock_fsid failed to lock
/var/lib/ceph/osd/ceph-1/fsid (is another ceph-osd still running?)(11)
Resource temporarily unavailable
# path outside
host1:~ # ll /var/lib/ceph/152fd738-01bc-11ec-a7fd-fa163e672db2/osd.1/
insgesamt 60
lrwxrwxrwx 1 ceph ceph 93 29. Sep 08:43 block ->
/dev/ceph-b1ddff4b-95e8-4b91-b451-a3ea35d16ec0/osd-block-b4c772aa-07f8-483d-ae58-0ab97b8d0cc4
lrwxrwxrwx 1 ceph ceph 90 29. Sep 08:43 block.db ->
/dev/ceph-6f1b8f49-daf2-4631-a2ef-12e9452b01ea/osd-db-69b11aa0-af96-443e-8f03-5afa5272131f
---snip---
But if I shutdown the OSD I can't access the block and block.db
devices. I'm not even sure how this is supposed to work with cephadm.
Maybe I'm misunderstanding, though. Or is there a way to provide the
offline block.db path to 'ceph-volume lvm migrate'?
Zitat von 胡 玮文 <huww98@xxxxxxxxxxx>:
You may need to use `ceph-volume lvm migrate’ [1] instead of
ceph-bluestore-tool. If I recall correctly, this is a pretty new
feature, I’m not sure whether it is available to your version.
If you use ceph-bluestore-tool, then you need to modify the LVM tags
manually. Please refer to the previous threads, e.g. [2] and some
more.
[1]: https://docs.ceph.com/en/latest/man/8/ceph-volume/#migrate
[2]:
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/VX23NQ66P3PPEX36T3PYYMHPLBSFLMYA/#JLNDFGXR4ZLY27DHD3RJTTZEDHRZJO4Q
发件人: Szabo, Istvan (Agoda)<mailto:Istvan.Szabo@xxxxxxxxx>
发送时间: 2021年9月28日 18:20
收件人: Eugen Block<mailto:eblock@xxxxxx>;
ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
主题: Re: is it possible to remove the db+wal from an
external device (nvme)
Gave a try of it, so all the 3 osds finally failed :/ Not sure what
went wrong.
Do the normal maintenance things, ceph osd set noout, ceph osd set
norebalance, stop the osd and run this command:
ceph-bluestore-tool bluefs-bdev-migrate --dev-target
/var/lib/ceph/osd/ceph-0/block --devs-source
/var/lib/ceph/osd/ceph-8/block.db --path /var/lib/ceph/osd/ceph-8/
Output:
device removed:1 /var/lib/ceph/osd/ceph-8/block.db
device added: 1 /dev/dm-2
When tried to start I got this in the log:
osd.8 0 OSD:init: unable to mount object store
** ERROR: osd init failed: (13) Permission denied
set uid:gid to 167:167 (ceph:ceph)
ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2)
octopus (stable), process ceph-osd, pid 1512261
pidfile_write: ignore empty --pid-file
From the another 2 osds the block.db removed and I can start it back.
I've zapped the db drive just to be removed from the device
completely and after machine restart none of these 2 osds came back,
I guess missing the db device.
Is there any steps missing?
1.Noout+norebalance
2. Stop osd
3. migrate with the above command the block.db to the block.
4. do on the other osds which is sharing the same db device that
want to remove.
5. zap the db device
6. start back the osds.
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@xxxxxxxxx
---------------------------------------------------
-----Original Message-----
From: Eugen Block <eblock@xxxxxx>
Sent: Monday, September 27, 2021 7:42 PM
To: ceph-users@xxxxxxx
Subject: Re: is it possible to remove the db+wal from
an external device (nvme)
Email received from the internet. If in doubt, don't click any link
nor open any attachment !
________________________________
Hi,
I think 'ceph-bluestore-tool bluefs-bdev-migrate' could be of use
here. I haven't tried it in a production environment yet, only in
virtual labs.
Regards,
Eugen
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@xxxxxxxxx>:
Hi,
Seems like in our config the nvme device as a wal+db in front of the
ssd slowing down the ssds osds.
I'd like to avoid to rebuild all the osd-, is there a way somehow
migrate to the "slower device" the wal+db without reinstall?
Ty
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an
email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an
email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
