I’ve not tried it, but how about:
cephadm shell -n osd.0
then run “ceph-volume” commands in the newly opened shell. The
directory structure seems fine.
$ sudo cephadm shell -n osd.0
Inferring fsid e88d509a-f6fc-11ea-b25d-a0423f3ac864
Inferring config
/var/lib/ceph/e88d509a-f6fc-11ea-b25d-a0423f3ac864/osd.0/config
Using recent ceph image
cr.example.com/infra/ceph@sha256:8a0f6f285edcd6488e2c91d3f9fa43534d37d7a9b37db1e0ff6691aae6466530
root@host0:/# ll /var/lib/ceph/osd/ceph-0/
total 68
drwx------ 2 ceph ceph 4096 Sep 20 04:15 ./
drwxr-x--- 1 ceph ceph 4096 Sep 29 13:32 ../
lrwxrwxrwx 1 ceph ceph 24 Sep 20 04:15 block -> /dev/ceph-hdd/osd.0.data
lrwxrwxrwx 1 ceph ceph 23 Sep 20 04:15 block.db -> /dev/ubuntu-vg/osd.0.db
-rw------- 1 ceph ceph 37 Sep 20 04:15 ceph_fsid
-rw------- 1 ceph ceph 387 Jun 21 13:24 config
-rw------- 1 ceph ceph 37 Sep 20 04:15 fsid
-rw------- 1 ceph ceph 55 Sep 20 04:15 keyring
-rw------- 1 ceph ceph 6 Sep 20 04:15 ready
-rw------- 1 ceph ceph 3 Apr 2 01:46 require_osd_release
-rw------- 1 ceph ceph 10 Sep 20 04:15 type
-rw------- 1 ceph ceph 38 Sep 17 14:26 unit.configured
-rw------- 1 ceph ceph 48 Nov 9 2020 unit.created
-rw------- 1 ceph ceph 35 Sep 17 14:26 unit.image
-rw------- 1 ceph ceph 306 Sep 17 14:26 unit.meta
-rw------- 1 ceph ceph 1317 Sep 17 14:26 unit.poststop
-rw------- 1 ceph ceph 3021 Sep 17 14:26 unit.run
-rw------- 1 ceph ceph 142 Sep 17 14:26 unit.stop
-rw------- 1 ceph ceph 2 Sep 20 04:15 whoami
发件人: Eugen Block<mailto:eblock@xxxxxx>
发送时间: 2021年9月29日 21:29
收件人: Igor Fedotov<mailto:ifedotov@xxxxxxx>
抄送: 胡 玮文<mailto:huww98@xxxxxxxxxxx>; Szabo, Istvan
(Agoda)<mailto:Istvan.Szabo@xxxxxxxxx>;
ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
主题: Re: Re: 回复: [ceph-users] Re: is it possible to
remove the db+wal from an external device (nvme)
Hi Igor,
thanks for your input. I haven't done this in a prod env yet either,
still playing around in a virtual lab env.
I tried the symlink suggestion but it's not that easy, because it
looks different underneath the ceph directory than ceph-volume expects
it. These are the services underneath:
ses7-host1:~ # ll /var/lib/ceph/152fd738-01bc-11ec-a7fd-fa163e672db2/
insgesamt 48
drwx------ 3 root root 4096 16. Sep 16:11 alertmanager.ses7-host1
drwx------ 3 ceph ceph 4096 29. Sep 09:03 crash
drwx------ 2 ceph ceph 4096 16. Sep 16:39 crash.ses7-host1
drwx------ 4 messagebus lp 4096 16. Sep 16:23 grafana.ses7-host1
drw-rw---- 2 root root 4096 24. Aug 10:00 home
drwx------ 2 ceph ceph 4096 16. Sep 16:37 mgr.ses7-host1.wmgyit
drwx------ 3 ceph ceph 4096 16. Sep 16:37 mon.ses7-host1
drwx------ 2 nobody nobody 4096 16. Sep 16:37 node-exporter.ses7-host1
drwx------ 2 ceph ceph 4096 29. Sep 08:43 osd.0
drwx------ 2 ceph ceph 4096 29. Sep 15:11 osd.1
drwx------ 4 root root 4096 16. Sep 16:12 prometheus.ses7-host1
While the directory in a non-containerized deployment looks like this:
nautilus:~ # ll /var/lib/ceph/osd/ceph-0/
insgesamt 24
lrwxrwxrwx 1 ceph ceph 93 29. Sep 12:21 block ->
/dev/ceph-a6d78a29-637f-494b-a839-76251fcff67e/osd-block-39340a48-54b3-4689-9896-f54d005c535d
-rw------- 1 ceph ceph 37 29. Sep 12:21 ceph_fsid
-rw------- 1 ceph ceph 37 29. Sep 12:21 fsid
-rw------- 1 ceph ceph 55 29. Sep 12:21 keyring
-rw------- 1 ceph ceph 6 29. Sep 12:21 ready
-rw------- 1 ceph ceph 10 29. Sep 12:21 type
-rw------- 1 ceph ceph 2 29. Sep 12:21 whoami
But even if I create the symlink to the osd directory it fails because
I only have ceph-volume within the containers where the symlink is not
visible to cephadm.
ses7-host1:~ # ll /var/lib/ceph/osd/ceph-1
lrwxrwxrwx 1 root root 57 29. Sep 15:08 /var/lib/ceph/osd/ceph-1 ->
/var/lib/ceph/152fd738-01bc-11ec-a7fd-fa163e672db2/osd.1/
ses7-host1:~ # cephadm ceph-volume lvm migrate --osd-id 1 --osd-fsid
b4c772aa-07f8-483d-ae58-0ab97b8d0cc4 --from db --target
ceph-b1ddff4b-95e8-4b91-b451-a3ea35d16ec0/osd-block-b4c772aa-07f8-483d-ae58-0ab97b8d0cc4
Inferring fsid 152fd738-01bc-11ec-a7fd-fa163e672db2
[...]
/usr/bin/podman: stderr --> Migrate to existing, Source:
['--devs-source', '/var/lib/ceph/osd/ceph-1/block.db'] Target:
/var/lib/ceph/osd/ceph-1/block
/usr/bin/podman: stderr stdout: inferring bluefs devices from
bluestore path
/usr/bin/podman: stderr stderr: can't migrate
/var/lib/ceph/osd/ceph-1/block.db, not a valid bluefs volume
/usr/bin/podman: stderr --> Failed to migrate device, error code:1
/usr/bin/podman: stderr --> Undoing lv tag set
/usr/bin/podman: stderr Failed to migrate to :
ceph-b1ddff4b-95e8-4b91-b451-a3ea35d16ec0/osd-block-b4c772aa-07f8-483d-ae58-0ab97b8d0cc4
Traceback (most recent call last):
File "/usr/sbin/cephadm", line 6225, in <module>
r = args.func()
File "/usr/sbin/cephadm", line 1363, in _infer_fsid
return func()
File "/usr/sbin/cephadm", line 1422, in _infer_image
return func()
File "/usr/sbin/cephadm", line 3687, in command_ceph_volume
out, err, code = call_throws(c.run_cmd(),
verbosity=CallVerbosity.VERBOSE)
File "/usr/sbin/cephadm", line 1101, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
[...]
I could install the package ceph-osd (where ceph-volume is packaged
in) but it's not available by default (as you see this is a SES 7
environment).
I'm not sure what the design is here, it feels like the ceph-volume
migrate command is not applicable to containers yet.
Regards,
Eugen
Zitat von Igor Fedotov <ifedotov@xxxxxxx>:
Hi Eugen,
indeed this looks like an issue related to containerized deployment,
"ceph-volume lvm migrate" expects osd folder to be under
/var/lib/ceph/osd:
stderr: 2021-09-29T06:56:24.787+0000 7fde05b96180 -1
bluestore(/var/lib/ceph/osd/ceph-1) _lock_fsid failed to lock
/var/lib/ceph/osd/ceph-1/fsid (is another ceph-osd still
running?)(11) Resource temporarily unavailable
As a workaround you might want to try to create a symlink to your
actual location before issuing the migrate command:
/var/lib/ceph/osd ->
/var/lib/ceph/152fd738-01bc-11ec-a7fd-fa163e672db2/
More complicated (and more general IMO) way would be to run the
migrate command from within a container deployed similarly (i.e.
with all the proper subfolder mappings) to ceph-osd one. Just
speculating - not a big expert in containers and never tried that
with properly deployed production cluster...
Thanks,
Igor
On 9/29/2021 10:07 AM, Eugen Block wrote:
Hi,
I just tried with 'ceph-volume lvm migrate' in Octopus but it
doesn't really work. I'm not sure if I'm missing something here,
but I believe it's again the already discussed containers issue. To
be able to run the command for an OSD the OSD has to be offline,
but then you don't have access to the block.db because the path is
different from outside the container:
---snip---
[ceph: root@host1 /]# ceph-volume lvm migrate --osd-id 1 --osd-fsid
b4c772aa-07f8-483d-ae58-0ab97b8d0cc4 --from db --target
ceph-b1ddff4b-95e8-4b91-b451-a3ea35d16ec0/osd-block-b4c772aa-07f8-483d-ae58-0ab97b8d0cc4 --> Migrate to existing, Source: ['--devs-source',
'/var/lib/ceph/osd/ceph-1/block.db']
Target:
/var/lib/ceph/osd/ceph-1/block
stdout: inferring bluefs devices from bluestore path
stderr:
/home/abuild/rpmbuild/BUILD/ceph-15.2.14-84-gb6e5642e260/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_mount_for_bluefs()' thread
7fde05b96180
time
2021-09-29T06:56:24.790161+0000
stderr:
/home/abuild/rpmbuild/BUILD/ceph-15.2.14-84-gb6e5642e260/src/os/bluestore/BlueStore.cc: 6876: FAILED
ceph_assert(r
==
0)
stderr: 2021-09-29T06:56:24.787+0000 7fde05b96180 -1
bluestore(/var/lib/ceph/osd/ceph-1) _lock_fsid failed to lock
/var/lib/ceph/osd/ceph-1/fsid (is another ceph-osd still
running?)(11) Resource temporarily unavailable
# path outside
host1:~ # ll /var/lib/ceph/152fd738-01bc-11ec-a7fd-fa163e672db2/osd.1/
insgesamt 60
lrwxrwxrwx 1 ceph ceph 93 29. Sep 08:43 block ->
/dev/ceph-b1ddff4b-95e8-4b91-b451-a3ea35d16ec0/osd-block-b4c772aa-07f8-483d-ae58-0ab97b8d0cc4
lrwxrwxrwx 1 ceph ceph 90 29. Sep 08:43 block.db ->
/dev/ceph-6f1b8f49-daf2-4631-a2ef-12e9452b01ea/osd-db-69b11aa0-af96-443e-8f03-5afa5272131f
---snip---
But if I shutdown the OSD I can't access the block and block.db
devices. I'm not even sure how this is supposed to work with
cephadm. Maybe I'm misunderstanding, though. Or is there a way to
provide the offline block.db path to 'ceph-volume lvm migrate'?
Zitat von 胡 玮文 <huww98@xxxxxxxxxxx>:
You may need to use `ceph-volume lvm migrate’ [1] instead of
ceph-bluestore-tool. If I recall correctly, this is a pretty new
feature, I’m not sure whether it is available to your version.
If you use ceph-bluestore-tool, then you need to modify the LVM
tags manually. Please refer to the previous threads, e.g. [2] and
some more.
[1]: https://docs.ceph.com/en/latest/man/8/ceph-volume/#migrate
[2]:
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/VX23NQ66P3PPEX36T3PYYMHPLBSFLMYA/#JLNDFGXR4ZLY27DHD3RJTTZEDHRZJO4Q
发件人: Szabo, Istvan (Agoda)<mailto:Istvan.Szabo@xxxxxxxxx>
发送时间: 2021年9月28日 18:20
收件人: Eugen Block<mailto:eblock@xxxxxx>;
ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
主题: Re: is it possible to remove the db+wal from an
external device (nvme)
Gave a try of it, so all the 3 osds finally failed :/ Not sure
what went wrong.
Do the normal maintenance things, ceph osd set noout, ceph osd set
norebalance, stop the osd and run this command:
ceph-bluestore-tool bluefs-bdev-migrate --dev-target
/var/lib/ceph/osd/ceph-0/block --devs-source
/var/lib/ceph/osd/ceph-8/block.db --path /var/lib/ceph/osd/ceph-8/
Output:
device removed:1 /var/lib/ceph/osd/ceph-8/block.db
device added: 1 /dev/dm-2
When tried to start I got this in the log:
osd.8 0 OSD:init: unable to mount object store
** ERROR: osd init failed: (13) Permission denied
set uid:gid to 167:167 (ceph:ceph)
ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2)
octopus (stable), process ceph-osd, pid 1512261
pidfile_write: ignore empty --pid-file
From the another 2 osds the block.db removed and I can start it back.
I've zapped the db drive just to be removed from the device
completely and after machine restart none of these 2 osds came
back, I guess missing the db device.
Is there any steps missing?
1.Noout+norebalance
2. Stop osd
3. migrate with the above command the block.db to the block.
4. do on the other osds which is sharing the same db device that
want to remove.
5. zap the db device
6. start back the osds.
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@xxxxxxxxx
---------------------------------------------------
-----Original Message-----
From: Eugen Block <eblock@xxxxxx>
Sent: Monday, September 27, 2021 7:42 PM
To: ceph-users@xxxxxxx
Subject: Re: is it possible to remove the db+wal from
an external device (nvme)
Email received from the internet. If in doubt, don't click any
link nor open any attachment !
________________________________
Hi,
I think 'ceph-bluestore-tool bluefs-bdev-migrate' could be of use
here. I haven't tried it in a production environment yet, only in
virtual labs.
Regards,
Eugen
Zitat von "Szabo, Istvan (Agoda)" <Istvan.Szabo@xxxxxxxxx>:
Hi,
Seems like in our config the nvme device as a wal+db in front of the
ssd slowing down the ssds osds.
I'd like to avoid to rebuild all the osd-, is there a way somehow
migrate to the "slower device" the wal+db without reinstall?
Ty
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an
email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send
an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
