Hector,
One more thing to mention - after expansion please run fsck using
ceph-bluestore-tool prior to running osd daemon and collect another log
using CEPH_ARGS variable.
Thanks,
Igor
On 12/27/2018 2:41 PM, Igor Fedotov wrote:
Hi Hector,
I've never tried bluefs-bdev-expand over encrypted volumes but it
works absolutely fine for me in other cases.
So it would be nice to troubleshoot this a bit.
Suggest to do the following:
1) Backup first 8K for all OSD.1 devices (block, db and wal) using dd.
This will probably allow to recover from the failed expansion and
repeat it multiple times.
2) Collect current volume sizes with bluefs-bdev-sizes command and
actual devices sizes using 'lsblk --bytes'.
3) Do lvm volume expansion and then collect dev sizes with 'lsblk
--bytes' once again
4) Invoke bluefs-bdev-expand for osd.1 with
CEPH_ARGS="--debug-bluestore 20 --debug-bluefs 20 --log-file
bluefs-bdev-expand.log"
Perhaps it makes sense to open a ticket at ceph bug tracker to proceed...
Thanks,
Igor
On 12/27/2018 12:19 PM, Hector Martin wrote:
Hi list,
I'm slightly expanding the underlying LV for two OSDs and figured I
could use ceph-bluestore-tool to avoid having to re-create them from
scratch.
I first shut down the OSD, expanded the LV, and then ran:
ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-0
I forgot I was using encryption, so the overlying dm-crypt mapping
stayed the same when I resized the underlying LV. I was surprised by
the output of ceph-bluestore-tool, which suggested a size change by a
significant amount (I was changing the LV size only by a few
percent). I then checked the underlying `block` device and realized
its size had not changed, so the command should've been a no-op. I
then tried to restart the OSD, and it failed with an I/O error. I
ended up re-creating that OSD and letting it recover.
I have another OSD (osd.1) in the original state where I could run
this test again if needed. Unfortunately I don't have the output of
the first test any more.
Is `ceph-bluestore-tool bluefs-bdev-expand` supposed to work? I get
the feeling it gets the size wrong and corrupts OSDs by expanding it
too much. If this is indeed supposed to work I would be happy to test
this again with osd.1 if needed and see if I can get it fixed.
Otherwise I'll just re-create it and move on.
# ceph --version
ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic
(stable)
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
