Thank you everyone, I got it. There is no way to fix out-of-space
bluestore without expanding it.
Therefore, in production we would stick with 99%FREE size for LV, as it
gives operators 'last chance' to repair the cluster in case of
emergency. It's a bit unfortunate that we need to give up the whole per
cent (1 % is too much for 4Tb drives).
On 31/10/2019 15:04, Nathan Fish wrote:
The best way to prevent this on a testing cluster with tiny virtual
drives is probably to lower the various full_ratio's significantly.
On Thu, Oct 31, 2019 at 7:17 AM Paul Emmerich <paul.emmerich@xxxxxxxx> wrote:
BlueStore doesn't handle running out of space gracefully because that
doesn't happen on a real disk because full_ratio (95%) and the
failsafe_full_ratio (97%? some obscure config option) kick in before
that happens.
Yeah, I've also lost some test cluster with tiny disks to this. Usual
reason is keeping it in a degraded state for weeks at a time...
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Thu, Oct 31, 2019 at 10:50 AM George Shuklin
<george.shuklin@xxxxxxxxx> wrote:
Hello.
In my lab a nautilus cluster with a bluestore suddenly went dark. As I
found it had used 98% of the space and most of OSDs (small, 10G each)
went offline. Any attempt to restart them failed with this message:
# /usr/bin/ceph-osd -f --cluster ceph --id 18 --setuser ceph --setgroup
ceph
2019-10-31 09:44:37.591 7f73d54b3f80 -1 osd.18 271 log_to_monitors
{default=true}
2019-10-31 09:44:37.615 7f73bff99700 -1
bluestore(/var/lib/ceph/osd/ceph-18) _do_alloc_write failed to allocate
0x10000 allocated 0x ffffffffffffffe4 min_alloc_size 0x10000 available 0x 0
2019-10-31 09:44:37.615 7f73bff99700 -1
bluestore(/var/lib/ceph/osd/ceph-18) _do_write _do_alloc_write failed
with (28) No space left on device
2019-10-31 09:44:37.615 7f73bff99700 -1
bluestore(/var/lib/ceph/osd/ceph-18) _txc_add_transaction error (28) No
space left on device not handled on operation 10 (op 30, counting from 0)
2019-10-31 09:44:37.615 7f73bff99700 -1
bluestore(/var/lib/ceph/osd/ceph-18) ENOSPC from bluestore,
misconfigured cluster
/build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: In function 'void
BlueStore::_txc_add_transaction(BlueStore::TransContext*,
ObjectStore::Transaction*)' thread 7f73bff99700 time 2019-10-31
09:44:37.620694
/build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: 11455:
ceph_abort_msg("unexpected error")
I was able to recover cluster by adding some more space into VGs for
some of OSDs and using this command:
ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-xx
--command bluefs-bdev-expand
It worked but only because I added some space into OSD.
I'm curious, is there a way to recover such OSD without growing it? On
the old filestore I can just remove some objects to gain space, is this
possible for bluestore? My main concern is that OSD daemon simply
crashes at start, so I can't just add 'more OSD' to cluster - all data
become unavailable, because OSDs are completely dead.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
