BlueStore doesn't handle running out of space gracefully because that doesn't happen on a real disk because full_ratio (95%) and the failsafe_full_ratio (97%? some obscure config option) kick in before that happens. Yeah, I've also lost some test cluster with tiny disks to this. Usual reason is keeping it in a degraded state for weeks at a time... Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Thu, Oct 31, 2019 at 10:50 AM George Shuklin <george.shuklin@xxxxxxxxx> wrote: > > Hello. > > In my lab a nautilus cluster with a bluestore suddenly went dark. As I > found it had used 98% of the space and most of OSDs (small, 10G each) > went offline. Any attempt to restart them failed with this message: > > # /usr/bin/ceph-osd -f --cluster ceph --id 18 --setuser ceph --setgroup > ceph > > 2019-10-31 09:44:37.591 7f73d54b3f80 -1 osd.18 271 log_to_monitors > {default=true} > 2019-10-31 09:44:37.615 7f73bff99700 -1 > bluestore(/var/lib/ceph/osd/ceph-18) _do_alloc_write failed to allocate > 0x10000 allocated 0x ffffffffffffffe4 min_alloc_size 0x10000 available 0x 0 > 2019-10-31 09:44:37.615 7f73bff99700 -1 > bluestore(/var/lib/ceph/osd/ceph-18) _do_write _do_alloc_write failed > with (28) No space left on device > 2019-10-31 09:44:37.615 7f73bff99700 -1 > bluestore(/var/lib/ceph/osd/ceph-18) _txc_add_transaction error (28) No > space left on device not handled on operation 10 (op 30, counting from 0) > 2019-10-31 09:44:37.615 7f73bff99700 -1 > bluestore(/var/lib/ceph/osd/ceph-18) ENOSPC from bluestore, > misconfigured cluster > /build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: In function 'void > BlueStore::_txc_add_transaction(BlueStore::TransContext*, > ObjectStore::Transaction*)' thread 7f73bff99700 time 2019-10-31 > 09:44:37.620694 > /build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: 11455: > ceph_abort_msg("unexpected error") > > I was able to recover cluster by adding some more space into VGs for > some of OSDs and using this command: > > ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-xx > --command bluefs-bdev-expand > > It worked but only because I added some space into OSD. > > I'm curious, is there a way to recover such OSD without growing it? On > the old filestore I can just remove some objects to gain space, is this > possible for bluestore? My main concern is that OSD daemon simply > crashes at start, so I can't just add 'more OSD' to cluster - all data > become unavailable, because OSDs are completely dead. > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |