The best way to prevent this on a testing cluster with tiny virtual drives is probably to lower the various full_ratio's significantly. On Thu, Oct 31, 2019 at 7:17 AM Paul Emmerich <paul.emmerich@xxxxxxxx> wrote: > > BlueStore doesn't handle running out of space gracefully because that > doesn't happen on a real disk because full_ratio (95%) and the > failsafe_full_ratio (97%? some obscure config option) kick in before > that happens. > > Yeah, I've also lost some test cluster with tiny disks to this. Usual > reason is keeping it in a degraded state for weeks at a time... > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Thu, Oct 31, 2019 at 10:50 AM George Shuklin > <george.shuklin@xxxxxxxxx> wrote: > > > > Hello. > > > > In my lab a nautilus cluster with a bluestore suddenly went dark. As I > > found it had used 98% of the space and most of OSDs (small, 10G each) > > went offline. Any attempt to restart them failed with this message: > > > > # /usr/bin/ceph-osd -f --cluster ceph --id 18 --setuser ceph --setgroup > > ceph > > > > 2019-10-31 09:44:37.591 7f73d54b3f80 -1 osd.18 271 log_to_monitors > > {default=true} > > 2019-10-31 09:44:37.615 7f73bff99700 -1 > > bluestore(/var/lib/ceph/osd/ceph-18) _do_alloc_write failed to allocate > > 0x10000 allocated 0x ffffffffffffffe4 min_alloc_size 0x10000 available 0x 0 > > 2019-10-31 09:44:37.615 7f73bff99700 -1 > > bluestore(/var/lib/ceph/osd/ceph-18) _do_write _do_alloc_write failed > > with (28) No space left on device > > 2019-10-31 09:44:37.615 7f73bff99700 -1 > > bluestore(/var/lib/ceph/osd/ceph-18) _txc_add_transaction error (28) No > > space left on device not handled on operation 10 (op 30, counting from 0) > > 2019-10-31 09:44:37.615 7f73bff99700 -1 > > bluestore(/var/lib/ceph/osd/ceph-18) ENOSPC from bluestore, > > misconfigured cluster > > /build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: In function 'void > > BlueStore::_txc_add_transaction(BlueStore::TransContext*, > > ObjectStore::Transaction*)' thread 7f73bff99700 time 2019-10-31 > > 09:44:37.620694 > > /build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: 11455: > > ceph_abort_msg("unexpected error") > > > > I was able to recover cluster by adding some more space into VGs for > > some of OSDs and using this command: > > > > ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-xx > > --command bluefs-bdev-expand > > > > It worked but only because I added some space into OSD. > > > > I'm curious, is there a way to recover such OSD without growing it? On > > the old filestore I can just remove some objects to gain space, is this > > possible for bluestore? My main concern is that OSD daemon simply > > crashes at start, so I can't just add 'more OSD' to cluster - all data > > become unavailable, because OSDs are completely dead. > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |