I hit this a couple times and wondered the same thing. Why does the OSD need to bail when it runs out of journal space? On Wed, Jun 20, 2012 at 3:56 PM, Travis Rhoden <trhoden@xxxxxxxxx> wrote: > Not sure if this is a bug or not. It was definitely user error -- but > since the OSD process bailed, figured I would report it. > > I had /tmpfs mounted with 2.5GB of space: > > tmpfs on /tmpfs type tmpfs (rw,size=2560m) > > Then I decided to increase my journal size to 5G, but forgot to > increase the limit on /tmpfs. =) > > osd journal size = 5000 > > > Predictably, things didn't go well when I ran a rados bench that > filled up the journal. I'm not sure if such a case can be handled > more gracefully: > > > -4> 2012-06-20 12:39:36.648773 7fc042a5f780 1 journal _open > /tmpfs/osd.2.journal fd 30: 5242880000 bytes, block size 4096 bytes, > directio = 0, aio = 0 > -3> 2012-06-20 12:42:23.179164 7fc02e1ad700 1 > CephxAuthorizeHandler::verify_authorizer isvalid=1 > -2> 2012-06-20 12:42:46.643205 7fc0396cf700 -1 journal > FileJournal::write_bl : write_fd failed: (28) No space left on device > -1> 2012-06-20 12:42:46.643245 7fc0396cf700 -1 journal > FileJournal::do_write: write_bl(pos=2678079488) failed > 0> 2012-06-20 12:42:46.676991 7fc0396cf700 -1 os/FileJournal.cc: > In function 'void FileJournal::do_write(ceph::bufferlist&)' thread > 7fc0396cf700 time 2012-06-20 12:42:46.643315 > os/FileJournal.cc: 994: FAILED assert(0) > > ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372) > 1: (FileJournal::do_write(ceph::buffer::list&)+0xe22) [0x653082] > 2: (FileJournal::write_thread_entry()+0x735) [0x659545] > 3: (FileJournal::Writer::entry()+0xd) [0x5de41d] > 4: (()+0x7e9a) [0x7fc042434e9a] > 5: (clone()+0x6d) [0x7fc0409e94bd] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > --- end dump of recent events --- > 2012-06-20 12:42:46.693963 7fc0396cf700 -1 *** Caught signal (Aborted) ** > in thread 7fc0396cf700 > > ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372) > 1: /usr/bin/ceph-osd() [0x6eb32a] > 2: (()+0xfcb0) [0x7fc04243ccb0] > 3: (gsignal()+0x35) [0x7fc04092d445] > 4: (abort()+0x17b) [0x7fc040930bab] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fc04127b69d] > 6: (()+0xb5846) [0x7fc041279846] > 7: (()+0xb5873) [0x7fc041279873] > 8: (()+0xb596e) [0x7fc04127996e] > 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x282) [0x79dd02] > 10: (FileJournal::do_write(ceph::buffer::list&)+0xe22) [0x653082] > 11: (FileJournal::write_thread_entry()+0x735) [0x659545] > 12: (FileJournal::Writer::entry()+0xd) [0x5de41d] > 13: (()+0x7e9a) [0x7fc042434e9a] > 14: (clone()+0x6d) [0x7fc0409e94bd] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > --- begin dump of recent events --- > 0> 2012-06-20 12:42:46.693963 7fc0396cf700 -1 *** Caught signal > (Aborted) ** > in thread 7fc0396cf700 > > ceph version 0.47.2 (commit:8bf9fde89bd6ebc4b0645b2fe02dadb1c17ad372) > 1: /usr/bin/ceph-osd() [0x6eb32a] > 2: (()+0xfcb0) [0x7fc04243ccb0] > 3: (gsignal()+0x35) [0x7fc04092d445] > 4: (abort()+0x17b) [0x7fc040930bab] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fc04127b69d] > 6: (()+0xb5846) [0x7fc041279846] > 7: (()+0xb5873) [0x7fc041279873] > 8: (()+0xb596e) [0x7fc04127996e] > 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x282) [0x79dd02] > 10: (FileJournal::do_write(ceph::buffer::list&)+0xe22) [0x653082] > 11: (FileJournal::write_thread_entry()+0x735) [0x659545] > 12: (FileJournal::Writer::entry()+0xd) [0x5de41d] > 13: (()+0x7e9a) [0x7fc042434e9a] > 14: (clone()+0x6d) [0x7fc0409e94bd] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > --- end dump of recent events --- > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
