On Tue, May 21, 2013 at 9:01 AM, Emil Renner Berthing <ceph@xxxxxxxx> wrote: > On 21 May 2013 17:55, Gregory Farnum <greg@xxxxxxxxxxx> wrote: >> On Tue, May 21, 2013 at 8:44 AM, Emil Renner Berthing <ceph@xxxxxxxx> wrote: >>> Hi Greg, >>> >>> Here are some more stats on our servers: >>> - each server has 64GB ram, >>> - there are 12 OSDs pr. server, >>> - each OSD uses around 1.5 GB of memory, >>> - we have 18432 PGs, >>> - around 5 to 10 MB writes/s is written to each OSD and almost no reads (yet). >> >> What interface are you writing with? How many OSD servers are there? > > We're using librados and there are 132 OSDs so far. Okay, so the allocation is happening in the depths of LevelDB — maybe the issue is there somewhere. Are you doing anything weird with omap, snapshots, or xattrs? -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com > > /Emil > >> -Greg >> >> >>> >>> /Emil >>> >>> On 21 May 2013 17:10, Gregory Farnum <greg@xxxxxxxxxxx> wrote: >>>> That looks like an attempt at a 370MB memory allocation. :? What's the >>>> memory use like on those nodes, and what's your workload? >>>> -Greg >>>> >>>> >>>> On Tuesday, May 21, 2013, Emil Renner Berthing wrote: >>>>> >>>>> Hi, >>>>> >>>>> We're experiencing random segmentation faults in the osd daemon from >>>>> the 0.61.2-1~bpo70+1 debian packages. It happens across all our >>>>> servers and we've seen around 40 crashes in the last week. >>>>> >>>>> It seems to happen more often on loaded servers, but at least they all >>>>> return the same error in the logs. An example can be found here: >>>>> http://esmil.dk/osdcrash.txt >>>>> >>>>> Here is the backtrace from the core dump: >>>>> >>>>> #0 0x00007f87b148eefb in raise () from >>>>> /lib/x86_64-linux-gnu/libpthread.so.0 >>>>> #1 0x0000000000853a89 in reraise_fatal (signum=11) at >>>>> global/signal_handler.cc:58 >>>>> #2 handle_fatal_signal (signum=11) at global/signal_handler.cc:104 >>>>> #3 <signal handler called> >>>>> #4 0x00007f87b06a96f3 in do_malloc (size=388987616) at >>>>> src/tcmalloc.cc:1059 >>>>> #5 cpp_alloc (nothrow=false, size=388987616) at src/tcmalloc.cc:1354 >>>>> #6 tc_new (size=388987616) at src/tcmalloc.cc:1530 >>>>> #7 0x00007f87a60c89b0 in ?? () >>>>> #8 0x00000000172f7ae0 in ?? () >>>>> #9 0x00007f87b0459b21 in ?? () from >>>>> /usr/lib/x86_64-linux-gnu/libleveldb.so.1 >>>>> #10 0x00007f87b0456ba8 in ?? () from >>>>> /usr/lib/x86_64-linux-gnu/libleveldb.so.1 >>>>> #11 0x00007f87b04424d4 in ?? () from >>>>> /usr/lib/x86_64-linux-gnu/libleveldb.so.1 >>>>> #12 0x0000000000840977 in >>>>> LevelDBStore::LevelDBWholeSpaceIteratorImpl::lower_bound >>>>> (this=0x20910a20, prefix=..., to=...) at os/LevelDBStore.h:204 >>>>> #13 0x000000000083f351 in LevelDBStore::get (this=<optimized out>, >>>>> prefix=..., keys=..., out=0x7f87a60c8d00) at os/LevelDBStore.cc:106 >>>>> #14 0x0000000000838449 in DBObjectMap::_lookup_map_header >>>>> (this=this@entry=0x316d4a0, hoid=...) at os/DBObjectMap.cc:1080 >>>>> #15 0x000000000083e4a9 in DBObjectMap::lookup_map_header >>>>> (this=this@entry=0x316d4a0, hoid=...) at os/DBObjectMap.h:404 >>>>> #16 0x0000000000839e06 in DBObjectMap::rm_keys (this=0x316d4a0, >>>>> hoid=..., to_clear=..., spos=0x7f87a60c9400) at os/DBObjectMap.cc:696 >>>>> #17 0x00000000007f40c1 in FileStore::_omap_rmkeys >>>>> (this=this@entry=0x3188000, cid=..., hoid=..., keys=..., spos=...) at >>>>> os/FileStore.cc:4765 >>>>> #18 0x000000000080f610 in FileStore::_do_transaction >>>>> (this=this@entry=0x3188000, t=..., op_seq=op_seq@entry=4760123, >>>>> trans_num=trans_num@entry=0) at os/FileStore.cc:2595 >>>>> #19 0x0000000000812999 in FileStore::_do_transactions >>>>> (this=this@entry=0x3188000, tls=..., op_seq=4760123, >>>>> handle=handle@entry=0x7f87a60c9b80) at os/FileStore.cc:2151 >>>>> #20 0x0000000000812b2e in FileStore::_do_op (this=0x3188000, >>>>> osr=<optimized out>, handle=...) at os/FileStore.cc:1985 >>>>> #21 0x00000000008f52ea in ThreadPool::worker (this=0x3188a08, >>>>> wt=0x319c3e0) at common/WorkQueue.cc:119 >>>>> #22 0x00000000008f6590 in ThreadPool::WorkThread::entry >>>>> (this=<optimized out>) at common/WorkQueue.h:316 >>>>> #23 0x00007f87b1486b50 in start_thread () from >>>>> /lib/x86_64-linux-gnu/libpthread.so.0 >>>>> #24 0x00007f87af9c2a7d in clone () from /lib/x86_64-linux-gnu/libc.so.6 >>>>> #25 0x0000000000000000 in ?? () >>>>> >>>>> Please let me know if can provide any other info to help find this bug. >>>>> /Emil >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> >>>> >>>> -- >>>> Software Engineer #42 @ http://inktank.com | http://ceph.com -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
