On Fri, 11 Nov 2016, bobobo1618@xxxxxxxxx wrote: > Any more data needed? > > On Wed, Nov 9, 2016 at 9:29 AM, bobobo1618@xxxxxxxxx > <bobobo1618@xxxxxxxxx> wrote: > > Here it is after running overnight (~9h): http://ix.io/1DNi I'm getting a 500 on that URL... sage > > > > On Tue, Nov 8, 2016 at 11:00 PM, bobobo1618@xxxxxxxxx > > <bobobo1618@xxxxxxxxx> wrote: > >> Ah, I was actually mistaken. After running without Valgrind, it seems > >> I just estimated how slowed down it was. I'll leave it to run > >> overnight as suggested. > >> > >> On Tue, Nov 8, 2016 at 10:44 PM, bobobo1618@xxxxxxxxx > >> <bobobo1618@xxxxxxxxx> wrote: > >>> Okay, I left it for 3h and it seemed to actually stabilise at around > >>> 2.3G: http://ix.io/1DEK > >>> > >>> This was only after disabling other services on the system however. > >>> Generally this much RAM isn't available to Ceph (hence the OOM > >>> previously). > >>> > >>> On Tue, Nov 8, 2016 at 9:00 AM, Mark Nelson <mnelson@xxxxxxxxxx> wrote: > >>>> It should be running much slower through valgrind so probably won't > >>>> accumulate very quickly. That was the problem with the earlier trace, there > >>>> wasn't enough memory used yet to really get us out of the weeds. If it's > >>>> still accumulating quickly, try to wait until the OSD is up to 4+GB RSS if > >>>> you can. I usually kill the valgrind/osd process with SIGTERM to make sure > >>>> the output is preserved. Not sure what will happen with OOM killer as I > >>>> haven't let it get that far before killing. > >>>> > >>>> Mark > >>>> > >>>> On 11/08/2016 10:37 AM, bobobo1618@xxxxxxxxx wrote: > >>>>> > >>>>> Unfortunately I don't think overnight is possible. The OOM will kill it > >>>>> in hours, if not minutes. Will the output be preserved/usable if the > >>>>> process is uncleanly terminated? > >>>>> > >>>>> > >>>>> On 8 Nov 2016 08:33, "Mark Nelson" <mnelson@xxxxxxxxxx > >>>>> <mailto:mnelson@xxxxxxxxxx>> wrote: > >>>>> > >>>>> Heya, > >>>>> > >>>>> Sorry got distracted with other stuff yesterday. Any chance you > >>>>> could run this for longer? It's tough to tell what's going on from > >>>>> this run unfortunately. Maybe overnight if possible. > >>>>> > >>>>> Thanks! > >>>>> Mark > >>>>> > >>>>> > >>>>> > >>>>> On 11/08/2016 01:10 AM, bobobo1618@xxxxxxxxx > >>>>> <mailto:bobobo1618@xxxxxxxxx> wrote: > >>>>> > >>>>> Just bumping this and CCing directly since I foolishly broke the > >>>>> threading on my reply. > >>>>> > >>>>> > >>>>> On 4 Nov. 2016 8:40 pm, "bobobo1618@xxxxxxxxx > >>>>> <mailto:bobobo1618@xxxxxxxxx> > >>>>> <mailto:bobobo1618@xxxxxxxxx <mailto:bobobo1618@xxxxxxxxx>>" > >>>>> <bobobo1618@xxxxxxxxx <mailto:bobobo1618@xxxxxxxxx> > >>>>> > >>>>> <mailto:bobobo1618@xxxxxxxxx <mailto:bobobo1618@xxxxxxxxx>>> > >>>>> wrote: > >>>>> > >>>>> > Then you can view the output data with ms_print or with > >>>>> massif-visualizer. This may help narrow down where in the > >>>>> code we > >>>>> are using the memory. > >>>>> > >>>>> Done! I've dumped the output from ms_print here: > >>>>> http://ix.io/1CrS > >>>>> > >>>>> It seems most of the memory comes from here: > >>>>> > >>>>> 92.78% (998,248,799B) (heap allocation functions) > >>>>> malloc/new/new[], > >>>>> --alloc-fns, etc. > >>>>> ->46.63% (501,656,678B) 0xD38936: > >>>>> ceph::buffer::create_aligned(unsigned int, unsigned int) (in > >>>>> /usr/bin/ceph-osd) > >>>>> | ->45.07% (484,867,174B) 0xDAFED9: > >>>>> AsyncConnection::process() (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | ->45.07% (484,867,174B) 0xC410EB: > >>>>> EventCenter::process_events(int) > >>>>> (in /usr/bin/ceph-osd) > >>>>> | | ->45.07% (484,867,174B) 0xC45210: ??? (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | ->45.07% (484,867,174B) 0xC6FA31D: > >>>>> execute_native_thread_routine (thread.cc:83) > >>>>> | | ->45.07% (484,867,174B) 0xBE06452: start_thread (in > >>>>> /usr/lib/libpthread-2.24.so <http://libpthread-2.24.so> > >>>>> <http://libpthread-2.24.so>) > >>>>> | | ->45.07% (484,867,174B) 0xCFCA7DD: clone (in > >>>>> /usr/lib/libc-2.24.so <http://libc-2.24.so> > >>>>> <http://libc-2.24.so>) > >>>>> | | > >>>>> | ->01.56% (16,789,504B) in 6 places, all below massif's > >>>>> threshold > >>>>> (1.00%) > >>>>> | > >>>>> ->22.70% (244,179,072B) 0x9C9807: BitMapZone::init(long, > >>>>> long, bool) > >>>>> (in /usr/bin/ceph-osd) > >>>>> | ->22.70% (244,179,072B) 0x9CACED: > >>>>> BitMapAreaLeaf::init(long, long, > >>>>> bool) (in /usr/bin/ceph-osd) > >>>>> | ->22.70% (244,179,072B) 0x9CAE88: > >>>>> BitMapAreaLeaf::BitMapAreaLeaf(long, long, bool) (in > >>>>> /usr/bin/ceph-osd) > >>>>> | ->22.67% (243,924,992B) 0x9CAF79: > >>>>> BitMapAreaIN::init(long, long, > >>>>> bool) (in /usr/bin/ceph-osd) > >>>>> | | ->12.46% (134,086,656B) 0x9CAFBE: > >>>>> BitMapAreaIN::init(long, > >>>>> long, bool) (in /usr/bin/ceph-osd) > >>>>> | | | ->12.46% (134,086,656B) 0x9CB237: > >>>>> BitAllocator::init_check(long, long, bmap_alloc_mode, bool, > >>>>> bool) (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | | ->12.46% (134,086,656B) 0x9CB431: > >>>>> BitAllocator::BitAllocator(long, long, bmap_alloc_mode, > >>>>> bool) (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | | ->12.46% (134,086,656B) 0x9C5C32: > >>>>> BitMapAllocator::BitMapAllocator(long, long) (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | | ->12.46% (134,086,656B) 0x968FF1: > >>>>> Allocator::create(std::__cxx11::basic_string<char, > >>>>> std::char_traits<char>, std::allocator<char> >, long, long) > >>>>> (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | | ->12.46% (134,086,656B) 0x87F65C: > >>>>> BlueStore::_open_alloc() (in /usr/bin/ceph-osd) > >>>>> | | | ->12.46% (134,086,656B) 0x8D8CDD: > >>>>> BlueStore::mount() (in /usr/bin/ceph-osd) > >>>>> | | | ->12.46% (134,086,656B) 0x4C15EA: > >>>>> OSD::init() > >>>>> (in /usr/bin/ceph-osd) > >>>>> | | | ->12.46% (134,086,656B) 0x40854C: > >>>>> main (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | | > >>>>> | | ->10.21% (109,838,336B) 0x9CB00B: > >>>>> BitMapAreaIN::init(long, > >>>>> long, bool) (in /usr/bin/ceph-osd) > >>>>> | | ->10.21% (109,838,336B) 0x9CB237: > >>>>> BitAllocator::init_check(long, long, bmap_alloc_mode, bool, > >>>>> bool) (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | ->10.21% (109,838,336B) 0x9CB431: > >>>>> BitAllocator::BitAllocator(long, long, bmap_alloc_mode, > >>>>> bool) (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | ->10.21% (109,838,336B) 0x9C5C32: > >>>>> BitMapAllocator::BitMapAllocator(long, long) (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | ->10.21% (109,838,336B) 0x968FF1: > >>>>> Allocator::create(std::__cxx11::basic_string<char, > >>>>> std::char_traits<char>, std::allocator<char> >, long, long) > >>>>> (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | ->10.21% (109,838,336B) 0x87F65C: > >>>>> BlueStore::_open_alloc() (in /usr/bin/ceph-osd) > >>>>> | | ->10.21% (109,838,336B) 0x8D8CDD: > >>>>> BlueStore::mount() (in /usr/bin/ceph-osd) > >>>>> | | ->10.21% (109,838,336B) 0x4C15EA: > >>>>> OSD::init() > >>>>> (in /usr/bin/ceph-osd) > >>>>> | | ->10.21% (109,838,336B) 0x40854C: > >>>>> main (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | > >>>>> | ->00.02% (254,080B) in 1+ places, all below ms_print's > >>>>> threshold (01.00%) > >>>>> | > >>>>> ->12.77% (137,350,728B) 0x9CACD8: BitMapAreaLeaf::init(long, > >>>>> long, > >>>>> bool) (in /usr/bin/ceph-osd) > >>>>> | ->12.77% (137,350,728B) 0x9CAE88: > >>>>> BitMapAreaLeaf::BitMapAreaLeaf(long, long, bool) (in > >>>>> /usr/bin/ceph-osd) > >>>>> | ->12.75% (137,207,808B) 0x9CAF79: > >>>>> BitMapAreaIN::init(long, long, > >>>>> bool) (in /usr/bin/ceph-osd) > >>>>> | | ->07.01% (75,423,744B) 0x9CAFBE: > >>>>> BitMapAreaIN::init(long, long, > >>>>> bool) (in /usr/bin/ceph-osd) > >>>>> | | | ->07.01% (75,423,744B) 0x9CB237: > >>>>> BitAllocator::init_check(long, long, bmap_alloc_mode, bool, > >>>>> bool) (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | | ->07.01% (75,423,744B) 0x9CB431: > >>>>> BitAllocator::BitAllocator(long, long, bmap_alloc_mode, > >>>>> bool) (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | | ->07.01% (75,423,744B) 0x9C5C32: > >>>>> BitMapAllocator::BitMapAllocator(long, long) (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | | ->07.01% (75,423,744B) 0x968FF1: > >>>>> Allocator::create(std::__cxx11::basic_string<char, > >>>>> std::char_traits<char>, std::allocator<char> >, long, long) > >>>>> (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | | ->07.01% (75,423,744B) 0x87F65C: > >>>>> BlueStore::_open_alloc() (in /usr/bin/ceph-osd) > >>>>> | | | ->07.01% (75,423,744B) 0x8D8CDD: > >>>>> BlueStore::mount() > >>>>> (in /usr/bin/ceph-osd) > >>>>> | | | ->07.01% (75,423,744B) 0x4C15EA: > >>>>> OSD::init() (in > >>>>> /usr/bin/ceph-osd) > >>>>> | | | ->07.01% (75,423,744B) 0x40854C: main > >>>>> (in > >>>>> /usr/bin/ceph-osd) > >>>>> > >>>> > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |