Here it is after running overnight (~9h): http://ix.io/1DNi On Tue, Nov 8, 2016 at 11:00 PM, bobobo1618@xxxxxxxxx <bobobo1618@xxxxxxxxx> wrote: > Ah, I was actually mistaken. After running without Valgrind, it seems > I just estimated how slowed down it was. I'll leave it to run > overnight as suggested. > > On Tue, Nov 8, 2016 at 10:44 PM, bobobo1618@xxxxxxxxx > <bobobo1618@xxxxxxxxx> wrote: >> Okay, I left it for 3h and it seemed to actually stabilise at around >> 2.3G: http://ix.io/1DEK >> >> This was only after disabling other services on the system however. >> Generally this much RAM isn't available to Ceph (hence the OOM >> previously). >> >> On Tue, Nov 8, 2016 at 9:00 AM, Mark Nelson <mnelson@xxxxxxxxxx> wrote: >>> It should be running much slower through valgrind so probably won't >>> accumulate very quickly. That was the problem with the earlier trace, there >>> wasn't enough memory used yet to really get us out of the weeds. If it's >>> still accumulating quickly, try to wait until the OSD is up to 4+GB RSS if >>> you can. I usually kill the valgrind/osd process with SIGTERM to make sure >>> the output is preserved. Not sure what will happen with OOM killer as I >>> haven't let it get that far before killing. >>> >>> Mark >>> >>> On 11/08/2016 10:37 AM, bobobo1618@xxxxxxxxx wrote: >>>> >>>> Unfortunately I don't think overnight is possible. The OOM will kill it >>>> in hours, if not minutes. Will the output be preserved/usable if the >>>> process is uncleanly terminated? >>>> >>>> >>>> On 8 Nov 2016 08:33, "Mark Nelson" <mnelson@xxxxxxxxxx >>>> <mailto:mnelson@xxxxxxxxxx>> wrote: >>>> >>>> Heya, >>>> >>>> Sorry got distracted with other stuff yesterday. Any chance you >>>> could run this for longer? It's tough to tell what's going on from >>>> this run unfortunately. Maybe overnight if possible. >>>> >>>> Thanks! >>>> Mark >>>> >>>> >>>> >>>> On 11/08/2016 01:10 AM, bobobo1618@xxxxxxxxx >>>> <mailto:bobobo1618@xxxxxxxxx> wrote: >>>> >>>> Just bumping this and CCing directly since I foolishly broke the >>>> threading on my reply. >>>> >>>> >>>> On 4 Nov. 2016 8:40 pm, "bobobo1618@xxxxxxxxx >>>> <mailto:bobobo1618@xxxxxxxxx> >>>> <mailto:bobobo1618@xxxxxxxxx <mailto:bobobo1618@xxxxxxxxx>>" >>>> <bobobo1618@xxxxxxxxx <mailto:bobobo1618@xxxxxxxxx> >>>> >>>> <mailto:bobobo1618@xxxxxxxxx <mailto:bobobo1618@xxxxxxxxx>>> >>>> wrote: >>>> >>>> > Then you can view the output data with ms_print or with >>>> massif-visualizer. This may help narrow down where in the >>>> code we >>>> are using the memory. >>>> >>>> Done! I've dumped the output from ms_print here: >>>> http://ix.io/1CrS >>>> >>>> It seems most of the memory comes from here: >>>> >>>> 92.78% (998,248,799B) (heap allocation functions) >>>> malloc/new/new[], >>>> --alloc-fns, etc. >>>> ->46.63% (501,656,678B) 0xD38936: >>>> ceph::buffer::create_aligned(unsigned int, unsigned int) (in >>>> /usr/bin/ceph-osd) >>>> | ->45.07% (484,867,174B) 0xDAFED9: >>>> AsyncConnection::process() (in >>>> /usr/bin/ceph-osd) >>>> | | ->45.07% (484,867,174B) 0xC410EB: >>>> EventCenter::process_events(int) >>>> (in /usr/bin/ceph-osd) >>>> | | ->45.07% (484,867,174B) 0xC45210: ??? (in >>>> /usr/bin/ceph-osd) >>>> | | ->45.07% (484,867,174B) 0xC6FA31D: >>>> execute_native_thread_routine (thread.cc:83) >>>> | | ->45.07% (484,867,174B) 0xBE06452: start_thread (in >>>> /usr/lib/libpthread-2.24.so <http://libpthread-2.24.so> >>>> <http://libpthread-2.24.so>) >>>> | | ->45.07% (484,867,174B) 0xCFCA7DD: clone (in >>>> /usr/lib/libc-2.24.so <http://libc-2.24.so> >>>> <http://libc-2.24.so>) >>>> | | >>>> | ->01.56% (16,789,504B) in 6 places, all below massif's >>>> threshold >>>> (1.00%) >>>> | >>>> ->22.70% (244,179,072B) 0x9C9807: BitMapZone::init(long, >>>> long, bool) >>>> (in /usr/bin/ceph-osd) >>>> | ->22.70% (244,179,072B) 0x9CACED: >>>> BitMapAreaLeaf::init(long, long, >>>> bool) (in /usr/bin/ceph-osd) >>>> | ->22.70% (244,179,072B) 0x9CAE88: >>>> BitMapAreaLeaf::BitMapAreaLeaf(long, long, bool) (in >>>> /usr/bin/ceph-osd) >>>> | ->22.67% (243,924,992B) 0x9CAF79: >>>> BitMapAreaIN::init(long, long, >>>> bool) (in /usr/bin/ceph-osd) >>>> | | ->12.46% (134,086,656B) 0x9CAFBE: >>>> BitMapAreaIN::init(long, >>>> long, bool) (in /usr/bin/ceph-osd) >>>> | | | ->12.46% (134,086,656B) 0x9CB237: >>>> BitAllocator::init_check(long, long, bmap_alloc_mode, bool, >>>> bool) (in >>>> /usr/bin/ceph-osd) >>>> | | | ->12.46% (134,086,656B) 0x9CB431: >>>> BitAllocator::BitAllocator(long, long, bmap_alloc_mode, >>>> bool) (in >>>> /usr/bin/ceph-osd) >>>> | | | ->12.46% (134,086,656B) 0x9C5C32: >>>> BitMapAllocator::BitMapAllocator(long, long) (in >>>> /usr/bin/ceph-osd) >>>> | | | ->12.46% (134,086,656B) 0x968FF1: >>>> Allocator::create(std::__cxx11::basic_string<char, >>>> std::char_traits<char>, std::allocator<char> >, long, long) >>>> (in >>>> /usr/bin/ceph-osd) >>>> | | | ->12.46% (134,086,656B) 0x87F65C: >>>> BlueStore::_open_alloc() (in /usr/bin/ceph-osd) >>>> | | | ->12.46% (134,086,656B) 0x8D8CDD: >>>> BlueStore::mount() (in /usr/bin/ceph-osd) >>>> | | | ->12.46% (134,086,656B) 0x4C15EA: >>>> OSD::init() >>>> (in /usr/bin/ceph-osd) >>>> | | | ->12.46% (134,086,656B) 0x40854C: >>>> main (in >>>> /usr/bin/ceph-osd) >>>> | | | >>>> | | ->10.21% (109,838,336B) 0x9CB00B: >>>> BitMapAreaIN::init(long, >>>> long, bool) (in /usr/bin/ceph-osd) >>>> | | ->10.21% (109,838,336B) 0x9CB237: >>>> BitAllocator::init_check(long, long, bmap_alloc_mode, bool, >>>> bool) (in >>>> /usr/bin/ceph-osd) >>>> | | ->10.21% (109,838,336B) 0x9CB431: >>>> BitAllocator::BitAllocator(long, long, bmap_alloc_mode, >>>> bool) (in >>>> /usr/bin/ceph-osd) >>>> | | ->10.21% (109,838,336B) 0x9C5C32: >>>> BitMapAllocator::BitMapAllocator(long, long) (in >>>> /usr/bin/ceph-osd) >>>> | | ->10.21% (109,838,336B) 0x968FF1: >>>> Allocator::create(std::__cxx11::basic_string<char, >>>> std::char_traits<char>, std::allocator<char> >, long, long) >>>> (in >>>> /usr/bin/ceph-osd) >>>> | | ->10.21% (109,838,336B) 0x87F65C: >>>> BlueStore::_open_alloc() (in /usr/bin/ceph-osd) >>>> | | ->10.21% (109,838,336B) 0x8D8CDD: >>>> BlueStore::mount() (in /usr/bin/ceph-osd) >>>> | | ->10.21% (109,838,336B) 0x4C15EA: >>>> OSD::init() >>>> (in /usr/bin/ceph-osd) >>>> | | ->10.21% (109,838,336B) 0x40854C: >>>> main (in >>>> /usr/bin/ceph-osd) >>>> | | >>>> | ->00.02% (254,080B) in 1+ places, all below ms_print's >>>> threshold (01.00%) >>>> | >>>> ->12.77% (137,350,728B) 0x9CACD8: BitMapAreaLeaf::init(long, >>>> long, >>>> bool) (in /usr/bin/ceph-osd) >>>> | ->12.77% (137,350,728B) 0x9CAE88: >>>> BitMapAreaLeaf::BitMapAreaLeaf(long, long, bool) (in >>>> /usr/bin/ceph-osd) >>>> | ->12.75% (137,207,808B) 0x9CAF79: >>>> BitMapAreaIN::init(long, long, >>>> bool) (in /usr/bin/ceph-osd) >>>> | | ->07.01% (75,423,744B) 0x9CAFBE: >>>> BitMapAreaIN::init(long, long, >>>> bool) (in /usr/bin/ceph-osd) >>>> | | | ->07.01% (75,423,744B) 0x9CB237: >>>> BitAllocator::init_check(long, long, bmap_alloc_mode, bool, >>>> bool) (in >>>> /usr/bin/ceph-osd) >>>> | | | ->07.01% (75,423,744B) 0x9CB431: >>>> BitAllocator::BitAllocator(long, long, bmap_alloc_mode, >>>> bool) (in >>>> /usr/bin/ceph-osd) >>>> | | | ->07.01% (75,423,744B) 0x9C5C32: >>>> BitMapAllocator::BitMapAllocator(long, long) (in >>>> /usr/bin/ceph-osd) >>>> | | | ->07.01% (75,423,744B) 0x968FF1: >>>> Allocator::create(std::__cxx11::basic_string<char, >>>> std::char_traits<char>, std::allocator<char> >, long, long) >>>> (in >>>> /usr/bin/ceph-osd) >>>> | | | ->07.01% (75,423,744B) 0x87F65C: >>>> BlueStore::_open_alloc() (in /usr/bin/ceph-osd) >>>> | | | ->07.01% (75,423,744B) 0x8D8CDD: >>>> BlueStore::mount() >>>> (in /usr/bin/ceph-osd) >>>> | | | ->07.01% (75,423,744B) 0x4C15EA: >>>> OSD::init() (in >>>> /usr/bin/ceph-osd) >>>> | | | ->07.01% (75,423,744B) 0x40854C: main >>>> (in >>>> /usr/bin/ceph-osd) >>>> >>> _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |