Re: Bluestore + erasure coding memory usage

Mark Nelson <mnelson@xxxxxxxxxx> · Tue, 8 Nov 2016 11:00:31 -0600

It should be running much slower through valgrind so probably won't 
accumulate very quickly.  That was the problem with the earlier trace, 
there wasn't enough memory used yet to really get us out of the weeds. 
If it's still accumulating quickly, try to wait until the OSD is up to 
4+GB RSS if you can.  I usually kill the valgrind/osd process with 
SIGTERM to make sure the output is preserved.  Not sure what will happen 
with OOM killer as I haven't let it get that far before killing.

Mark

On 11/08/2016 10:37 AM, bobobo1618@xxxxxxxxx wrote:
Unfortunately I don't think overnight is possible. The OOM will kill it
in hours, if not minutes. Will the output be preserved/usable if the
process is uncleanly terminated?

On 8 Nov 2016 08:33, "Mark Nelson" <mnelson@xxxxxxxxxx
<mailto:mnelson@xxxxxxxxxx>> wrote:

    Heya,

    Sorry got distracted with other stuff yesterday.  Any chance you
    could run this for longer?  It's tough to tell what's going on from
    this run unfortunately.  Maybe overnight if possible.

    Thanks!
    Mark

    On 11/08/2016 01:10 AM, bobobo1618@xxxxxxxxx
    <mailto:bobobo1618@xxxxxxxxx> wrote:

        Just bumping this and CCing directly since I foolishly broke the
        threading on my reply.

        On 4 Nov. 2016 8:40 pm, "bobobo1618@xxxxxxxxx
        <mailto:bobobo1618@xxxxxxxxx>
        <mailto:bobobo1618@xxxxxxxxx <mailto:bobobo1618@xxxxxxxxx>>"
        <bobobo1618@xxxxxxxxx <mailto:bobobo1618@xxxxxxxxx>
        <mailto:bobobo1618@xxxxxxxxx <mailto:bobobo1618@xxxxxxxxx>>> wrote:

            > Then you can view the output data with ms_print or with
            massif-visualizer.  This may help narrow down where in the
        code we
            are using the memory.

            Done! I've dumped the output from ms_print here:
        http://ix.io/1CrS

            It seems most of the memory comes from here:

            92.78% (998,248,799B) (heap allocation functions)
        malloc/new/new[],
            --alloc-fns, etc.
            ->46.63% (501,656,678B) 0xD38936:
            ceph::buffer::create_aligned(unsigned int, unsigned int) (in
            /usr/bin/ceph-osd)
            | ->45.07% (484,867,174B) 0xDAFED9:
        AsyncConnection::process() (in
            /usr/bin/ceph-osd)
            | | ->45.07% (484,867,174B) 0xC410EB:
        EventCenter::process_events(int)
            (in /usr/bin/ceph-osd)
            | |   ->45.07% (484,867,174B) 0xC45210: ??? (in
        /usr/bin/ceph-osd)
            | |     ->45.07% (484,867,174B) 0xC6FA31D:
            execute_native_thread_routine (thread.cc:83)
            | |       ->45.07% (484,867,174B) 0xBE06452: start_thread (in
            /usr/lib/libpthread-2.24.so <http://libpthread-2.24.so>
        <http://libpthread-2.24.so>)
            | |         ->45.07% (484,867,174B) 0xCFCA7DD: clone (in
            /usr/lib/libc-2.24.so <http://libc-2.24.so>
        <http://libc-2.24.so>)
            | |
            | ->01.56% (16,789,504B) in 6 places, all below massif's
        threshold
            (1.00%)
            |
            ->22.70% (244,179,072B) 0x9C9807: BitMapZone::init(long,
        long, bool)
            (in /usr/bin/ceph-osd)
            | ->22.70% (244,179,072B) 0x9CACED:
        BitMapAreaLeaf::init(long, long,
            bool) (in /usr/bin/ceph-osd)
            |   ->22.70% (244,179,072B) 0x9CAE88:
            BitMapAreaLeaf::BitMapAreaLeaf(long, long, bool) (in
            /usr/bin/ceph-osd)
            |     ->22.67% (243,924,992B) 0x9CAF79:
        BitMapAreaIN::init(long, long,
            bool) (in /usr/bin/ceph-osd)
            |     | ->12.46% (134,086,656B) 0x9CAFBE:
        BitMapAreaIN::init(long,
            long, bool) (in /usr/bin/ceph-osd)
            |     | | ->12.46% (134,086,656B) 0x9CB237:
            BitAllocator::init_check(long, long, bmap_alloc_mode, bool,
        bool) (in
            /usr/bin/ceph-osd)
            |     | |   ->12.46% (134,086,656B) 0x9CB431:
            BitAllocator::BitAllocator(long, long, bmap_alloc_mode,
        bool) (in
            /usr/bin/ceph-osd)
            |     | |     ->12.46% (134,086,656B) 0x9C5C32:
            BitMapAllocator::BitMapAllocator(long, long) (in
        /usr/bin/ceph-osd)
            |     | |       ->12.46% (134,086,656B) 0x968FF1:
            Allocator::create(std::__cxx11::basic_string<char,
            std::char_traits<char>, std::allocator<char> >, long, long) (in
            /usr/bin/ceph-osd)
            |     | |         ->12.46% (134,086,656B) 0x87F65C:
            BlueStore::_open_alloc() (in /usr/bin/ceph-osd)
            |     | |           ->12.46% (134,086,656B) 0x8D8CDD:
            BlueStore::mount() (in /usr/bin/ceph-osd)
            |     | |             ->12.46% (134,086,656B) 0x4C15EA:
        OSD::init()
            (in /usr/bin/ceph-osd)
            |     | |               ->12.46% (134,086,656B) 0x40854C:
        main (in
            /usr/bin/ceph-osd)
            |     | |
            |     | ->10.21% (109,838,336B) 0x9CB00B:
        BitMapAreaIN::init(long,
            long, bool) (in /usr/bin/ceph-osd)
            |     |   ->10.21% (109,838,336B) 0x9CB237:
            BitAllocator::init_check(long, long, bmap_alloc_mode, bool,
        bool) (in
            /usr/bin/ceph-osd)
            |     |     ->10.21% (109,838,336B) 0x9CB431:
            BitAllocator::BitAllocator(long, long, bmap_alloc_mode,
        bool) (in
            /usr/bin/ceph-osd)
            |     |       ->10.21% (109,838,336B) 0x9C5C32:
            BitMapAllocator::BitMapAllocator(long, long) (in
        /usr/bin/ceph-osd)
            |     |         ->10.21% (109,838,336B) 0x968FF1:
            Allocator::create(std::__cxx11::basic_string<char,
            std::char_traits<char>, std::allocator<char> >, long, long) (in
            /usr/bin/ceph-osd)
            |     |           ->10.21% (109,838,336B) 0x87F65C:
            BlueStore::_open_alloc() (in /usr/bin/ceph-osd)
            |     |             ->10.21% (109,838,336B) 0x8D8CDD:
            BlueStore::mount() (in /usr/bin/ceph-osd)
            |     |               ->10.21% (109,838,336B) 0x4C15EA:
        OSD::init()
            (in /usr/bin/ceph-osd)
            |     |                 ->10.21% (109,838,336B) 0x40854C:
        main (in
            /usr/bin/ceph-osd)
            |     |
            |     ->00.02% (254,080B) in 1+ places, all below ms_print's
            threshold (01.00%)
            |
            ->12.77% (137,350,728B) 0x9CACD8: BitMapAreaLeaf::init(long,
        long,
            bool) (in /usr/bin/ceph-osd)
            | ->12.77% (137,350,728B) 0x9CAE88:
            BitMapAreaLeaf::BitMapAreaLeaf(long, long, bool) (in
            /usr/bin/ceph-osd)
            |   ->12.75% (137,207,808B) 0x9CAF79:
        BitMapAreaIN::init(long, long,
            bool) (in /usr/bin/ceph-osd)
            |   | ->07.01% (75,423,744B) 0x9CAFBE:
        BitMapAreaIN::init(long, long,
            bool) (in /usr/bin/ceph-osd)
            |   | | ->07.01% (75,423,744B) 0x9CB237:
            BitAllocator::init_check(long, long, bmap_alloc_mode, bool,
        bool) (in
            /usr/bin/ceph-osd)
            |   | |   ->07.01% (75,423,744B) 0x9CB431:
            BitAllocator::BitAllocator(long, long, bmap_alloc_mode,
        bool) (in
            /usr/bin/ceph-osd)
            |   | |     ->07.01% (75,423,744B) 0x9C5C32:
            BitMapAllocator::BitMapAllocator(long, long) (in
        /usr/bin/ceph-osd)
            |   | |       ->07.01% (75,423,744B) 0x968FF1:
            Allocator::create(std::__cxx11::basic_string<char,
            std::char_traits<char>, std::allocator<char> >, long, long) (in
            /usr/bin/ceph-osd)
            |   | |         ->07.01% (75,423,744B) 0x87F65C:
            BlueStore::_open_alloc() (in /usr/bin/ceph-osd)
            |   | |           ->07.01% (75,423,744B) 0x8D8CDD:
        BlueStore::mount()
            (in /usr/bin/ceph-osd)
            |   | |             ->07.01% (75,423,744B) 0x4C15EA:
        OSD::init() (in
            /usr/bin/ceph-osd)
            |   | |               ->07.01% (75,423,744B) 0x40854C: main (in
            /usr/bin/ceph-osd)

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com