I don't think this is anything we've observed before. Normally when a Ceph node is using more memory than its peers it's a consequence of something in that node getting backed up. You might try looking at the perf counters via the admin socket and seeing if something about them is different between your ARM and AMD processors. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Tue, Nov 5, 2013 at 7:21 AM, Yu Changyuan <reivzy@xxxxxxxxx> wrote: > Finally, my tiny ceph cluster get 3 monitors, newly added mon.b and mon.c > both running on cubieboard2, which is cheap but still with enough cpu > power(dual-core arm A7 cpu, 1.2G) and memory(1G). > > But compare to mon.a which running on an amd64 cpu, both mon.b and mon.c > easily consume too much memory, so I want to know whether this is caused by > memory leak. Below is the output of 'ceph tell mon.a heap stats' and 'ceph > tell mon.c heap stats'(mon.c only start 12hr ago, while mon.a already > running for more than 10 days) > > mon.atcmalloc heap stats:------------------------------------------------ > MALLOC: 5480160 ( 5.2 MiB) Bytes in use by application > MALLOC: + 28065792 ( 26.8 MiB) Bytes in page heap freelist > MALLOC: + 15242312 ( 14.5 MiB) Bytes in central cache freelist > MALLOC: + 10116608 ( 9.6 MiB) Bytes in transfer cache freelist > MALLOC: + 10432216 ( 9.9 MiB) Bytes in thread cache freelists > MALLOC: + 1667224 ( 1.6 MiB) Bytes in malloc metadata > MALLOC: ------------ > MALLOC: = 71004312 ( 67.7 MiB) Actual memory used (physical + swap) > MALLOC: + 57540608 ( 54.9 MiB) Bytes released to OS (aka unmapped) > MALLOC: ------------ > MALLOC: = 128544920 ( 122.6 MiB) Virtual address space used > MALLOC: > MALLOC: 4655 Spans in use > MALLOC: 34 Thread heaps in use > MALLOC: 8192 Tcmalloc page size > ------------------------------------------------ > Call ReleaseFreeMemory() to release freelist memory to the OS (via > madvise()). > Bytes released to the > > > mon.ctcmalloc heap stats:------------------------------------------------ > MALLOC: 175861640 ( 167.7 MiB) Bytes in use by application > MALLOC: + 2220032 ( 2.1 MiB) Bytes in page heap freelist > MALLOC: + 1007560 ( 1.0 MiB) Bytes in central cache freelist > MALLOC: + 2871296 ( 2.7 MiB) Bytes in transfer cache freelist > MALLOC: + 4686000 ( 4.5 MiB) Bytes in thread cache freelists > MALLOC: + 2758880 ( 2.6 MiB) Bytes in malloc metadata > MALLOC: ------------ > MALLOC: = 189405408 ( 180.6 MiB) Actual memory used (physical + swap) > MALLOC: + 0 ( 0.0 MiB) Bytes released to OS (aka unmapped) > MALLOC: ------------ > MALLOC: = 189405408 ( 180.6 MiB) Virtual address space used > MALLOC: > MALLOC: 3445 Spans in use > MALLOC: 14 Thread heaps in use > MALLOC: 8192 Tcmalloc page size > ------------------------------------------------ > Call ReleaseFreeMemory() to release freelist memory to the OS (via > madvise()). > Bytes released to the > > The ceph versin is 0.67.4, compiled with tcmalloc enabled, > gcc(armv7a-hardfloat-linux-gnueabi-gcc) version 4.7.3 and I also try to dump > heap, but I can not find anything useful, below is a recent dump, output by > command "pprof --text /usr/bin/ceph-mon mon.c.profile.0021.heap". What extra > step should I take to make the dump more meaningful? > > Using local file /usr/bin/ceph-mon. > Using local file mon.c.profile.0021.heap. > Total: 149.3 MB > 146.2 97.9% 97.9% 146.2 97.9% 00000000b6a7ce7c > 1.4 0.9% 98.9% 1.4 0.9% std::basic_string::_Rep::_S_create > ??:0 > 1.4 0.9% 99.8% 1.4 0.9% 00000000002dd794 > 0.1 0.1% 99.9% 0.1 0.1% 00000000b6a81170 > 0.1 0.1% 99.9% 0.1 0.1% 00000000b6a80894 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7e2ac > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a81410 > 0.0 0.0% 100.0% 0.0 0.0% 0000000000367450 > 0.0 0.0% 100.0% 0.0 0.0% 00000000001d4474 > 0.0 0.0% 100.0% 0.0 0.0% 000000000028847c > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7e8d8 > 0.0 0.0% 100.0% 0.0 0.0% 000000000020c80c > 0.0 0.0% 100.0% 0.0 0.0% 000000000028bd20 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a63248 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a83478 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a806f0 > 0.0 0.0% 100.0% 0.0 0.0% 00000000002eb8b8 > 0.0 0.0% 100.0% 0.0 0.0% 000000000024efb4 > 0.0 0.0% 100.0% 0.0 0.0% 000000000027e550 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a77104 > 0.0 0.0% 100.0% 0.0 0.0% _dl_mcount ??:0 > 0.0 0.0% 100.0% 0.0 0.0% 00000000003673ec > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7a91c > 0.0 0.0% 100.0% 0.0 0.0% 0000000000295e44 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7ee38 > 0.0 0.0% 100.0% 0.0 0.0% 0000000000283948 > 0.0 0.0% 100.0% 0.0 0.0% 00000000002a53c4 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7665c > 0.0 0.0% 100.0% 0.0 0.0% 00000000002c4590 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7e88c > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a8456c > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a76ed4 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a842f0 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a72bd0 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a73cf8 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7100c > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7dec4 > 0.0 0.0% 100.0% 0.0 0.0% 000000000035e6e8 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a78f68 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7de9c > 0.0 0.0% 100.0% 0.0 0.0% 0000000000220528 > 0.0 0.0% 100.0% 0.0 0.0% 000000000035e7c0 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a6b2f8 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a80a04 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a62e7c > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a66f50 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a7e958 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a6cfb8 > 0.0 0.0% 100.0% 0.0 0.0% leveldb::DBImpl::MakeRoomForWrite > (inline) ??:0 > 0.0 0.0% 100.0% 0.0 0.0% 000000000020797c > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a69de0 > 0.0 0.0% 100.0% 0.0 0.0% 00000000001d0af0 > 0.0 0.0% 100.0% 0.0 0.0% 00000000001d0ebc > 0.0 0.0% 100.0% 0.0 0.0% 00000000002a0cd4 > 0.0 0.0% 100.0% 0.0 0.0% 000000000036909c > 0.0 0.0% 100.0% 0.0 0.0% 000000000040b02c > 0.0 0.0% 100.0% 0.0 0.0% 00000000001d0b68 > 0.0 0.0% 100.0% 0.0 0.0% 0000000000392fa0 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a64404 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a791b4 > 0.0 0.0% 100.0% 0.0 0.0% 00000000001d9824 > 0.0 0.0% 100.0% 0.0 0.0% 0000000000213928 > 0.0 0.0% 100.0% 0.0 0.0% 00000000002a0cb8 > 0.0 0.0% 100.0% 0.0 0.0% 00000000002a4fcc > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a725ac > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a66308 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a79068 > 0.0 0.0% 100.0% 0.0 0.0% 00000000000013b2 > 0.0 0.0% 100.0% 0.0 0.0% 000000000040b000 > 0.0 0.0% 100.0% 0.1 0.1% 00000000004d839b > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f29887 > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f3eb6b > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f6e1cb > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f6edab > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f873ab > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f8a26b > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f8b0cb > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f92dab > 0.0 0.0% 100.0% 0.0 0.0% 0000000000f9c96b > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fa24bf > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fadd8b > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fb06ab > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fb0d0b > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fb494b > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fbad6b > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fbb2cb > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fbea6b > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fed0eb > 0.0 0.0% 100.0% 0.0 0.0% 0000000000fed69b > 0.0 0.0% 100.0% 0.0 0.0% 000000000129920b > 0.0 0.0% 100.0% 0.0 0.0% 00000000014250eb > 0.0 0.0% 100.0% 0.0 0.0% 000000000166cfc5 > 0.0 0.0% 100.0% 0.1 0.1% 000000000166d711 > 0.0 0.0% 100.0% 0.0 0.0% 0000000003531d2b > 0.0 0.0% 100.0% 0.0 0.0% 000000000379adbb > 0.0 0.0% 100.0% 0.0 0.0% 0000000004e888fb > 0.0 0.0% 100.0% 0.0 0.0% 0000000004e894ab > 0.0 0.0% 100.0% 0.0 0.0% 0000000004e8951b > 0.0 0.0% 100.0% 0.0 0.0% 00000000060146d3 > 0.0 0.0% 100.0% 0.0 0.0% 000000000601482f > 0.0 0.0% 100.0% 0.0 0.0% 00000000060fcd2b > 0.0 0.0% 100.0% 0.0 0.0% 00000000060fd33b > 0.0 0.0% 100.0% 0.0 0.0% 00000000060fdfbb > 0.0 0.0% 100.0% 0.0 0.0% 000000000a820749 > 0.0 0.0% 100.0% 0.0 0.0% 000000000bfb1950 > 0.0 0.0% 100.0% 0.0 0.0% 00000000b6a43f23 > 0.0 0.0% 100.0% 0.0 0.0% __clone ??:0 > 0.0 0.0% 100.0% 0.0 0.0% leveldb::DBImpl::MakeRoomForWrite > ??:0 > 0.0 0.0% 100.0% 0.2 0.1% std::num_put::do_put@806e4 ??:0 > 0.0 0.0% 100.0% 0.4 0.2% std::num_put::do_put@80b44 ??:0 > 0.0 0.0% 100.0% 0.1 0.1% std::num_put::do_put@80e00 ??:0 > > > PS: there's a cubietruck > board(http://docs.cubieboard.org/products/start#cubietruck_cubieboard3) > released recently, which features a dual-core arm A7 cpu, 2G RAM, 1Gbit eth > port, and a sata 2.0 port, for $89, maybe suitable for cheap dedicate osd > server with single disk. > > -- > Best regards, > Changyuan > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |