Hi, Recently I observed one of the mallinfo fields had a negative value. DUMP-START-TIME: 2017-06-09 10:59:43.747440 [mallinfo] mallinfo_arena=-1517670400 mallinfo_ordblks=8008214 mallinfo_smblks=0 mallinfo_hblks=1009 mallinfo_hblkhd=863453184 mallinfo_usmblks=0 mallinfo_fsmblks=0 mallinfo_uordblks=1473090528 mallinfo_fordblks=1304206368 mallinfo_keepcost=2232208 As seen above mallinfo_arena is negative. On probing further I came across posts that said mallinfo is not the ideal interface to get metadata about memory allocated by malloc [1]. Instead there were two alternatives - malloc_stats and malloc_info - suggested. With all the three information taken statedump looks as below: DUMP-START-TIME: 2017-07-03 06:56:35.991499 [malloc_stats] Arena 0: system bytes = 622592 in use bytes = 498320 Arena 1: system bytes = 135168 in use bytes = 2224 Arena 2: system bytes = 135168 in use bytes = 2400 Arena 3: system bytes = 135168 in use bytes = 2256 Arena 4: system bytes = 135168 in use bytes = 2256 Arena 5: system bytes = 135168 in use bytes = 2256 Arena 6: system bytes = 135168 in use bytes = 2256 Arena 7: system bytes = 135168 in use bytes = 2256 Arena 8: system bytes = 135168 in use bytes = 2256 Arena 9: system bytes = 704512 in use bytes = 668864 Total (incl. mmap): system bytes = 17334272 in use bytes = 16111168 max mmap regions = 7 max mmap bytes = 14925824 [malloc_info] <malloc version="1"> <heap nr="0"> <sizes> <size from="81" to="81" total="81" count="1"/> <unsorted from="0" to="273" total="418" count="2"/> </sizes> <total type="fast" count="0" size="0"/> <total type="rest" count="3" size="499"/> <system type="current" size="622592"/> <system type="max" size="622592"/> <aspace type="total" size="622592"/> <aspace type="mprotect" size="622592"/> </heap> <heap nr="1"> <sizes> <unsorted from="0" to="273" total="418" count="2"/> </sizes> <total type="fast" count="0" size="0"/> <total type="rest" count="2" size="418"/> <system type="current" size="135168"/> <system type="max" size="135168"/> <aspace type="total" size="135168"/> <aspace type="mprotect" size="135168"/> </heap> <heap nr="2"> <sizes> <unsorted from="0" to="273" total="418" count="2"/> </sizes> <total type="fast" count="0" size="0"/> <total type="rest" count="2" size="418"/> <system type="current" size="135168"/> <system type="max" size="135168"/> <aspace type="total" size="135168"/> <aspace type="mprotect" size="135168"/> </heap> <heap nr="3"> <sizes> <unsorted from="0" to="273" total="418" count="2"/> </sizes> <total type="fast" count="0" size="0"/> <total type="rest" count="2" size="418"/> <system type="current" size="135168"/> <system type="max" size="135168"/> <aspace type="total" size="135168"/> <aspace type="mprotect" size="135168"/> </heap> <heap nr="4"> <sizes> <unsorted from="0" to="273" total="418" count="2"/> </sizes> <total type="fast" count="0" size="0"/> <total type="rest" count="2" size="418"/> <system type="current" size="135168"/> <system type="max" size="135168"/> <aspace type="total" size="135168"/> <aspace type="mprotect" size="135168"/> </heap> <heap nr="5"> <sizes> <unsorted from="0" to="273" total="418" count="2"/> </sizes> <total type="fast" count="0" size="0"/> <total type="rest" count="2" size="418"/> <system type="current" size="135168"/> <system type="max" size="135168"/> <aspace type="total" size="135168"/> <aspace type="mprotect" size="135168"/> </heap> <heap nr="6"> <sizes> <unsorted from="0" to="273" total="418" count="2"/> </sizes> <total type="fast" count="0" size="0"/> <total type="rest" count="2" size="418"/> <system type="current" size="135168"/> <system type="max" size="135168"/> <aspace type="total" size="135168"/> <aspace type="mprotect" size="135168"/> </heap> <heap nr="7"> <sizes> <unsorted from="0" to="273" total="418" count="2"/> </sizes> <total type="fast" count="0" size="0"/> <total type="rest" count="2" size="418"/> <system type="current" size="135168"/> <system type="max" size="135168"/> <aspace type="total" size="135168"/> <aspace type="mprotect" size="135168"/> </heap> <heap nr="8"> <sizes> <unsorted from="0" to="273" total="418" count="2"/> </sizes> <total type="fast" count="0" size="0"/> <total type="rest" count="2" size="418"/> <system type="current" size="135168"/> <system type="max" size="135168"/> <aspace type="total" size="135168"/> <aspace type="mprotect" size="135168"/> </heap> <heap nr="9"> <sizes> <size from="33" to="33" total="33" count="1"/> <size from="49" to="49" total="637" count="13"/> <size from="65" to="65" total="65" count="1"/> <size from="97" to="97" total="97" count="1"/> <size from="113" to="113" total="226" count="2"/> <size from="177" to="177" total="177" count="1"/> <size from="241" to="241" total="241" count="1"/> <size from="497" to="497" total="497" count="1"/> <unsorted from="0" to="2993" total="3556" count="4"/> </sizes> <total type="fast" count="0" size="0"/> <total type="rest" count="25" size="5529"/> <system type="current" size="704512"/> <system type="max" size="704512"/> <aspace type="total" size="704512"/> <aspace type="mprotect" size="704512"/> </heap> <total type="fast" count="0" size="0"/> <total type="rest" count="44" size="9372"/> <system type="current" size="2408448"/> <system type="max" size="2408448"/> <aspace type="total" size="2408448"/> <aspace type="mprotect" size="2408448"/> </malloc> [mallinfo] mallinfo_arena=622592 mallinfo_ordblks=4 mallinfo_smblks=0 mallinfo_hblks=7 mallinfo_hblkhd=14925824 mallinfo_usmblks=0 mallinfo_fsmblks=0 mallinfo_uordblks=498320 mallinfo_fordblks=124272 mallinfo_keepcost=123776 The questions now are * what among the above gives accurate and simple explanation about memory consumption of glusterfs? * Should we deprecate mallinfo and just retain malloc_stats and malloc_info outputs? IOW, which among these need to be retained in statedump? Since I've limited understanding of glibc memory allocator, I am reaching out to the wider community for feedback. [1] http://udrepper.livejournal.com/20948.html regards, Raghavendra _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-devel
