On Thu, Sep 19, 2013 at 8:12 PM, Joao Eduardo Luis <joao.luis@xxxxxxxxxxx> wrote: > On 09/19/2013 04:46 PM, Andrey Korolyov wrote: >> >> On Thu, Sep 19, 2013 at 1:00 PM, Joao Eduardo Luis >> <joao.luis@xxxxxxxxxxx> wrote: >>> >>> On 09/18/2013 11:25 PM, Andrey Korolyov wrote: >>>> >>>> >>>> Hello, >>>> >>>> Just restarted one of my mons after a month of uptime, memory commit >>>> raised ten times high than before: >>>> >>>> 13206 root 10 -10 12.8g 8.8g 107m S 65 14.0 0:53.97 ceph-mon >>>> >>>> normal one looks like >>>> >>>> 30092 root 10 -10 4411m 790m 46m S 1 1.2 1260:28 >>>> ceph-mon >>> >>> >>> >>> Try running 'ceph heap stats', followed by 'ceph heap release', and then >>> recheck the memory consumption for the monitor. >> >> >> It had shrinked to 350M RSS over night, so seems I need to restart >> this mon again or try with other one to reproduce the problem over >> next night. This monitor was a leader so I may check against other >> ones and see their peak consumption. > > > Was that monitor attempting to join the quorum? No, it had joined a long before. As we discussed in IRC, I restarted non-leader mon and there is some stat from freshly started mon process (which is joined quorum two minutes age) : ceph heap stats --keyfile admin -m 10.5.0.17:6789 mon.2tcmalloc heap stats:------------------------------------------------ MALLOC: 26256488 ( 25.0 MiB) Bytes in use by application MALLOC: + 11240284160 (10719.6 MiB) Bytes in page heap freelist MALLOC: + 3184848 ( 3.0 MiB) Bytes in central cache freelist MALLOC: + 8974848 ( 8.6 MiB) Bytes in transfer cache freelist MALLOC: + 15560904 ( 14.8 MiB) Bytes in thread cache freelists MALLOC: + 22114456 ( 21.1 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 11316375704 (10792.1 MiB) Actual memory used (physical + swap) MALLOC: + 90226688 ( 86.0 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 11406602392 (10878.2 MiB) Virtual address space used MALLOC: MALLOC: 4140 Spans in use MALLOC: 14 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). and after calling release: # ceph heap release --keyfile admin -m 10.5.0.17:6789 mon.2 releasing free RAM back to system. # ceph heap stats --keyfile admin -m 10.5.0.17:6789 mon.2tcmalloc heap stats:------------------------------------------------ MALLOC: 38925536 ( 37.1 MiB) Bytes in use by application MALLOC: + 13508608 ( 12.9 MiB) Bytes in page heap freelist MALLOC: + 2992112 ( 2.9 MiB) Bytes in central cache freelist MALLOC: + 12092416 ( 11.5 MiB) Bytes in transfer cache freelist MALLOC: + 17547056 ( 16.7 MiB) Bytes in thread cache freelists MALLOC: + 22114456 ( 21.1 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 107180184 ( 102.2 MiB) Actual memory used (physical + swap) MALLOC: + 11299422208 (10776.0 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 11406602392 (10878.2 MiB) Virtual address space used MALLOC: MALLOC: 4678 Spans in use MALLOC: 14 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the > > > -Joao > > > -- > Joao Eduardo Luis > Software Engineer | http://inktank.com | http://ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |