On Thu, Sep 19, 2013 at 1:00 PM, Joao Eduardo Luis <joao.luis@xxxxxxxxxxx> wrote: > On 09/18/2013 11:25 PM, Andrey Korolyov wrote: >> >> Hello, >> >> Just restarted one of my mons after a month of uptime, memory commit >> raised ten times high than before: >> >> 13206 root 10 -10 12.8g 8.8g 107m S 65 14.0 0:53.97 ceph-mon >> >> normal one looks like >> >> 30092 root 10 -10 4411m 790m 46m S 1 1.2 1260:28 ceph-mon > > > Try running 'ceph heap stats', followed by 'ceph heap release', and then > recheck the memory consumption for the monitor. It had shrinked to 350M RSS over night, so seems I need to restart this mon again or try with other one to reproduce the problem over next night. This monitor was a leader so I may check against other ones and see their peak consumption. > > > >> monstore has simular size about 15G per monitor so only one problem is >> very unusual and terrifying memory consumption. Also it is possible >> that remaining mons running 0.61.7 but binary was updated long ago so >> it`s hard to tell which version is running not doing dump and >> disrupting quorum for a little - anyway I should tame current one. > > > How big is the cluster? 15GB for the monitor store may not be that > surprising if you have a bunch of OSDs and they're not completely healthy, > as that will prevent the removal of old maps on the monitor side. ~8.5T commit and > 2.5M objects but cluster is completely healthy on the moment though recently I had run couple of recovery procedures and it may affect mondir data allocation on day-long intervals. > > This could also be an issue with store compaction. > > Also, you should check whether the monitors are running 0.61.7 and, if so, > update them to 0.61.8. You should be able to get that version using the > admin socket if you want to. Just checked, the same 0.61.8. > > > -Joao > > -- > Joao Eduardo Luis > Software Engineer | http://inktank.com | http://ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |