Since jemalloc tries to create arena on different thread for lock contention problems, no matter when system is busy.
This causes increasing memory usage.
I think we probably need to think how to a little bit carefully make use of:
pthread_create()
tcache
redzone
Etc...
And I also think it's better to think of chunk size.
There would be trade-off between performance and cost.
I'm not sure how busy system was.
There might had been memory leak. Which means that there might have been blocks which were not freed after malloc.
Did you check?
> it seems jemalloc is consuming ~200MB more memory/osd during IO run...
If I've missed anything, just point out it to me.
I'm really really pick in terms of performance tuning.
Shinobu
----- Original Message -----
From: "Somnath Roy" <Somnath.Roy@xxxxxxxxxxx>
To: "Chad William Seys" <cwseys@xxxxxxxxxxxxxxxx>, "池信泽" <xmdxcxz@xxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx
Sent: Saturday, August 29, 2015 12:55:34 AM
Subject: Re: RAM usage only very slowly decreases after cluster recovery
Yeah, that means tcmalloc probably caching those as I suspected..
There are some discussion going on in that front, but, unfortunately we concluded to have tcmalloc as default and if somebody needs performance should move to jemalloc.
One of the reason is, it seems jemalloc is consuming ~200MB more memory/osd during IO run...
But, I think this is one of the serious issue of tcmalloc we need to consider as well..I posted this findings earlier in ceph-devl during my write path optimization investigation.
There are some settings in tcmalloc that should expedite this memory release faster though. I tried, but, didn't work. I didn't dig down further in that route though.
Mark,
Did you observe similar tcmalloc behavior in your recovery experiment for tcmalloc vs jemalloc?
Thanks & Regards
Somnath
-----Original Message-----
From: Chad William Seys [mailto:cwseys@xxxxxxxxxxxxxxxx]
Sent: Friday, August 28, 2015 7:58 AM
To: 池信泽
Cc: Somnath Roy; Haomai Wang; ceph-users@xxxxxxxxxxxxxx
Subject: Re: RAM usage only very slowly decreases after cluster recovery
Thanks! 'ceph tell osd.* heap release' seems to have worked! Guess I'll sprinkle it around my maintenance scripts.
Somnath Is there a plan to make jemalloc standard in Ceph in the future?
Thanks!
Chad.
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
