On Tue, Dec 17, 2013 at 4:15 PM, Milosz Tanski <milosz@xxxxxxxxx> wrote: > I wanted to bring up an issue with Ceph's use of tcmalloc. I know that > in Ubuntu (12.04) Ceph uses the distro version of tcmalloc which > older. > > I've personally ran into issues with tcmalloc for our application > where the amount of private space for our application (analytics > database) is listed in 10s of gigs but the heap profiler only reports > a couple hundreds megs of use. > > It seams like tcmalloc while fast does a poor job with fragmentation. > In most cases I rather leave more memory to the OS to cache disk data > leave it in case the app needs to use it. I've observed issues with > the MDS when it's grown to a few gigs and it doesn't grow down even > after I've removed a 2/3 of the tree. > > It's possible that this is fixed in newer versions of tcmalloc > (https://code.google.com/p/gperftools/issues/detail?id=368). I was > curious if anyone else audited this / ran into similar issues and > bring it up. For us, we moved to jemalloc which does a fine job > returning the memory back to the OS. We've seen scattered instances of this happening on systems run by external users (generally on Gentoo or maybe a Fedora build), but have never experienced it anywhere that we could reproduce. Helpfully, when that's the case it's always been easy to resolve by running the "heap release" command (or whatever it actually is; search the docs) to make tcmalloc give up its free RAM. I haven't evaluated jemalloc, but in all the systems we've run it on tcmalloc behaves perfectly under our workloads and provides some helpful features (heap analysis, etc); the default system malloc on the other hand actually did grow without bound. I'm not sure your MDS memory use test is actually valid — removing inodes from the tree leaves them in cache until they've been cleaned up out of RADOS, and we have in the past had bugs related to things not being trimmed. You'd really want to look at the tcmalloc stats on the MDS to see if tcmalloc is misbehaving or not. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
