I've spent more time looking at this over the long time frame (since my last email in April) and I think I'm closer to understanding to what's going on here. I believe I was wrong in my original assumption that this is caused by tcmalloc since I tried this without tcmalloc (using glibc) and I was still exhibiting behavior. Having said that I think came onto a suggestion what might be wrong. When doing a version upgrade my MDS server primary / standby have switched... and now the other mds sever that was never running into MDS OOM scenarios has started going it and the one that was having the issue stopped. I ended up swapping the standby a couple times and it looks like it's the standby code that's causing this leak. TL;DR Standby is the one the leak... not sure what it is, but the primary doesn't exhibit this behavior. Best - Milosz On Mon, Apr 14, 2014 at 3:11 PM, Milosz Tanski <milosz@xxxxxxxxx> wrote: > Sorry for not including the last on last email. It was an accident. > > On Fri, Apr 11, 2014 at 6:23 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: >> On Fri, Apr 11, 2014 at 11:07 AM, Milosz Tanski <milosz@xxxxxxxxx> wrote: >>> On Fri, Apr 11, 2014 at 1:07 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: >>>> On Fri, Apr 11, 2014 at 8:59 AM, Milosz Tanski <milosz@xxxxxxxxx> wrote: >>>>> I'd like to restart this debate about tcmalloc slow leaks in MDS. This >>>>> time around I have some charts. Looking at OSDs and MONs, it doesn't >>>>> seam to affect those (as much). >>>>> >>>>> Here's the chart: http://i.imgur.com/xMCINAD.png The first two humps >>>>> are the latest stable MDS version with tcmalloc till MDS gets killed >>>>> by the OOM killer. The last restart MDS build of the same git tag >>>>> without tcmalloc linked into it. >>>> >>>> That's interesting, but your graph cuts off before we can really see >>>> the long-term behavior of the no-tcmalloc case. :) What's the >>>> longer-term pattern look like? >>> >>> I'm only about two weeks into running without the allocator. I'm going >>> to continue running it and report back in two weeks and a month. Sadly >>> it takes a long time to test / reproduce the issue. >> >> Hmm, that makes it sound to me like it's not a tcmalloc issue, but >> something changing in MDS state (a new workload that loads too much >> into memory or something). > > 13 days into last startup so far and the needle hasn't move on memory > usage (stable since 3 days in). Previously it took 20 days (twice in a > row) to get to OOM. But by now it would have grown much larger. The > workload hasn't changed. > >> >>>>> I know that older tcmalloc version have leaks when allocating larger >>>>> blocks of memory: >>>>> https://code.google.com/p/gperftools/issues/detail?id=368 So it's >>>>> possible that there is some kind of allocation pattern in MDS that >>>>> causes this behavior or exposes this tcmalloc bug. >>>> >>>> Hrm, we do use memory pools in the MDS that the OSD and monitor do >>>> not, so that could be influencing things. >>> >>> The issue I linked to is caused generally by making large allocations. >>> It's my understanding that prior to the fix was very bad >>> fragmenetation with large allocations. >>> >>>> >>>>> Last time I bought it up there was resistance to tossing tcmalloc, >>>>> which is fine. What I'd like to see is not linking against tcmalloc on >>>>> systems that are know to have a buggy tcmalloc (in this case ubuntu >>>>> 12.04, older Debian systems). >>>> >>>> The issue is that back when we did the investigation and testing (on >>>> older Debian systems) that made us switch to tcmalloc: >>>> 1) Memory growth without tcmalloc on the OSDs and monitor was so bad >>>> as to make them essentially unusable, >>>> 2) the MDS also behaved better with it (though I don't remember how much) >>>> 3) tcmalloc supplies some really nice memory analysis tools that I'd >>>> like to keep around. >>>> >>>> So we'd need to do something like find a different allocator that >>>> works for all three processes, or link the OSD and monitor with it but >>>> not the MDS *and* demonstrate that the default allocators in each of >>>> our platforms work for the MDS without issue (or go down the rat's >>>> nest of selecting allocator based on platform). Before we embark on >>>> that I'd like to get more data about what's causing the memory growth. >>>> Can you gather some heap dumps and stats? Have you tried just >>>> instructing the MDS to release unused memory when it passes some >>>> threshold? >>> >>> For another internal project we started off with tcmalloc and switched >>> to jemalloc. We ran into the same kind of pattern with tcmalloc on >>> ubuntu 12.04. >>> >>> Now in our case doing database equivalent of sorting 10s to low 100s >>> of gigabytes in background process (maintenance jobs for compacting >>> and dup removal) we did this in blocks of 0.25g using merge sort. >>> After about a day of runtime (when a lot of these jobs ran) we would >>> start running into OOM cases. I enabled the tcmalloc debugger (via >>> flags) and it would log every 1gb allocated. Tcmalloc reported that >>> the app was using low gigabytes of working memory during busy times >>> and and going into the low 10s of megabytes at idle times. Yet despite >>> those the memory consumed by the process was reaching 40 gigs. >> >> Did you try using the HeapRelease() command (or whatever it's called)? >> A few users have reported that tcmalloc was broken in one way or >> another on their platform (though usually on something like Gentoo >> rather than Ubuntu Precise!) and that call has invariably dealt with >> the issue. *shrug* > > For our use case I did end up playing with the various configuration > knobs for TCMALLOC (via environmental variables.) None of them ended > up helping (release rate, etc). We did not end up calling the tcmalloc > functions directly (like HeapRelease) because we didn't want to have > our app depend on tcmalloc. And, quite frankly I thought it was silly > for us to jump through a lot of hoops in order to make the allocator > not explode. > >> >>> We considered building tcmalloc from source, but noticed that redis in >>> ubuntu/debian jemalloc and switched to using it. In this case, yes I'm >>> shilling for jemalloc because it solved similar issues with >>> experienced. And after doing significant testing on performance to >>> compare the two it was within margin of error. Recent version of >>> jemalloc support can output heap profiling information in a format >>> understood by pprof (the google perftools). >> >> Interesting. Next time we wrangle some time to look at these issues >> I'll check jemalloc out. >> -Greg >> Software Engineer #42 @ http://inktank.com | http://ceph.com > > > > -- > Milosz Tanski > CTO > 10 East 53rd Street, 37th floor > New York, NY 10022 > > p: 646-253-9055 > e: milosz@xxxxxxxxx -- Milosz Tanski CTO 16 East 34th Street, 15th floor New York, NY 10016 p: 646-253-9055 e: milosz@xxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
