standby-replay On Mon, Jun 23, 2014 at 3:27 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: > Ah, excellent. What standby modes are you using? > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com > > > On Mon, Jun 23, 2014 at 2:54 PM, Milosz Tanski <milosz@xxxxxxxxx> wrote: >> I've spent more time looking at this over the long time frame (since my last >> email in April) and I think I'm closer to understanding to what's going on >> here. I believe I was wrong in my original assumption that this is caused by >> tcmalloc since I tried this without tcmalloc (using glibc) and I was still >> exhibiting behavior. >> >> Having said that I think came onto a suggestion what might be wrong. When >> doing a version upgrade my MDS server primary / standby have switched... and >> now the other mds sever that was never running into MDS OOM scenarios has >> started going it and the one that was having the issue stopped. I ended up >> swapping the standby a couple times and it looks like it's the standby code >> that's causing this leak. >> >> TL;DR Standby is the one the leak... not sure what it is, but the primary >> doesn't exhibit this behavior. >> >> Best >> - Milosz >> >> >> On Mon, Apr 14, 2014 at 3:11 PM, Milosz Tanski <milosz@xxxxxxxxx> wrote: >>> >>> Sorry for not including the last on last email. It was an accident. >>> >>> On Fri, Apr 11, 2014 at 6:23 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: >>> > On Fri, Apr 11, 2014 at 11:07 AM, Milosz Tanski <milosz@xxxxxxxxx> >>> > wrote: >>> >> On Fri, Apr 11, 2014 at 1:07 PM, Gregory Farnum <greg@xxxxxxxxxxx> >>> >> wrote: >>> >>> On Fri, Apr 11, 2014 at 8:59 AM, Milosz Tanski <milosz@xxxxxxxxx> >>> >>> wrote: >>> >>>> I'd like to restart this debate about tcmalloc slow leaks in MDS. >>> >>>> This >>> >>>> time around I have some charts. Looking at OSDs and MONs, it doesn't >>> >>>> seam to affect those (as much). >>> >>>> >>> >>>> Here's the chart: http://i.imgur.com/xMCINAD.png The first two humps >>> >>>> are the latest stable MDS version with tcmalloc till MDS gets killed >>> >>>> by the OOM killer. The last restart MDS build of the same git tag >>> >>>> without tcmalloc linked into it. >>> >>> >>> >>> That's interesting, but your graph cuts off before we can really see >>> >>> the long-term behavior of the no-tcmalloc case. :) What's the >>> >>> longer-term pattern look like? >>> >> >>> >> I'm only about two weeks into running without the allocator. I'm going >>> >> to continue running it and report back in two weeks and a month. Sadly >>> >> it takes a long time to test / reproduce the issue. >>> > >>> > Hmm, that makes it sound to me like it's not a tcmalloc issue, but >>> > something changing in MDS state (a new workload that loads too much >>> > into memory or something). >>> >>> 13 days into last startup so far and the needle hasn't move on memory >>> usage (stable since 3 days in). Previously it took 20 days (twice in a >>> row) to get to OOM. But by now it would have grown much larger. The >>> workload hasn't changed. >>> >>> > >>> >>>> I know that older tcmalloc version have leaks when allocating larger >>> >>>> blocks of memory: >>> >>>> https://code.google.com/p/gperftools/issues/detail?id=368 So it's >>> >>>> possible that there is some kind of allocation pattern in MDS that >>> >>>> causes this behavior or exposes this tcmalloc bug. >>> >>> >>> >>> Hrm, we do use memory pools in the MDS that the OSD and monitor do >>> >>> not, so that could be influencing things. >>> >> >>> >> The issue I linked to is caused generally by making large allocations. >>> >> It's my understanding that prior to the fix was very bad >>> >> fragmenetation with large allocations. >>> >> >>> >>> >>> >>>> Last time I bought it up there was resistance to tossing tcmalloc, >>> >>>> which is fine. What I'd like to see is not linking against tcmalloc >>> >>>> on >>> >>>> systems that are know to have a buggy tcmalloc (in this case ubuntu >>> >>>> 12.04, older Debian systems). >>> >>> >>> >>> The issue is that back when we did the investigation and testing (on >>> >>> older Debian systems) that made us switch to tcmalloc: >>> >>> 1) Memory growth without tcmalloc on the OSDs and monitor was so bad >>> >>> as to make them essentially unusable, >>> >>> 2) the MDS also behaved better with it (though I don't remember how >>> >>> much) >>> >>> 3) tcmalloc supplies some really nice memory analysis tools that I'd >>> >>> like to keep around. >>> >>> >>> >>> So we'd need to do something like find a different allocator that >>> >>> works for all three processes, or link the OSD and monitor with it but >>> >>> not the MDS *and* demonstrate that the default allocators in each of >>> >>> our platforms work for the MDS without issue (or go down the rat's >>> >>> nest of selecting allocator based on platform). Before we embark on >>> >>> that I'd like to get more data about what's causing the memory growth. >>> >>> Can you gather some heap dumps and stats? Have you tried just >>> >>> instructing the MDS to release unused memory when it passes some >>> >>> threshold? >>> >> >>> >> For another internal project we started off with tcmalloc and switched >>> >> to jemalloc. We ran into the same kind of pattern with tcmalloc on >>> >> ubuntu 12.04. >>> >> >>> >> Now in our case doing database equivalent of sorting 10s to low 100s >>> >> of gigabytes in background process (maintenance jobs for compacting >>> >> and dup removal) we did this in blocks of 0.25g using merge sort. >>> >> After about a day of runtime (when a lot of these jobs ran) we would >>> >> start running into OOM cases. I enabled the tcmalloc debugger (via >>> >> flags) and it would log every 1gb allocated. Tcmalloc reported that >>> >> the app was using low gigabytes of working memory during busy times >>> >> and and going into the low 10s of megabytes at idle times. Yet despite >>> >> those the memory consumed by the process was reaching 40 gigs. >>> > >>> > Did you try using the HeapRelease() command (or whatever it's called)? >>> > A few users have reported that tcmalloc was broken in one way or >>> > another on their platform (though usually on something like Gentoo >>> > rather than Ubuntu Precise!) and that call has invariably dealt with >>> > the issue. *shrug* >>> >>> For our use case I did end up playing with the various configuration >>> knobs for TCMALLOC (via environmental variables.) None of them ended >>> up helping (release rate, etc). We did not end up calling the tcmalloc >>> functions directly (like HeapRelease) because we didn't want to have >>> our app depend on tcmalloc. And, quite frankly I thought it was silly >>> for us to jump through a lot of hoops in order to make the allocator >>> not explode. >>> >>> > >>> >> We considered building tcmalloc from source, but noticed that redis in >>> >> ubuntu/debian jemalloc and switched to using it. In this case, yes I'm >>> >> shilling for jemalloc because it solved similar issues with >>> >> experienced. And after doing significant testing on performance to >>> >> compare the two it was within margin of error. Recent version of >>> >> jemalloc support can output heap profiling information in a format >>> >> understood by pprof (the google perftools). >>> > >>> > Interesting. Next time we wrangle some time to look at these issues >>> > I'll check jemalloc out. >>> > -Greg >>> > Software Engineer #42 @ http://inktank.com | http://ceph.com >>> >>> >>> >>> -- >>> Milosz Tanski >>> CTO >>> 10 East 53rd Street, 37th floor >>> New York, NY 10022 >>> >>> p: 646-253-9055 >>> e: milosz@xxxxxxxxx >> >> >> >> >> -- >> Milosz Tanski >> CTO >> 16 East 34th Street, 15th floor >> New York, NY 10016 >> >> p: 646-253-9055 >> e: milosz@xxxxxxxxx -- Milosz Tanski CTO 16 East 34th Street, 15th floor New York, NY 10016 p: 646-253-9055 e: milosz@xxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
