Okay, this might just be as simple as us creating a new root inode without deallocating the old one (MDS::open_root_inode, called from MDS::boot_start). Can you create a ticket? -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Mon, Jun 23, 2014 at 3:31 PM, Milosz Tanski <milosz@xxxxxxxxx> wrote: > standby-replay > > On Mon, Jun 23, 2014 at 3:27 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: >> Ah, excellent. What standby modes are you using? >> -Greg >> Software Engineer #42 @ http://inktank.com | http://ceph.com >> >> >> On Mon, Jun 23, 2014 at 2:54 PM, Milosz Tanski <milosz@xxxxxxxxx> wrote: >>> I've spent more time looking at this over the long time frame (since my last >>> email in April) and I think I'm closer to understanding to what's going on >>> here. I believe I was wrong in my original assumption that this is caused by >>> tcmalloc since I tried this without tcmalloc (using glibc) and I was still >>> exhibiting behavior. >>> >>> Having said that I think came onto a suggestion what might be wrong. When >>> doing a version upgrade my MDS server primary / standby have switched... and >>> now the other mds sever that was never running into MDS OOM scenarios has >>> started going it and the one that was having the issue stopped. I ended up >>> swapping the standby a couple times and it looks like it's the standby code >>> that's causing this leak. >>> >>> TL;DR Standby is the one the leak... not sure what it is, but the primary >>> doesn't exhibit this behavior. >>> >>> Best >>> - Milosz >>> >>> >>> On Mon, Apr 14, 2014 at 3:11 PM, Milosz Tanski <milosz@xxxxxxxxx> wrote: >>>> >>>> Sorry for not including the last on last email. It was an accident. >>>> >>>> On Fri, Apr 11, 2014 at 6:23 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: >>>> > On Fri, Apr 11, 2014 at 11:07 AM, Milosz Tanski <milosz@xxxxxxxxx> >>>> > wrote: >>>> >> On Fri, Apr 11, 2014 at 1:07 PM, Gregory Farnum <greg@xxxxxxxxxxx> >>>> >> wrote: >>>> >>> On Fri, Apr 11, 2014 at 8:59 AM, Milosz Tanski <milosz@xxxxxxxxx> >>>> >>> wrote: >>>> >>>> I'd like to restart this debate about tcmalloc slow leaks in MDS. >>>> >>>> This >>>> >>>> time around I have some charts. Looking at OSDs and MONs, it doesn't >>>> >>>> seam to affect those (as much). >>>> >>>> >>>> >>>> Here's the chart: http://i.imgur.com/xMCINAD.png The first two humps >>>> >>>> are the latest stable MDS version with tcmalloc till MDS gets killed >>>> >>>> by the OOM killer. The last restart MDS build of the same git tag >>>> >>>> without tcmalloc linked into it. >>>> >>> >>>> >>> That's interesting, but your graph cuts off before we can really see >>>> >>> the long-term behavior of the no-tcmalloc case. :) What's the >>>> >>> longer-term pattern look like? >>>> >> >>>> >> I'm only about two weeks into running without the allocator. I'm going >>>> >> to continue running it and report back in two weeks and a month. Sadly >>>> >> it takes a long time to test / reproduce the issue. >>>> > >>>> > Hmm, that makes it sound to me like it's not a tcmalloc issue, but >>>> > something changing in MDS state (a new workload that loads too much >>>> > into memory or something). >>>> >>>> 13 days into last startup so far and the needle hasn't move on memory >>>> usage (stable since 3 days in). Previously it took 20 days (twice in a >>>> row) to get to OOM. But by now it would have grown much larger. The >>>> workload hasn't changed. >>>> >>>> > >>>> >>>> I know that older tcmalloc version have leaks when allocating larger >>>> >>>> blocks of memory: >>>> >>>> https://code.google.com/p/gperftools/issues/detail?id=368 So it's >>>> >>>> possible that there is some kind of allocation pattern in MDS that >>>> >>>> causes this behavior or exposes this tcmalloc bug. >>>> >>> >>>> >>> Hrm, we do use memory pools in the MDS that the OSD and monitor do >>>> >>> not, so that could be influencing things. >>>> >> >>>> >> The issue I linked to is caused generally by making large allocations. >>>> >> It's my understanding that prior to the fix was very bad >>>> >> fragmenetation with large allocations. >>>> >> >>>> >>> >>>> >>>> Last time I bought it up there was resistance to tossing tcmalloc, >>>> >>>> which is fine. What I'd like to see is not linking against tcmalloc >>>> >>>> on >>>> >>>> systems that are know to have a buggy tcmalloc (in this case ubuntu >>>> >>>> 12.04, older Debian systems). >>>> >>> >>>> >>> The issue is that back when we did the investigation and testing (on >>>> >>> older Debian systems) that made us switch to tcmalloc: >>>> >>> 1) Memory growth without tcmalloc on the OSDs and monitor was so bad >>>> >>> as to make them essentially unusable, >>>> >>> 2) the MDS also behaved better with it (though I don't remember how >>>> >>> much) >>>> >>> 3) tcmalloc supplies some really nice memory analysis tools that I'd >>>> >>> like to keep around. >>>> >>> >>>> >>> So we'd need to do something like find a different allocator that >>>> >>> works for all three processes, or link the OSD and monitor with it but >>>> >>> not the MDS *and* demonstrate that the default allocators in each of >>>> >>> our platforms work for the MDS without issue (or go down the rat's >>>> >>> nest of selecting allocator based on platform). Before we embark on >>>> >>> that I'd like to get more data about what's causing the memory growth. >>>> >>> Can you gather some heap dumps and stats? Have you tried just >>>> >>> instructing the MDS to release unused memory when it passes some >>>> >>> threshold? >>>> >> >>>> >> For another internal project we started off with tcmalloc and switched >>>> >> to jemalloc. We ran into the same kind of pattern with tcmalloc on >>>> >> ubuntu 12.04. >>>> >> >>>> >> Now in our case doing database equivalent of sorting 10s to low 100s >>>> >> of gigabytes in background process (maintenance jobs for compacting >>>> >> and dup removal) we did this in blocks of 0.25g using merge sort. >>>> >> After about a day of runtime (when a lot of these jobs ran) we would >>>> >> start running into OOM cases. I enabled the tcmalloc debugger (via >>>> >> flags) and it would log every 1gb allocated. Tcmalloc reported that >>>> >> the app was using low gigabytes of working memory during busy times >>>> >> and and going into the low 10s of megabytes at idle times. Yet despite >>>> >> those the memory consumed by the process was reaching 40 gigs. >>>> > >>>> > Did you try using the HeapRelease() command (or whatever it's called)? >>>> > A few users have reported that tcmalloc was broken in one way or >>>> > another on their platform (though usually on something like Gentoo >>>> > rather than Ubuntu Precise!) and that call has invariably dealt with >>>> > the issue. *shrug* >>>> >>>> For our use case I did end up playing with the various configuration >>>> knobs for TCMALLOC (via environmental variables.) None of them ended >>>> up helping (release rate, etc). We did not end up calling the tcmalloc >>>> functions directly (like HeapRelease) because we didn't want to have >>>> our app depend on tcmalloc. And, quite frankly I thought it was silly >>>> for us to jump through a lot of hoops in order to make the allocator >>>> not explode. >>>> >>>> > >>>> >> We considered building tcmalloc from source, but noticed that redis in >>>> >> ubuntu/debian jemalloc and switched to using it. In this case, yes I'm >>>> >> shilling for jemalloc because it solved similar issues with >>>> >> experienced. And after doing significant testing on performance to >>>> >> compare the two it was within margin of error. Recent version of >>>> >> jemalloc support can output heap profiling information in a format >>>> >> understood by pprof (the google perftools). >>>> > >>>> > Interesting. Next time we wrangle some time to look at these issues >>>> > I'll check jemalloc out. >>>> > -Greg >>>> > Software Engineer #42 @ http://inktank.com | http://ceph.com >>>> >>>> >>>> >>>> -- >>>> Milosz Tanski >>>> CTO >>>> 10 East 53rd Street, 37th floor >>>> New York, NY 10022 >>>> >>>> p: 646-253-9055 >>>> e: milosz@xxxxxxxxx >>> >>> >>> >>> >>> -- >>> Milosz Tanski >>> CTO >>> 16 East 34th Street, 15th floor >>> New York, NY 10016 >>> >>> p: 646-253-9055 >>> e: milosz@xxxxxxxxx > > > > -- > Milosz Tanski > CTO > 16 East 34th Street, 15th floor > New York, NY 10016 > > p: 646-253-9055 > e: milosz@xxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
