On Mon, 25 Feb 2013, S?bastien Han wrote: > Hi Sage, > > Sorry it's a production system, so I can't test it. > So at the end, you can't get anything out of the core dump? I saw a bunch of dup object anmes, which is what led us to the pg log theory. I can look a bit more carefully to confirm, but in the end it would be nice to see users scrubbing without leaking. This may be a bit moot because we want to allow trimming for other reasons, so those patches are being tested and working their way into master. We'll backport when things are solid. In the meantime, if someone has been able to reproduce this in a test environment, testing is obviously welcome :) sage > > -- > Regards, > S?bastien Han. > > > On Sat, Feb 23, 2013 at 1:44 AM, Sage Weil <sage@xxxxxxxxxxx> wrote: > > On Fri, 22 Feb 2013, S?bastien Han wrote: > >> Hi all, > >> > >> I finally got a core dump. > >> > >> I did it with a kill -SEGV on the OSD process. > >> > >> https://www.dropbox.com/s/ahv6hm0ipnak5rf/core-ceph-osd-11-0-0-20100-1361539008 > >> > >> Hope we will get something out of it :-). > > > > AHA! We have a theory. The pg log isnt trimmed during scrub (because teh > > old scrub code required that), but the new (deep) scrub can take a very > > long time, which means the pg log will eat ram in the meantime.. > > especially under high iops. > > > > Can you try wip-osd-log-trim (which is bobtail + a simple patch) and see > > if that seems to work? Note that that patch shouldn't be run in a mixed > > argonaut+bobtail cluster, since it isn't properly checking if the scrub is > > class or chunky/deep. > > > > Thanks! > > sage > > > > > > > -- > >> Regards, > >> S?bastien Han. > >> > >> > >> On Fri, Jan 11, 2013 at 7:13 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: > >> > On Fri, Jan 11, 2013 at 6:57 AM, S?bastien Han <han.sebastien@xxxxxxxxx> wrote: > >> >>> Is osd.1 using the heap profiler as well? Keep in mind that active use > >> >>> of the memory profiler will itself cause memory usage to increase ? > >> >>> this sounds a bit like that to me since it's staying stable at a large > >> >>> but finite portion of total memory. > >> >> > >> >> Well, the memory consumption was already high before the profiler was > >> >> started. So yes with the memory profiler enable an OSD might consume > >> >> more memory but this doesn't cause the memory leaks. > >> > > >> > My concern is that maybe you saw a leak but when you restarted with > >> > the memory profiling you lost whatever conditions caused it. > >> > > >> >> Any ideas? Nothing to say about my scrumbing theory? > >> > I like it, but Sam indicates that without some heap dumps which > >> > capture the actual leak then scrub is too large to effectively code > >> > review for leaks. :( > >> > -Greg > >> -- > >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > >> the body of a message to majordomo@xxxxxxxxxxxxxxx > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > >> > >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
