Hi Sage, Sorry it's a production system, so I can't test it. So at the end, you can't get anything out of the core dump? -- Regards, Sébastien Han. On Sat, Feb 23, 2013 at 1:44 AM, Sage Weil <sage@xxxxxxxxxxx> wrote: > On Fri, 22 Feb 2013, S?bastien Han wrote: >> Hi all, >> >> I finally got a core dump. >> >> I did it with a kill -SEGV on the OSD process. >> >> https://www.dropbox.com/s/ahv6hm0ipnak5rf/core-ceph-osd-11-0-0-20100-1361539008 >> >> Hope we will get something out of it :-). > > AHA! We have a theory. The pg log isnt trimmed during scrub (because teh > old scrub code required that), but the new (deep) scrub can take a very > long time, which means the pg log will eat ram in the meantime.. > especially under high iops. > > Can you try wip-osd-log-trim (which is bobtail + a simple patch) and see > if that seems to work? Note that that patch shouldn't be run in a mixed > argonaut+bobtail cluster, since it isn't properly checking if the scrub is > class or chunky/deep. > > Thanks! > sage > > > > -- >> Regards, >> S?bastien Han. >> >> >> On Fri, Jan 11, 2013 at 7:13 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: >> > On Fri, Jan 11, 2013 at 6:57 AM, S?bastien Han <han.sebastien@xxxxxxxxx> wrote: >> >>> Is osd.1 using the heap profiler as well? Keep in mind that active use >> >>> of the memory profiler will itself cause memory usage to increase ? >> >>> this sounds a bit like that to me since it's staying stable at a large >> >>> but finite portion of total memory. >> >> >> >> Well, the memory consumption was already high before the profiler was >> >> started. So yes with the memory profiler enable an OSD might consume >> >> more memory but this doesn't cause the memory leaks. >> > >> > My concern is that maybe you saw a leak but when you restarted with >> > the memory profiling you lost whatever conditions caused it. >> > >> >> Any ideas? Nothing to say about my scrumbing theory? >> > I like it, but Sam indicates that without some heap dumps which >> > capture the actual leak then scrub is too large to effectively code >> > review for leaks. :( >> > -Greg >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
