On 02/16/2013 08:09 AM, Andrey Korolyov wrote:
Can anyone who hit this bug please confirm that your system contains libc 2.15+?
I've seen this with 0.56.2 as well on Ubuntu 12.04. Ubuntu 12.04 comes
with 2.15-0ubuntu10.3
Haven't gotten around to adding a heap profiler to it.
Wido
On Tue, Feb 5, 2013 at 1:27 AM, Sébastien Han <han.sebastien@xxxxxxxxx> wrote:
oh nice, the pattern also matches path :D, didn't know that
thanks Greg
--
Regards,
Sébastien Han.
On Mon, Feb 4, 2013 at 10:22 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote:
Set your /proc/sys/kernel/core_pattern file. :) http://linux.die.net/man/5/core
-Greg
On Mon, Feb 4, 2013 at 1:08 PM, Sébastien Han <han.sebastien@xxxxxxxxx> wrote:
ok I finally managed to get something on my test cluster,
unfortunately, the dump goes to /
any idea to change the destination path?
My production / won't be big enough...
--
Regards,
Sébastien Han.
On Mon, Feb 4, 2013 at 10:03 PM, Dan Mick <dan.mick@xxxxxxxxxxx> wrote:
...and/or do you have the corepath set interestingly, or one of the
core-trapping mechanisms turned on?
On 02/04/2013 11:29 AM, Sage Weil wrote:
On Mon, 4 Feb 2013, S?bastien Han wrote:
Hum just tried several times on my test cluster and I can't get any
core dump. Does Ceph commit suicide or something? Is it expected
behavior?
SIGSEGV should trigger the usual path that dumps a stack trace and then
dumps core. Was your ulimit -c set before the daemon was started?
sage
--
Regards,
S?bastien Han.
On Sun, Feb 3, 2013 at 10:03 PM, S?bastien Han <han.sebastien@xxxxxxxxx>
wrote:
Hi Lo?c,
Thanks for bringing our discussion on the ML. I'll check that tomorrow
:-).
Cheer
--
Regards,
S?bastien Han.
On Sun, Feb 3, 2013 at 10:01 PM, S?bastien Han <han.sebastien@xxxxxxxxx>
wrote:
Hi Lo?c,
Thanks for bringing our discussion on the ML. I'll check that tomorrow
:-).
Cheers
--
Regards,
S?bastien Han.
On Sun, Feb 3, 2013 at 7:17 PM, Loic Dachary <loic@xxxxxxxxxxx> wrote:
Hi,
As discussed during FOSDEM, the script you wrote to kill the OSD when
it
grows too much could be amended to core dump instead of just being
killed &
restarted. The binary + core could probably be used to figure out
where the
leak is.
You should make sure the OSD current working directory is in a file
system
with enough free disk space to accomodate for the dump and set
ulimit -c unlimited
before running it ( your system default is probably ulimit -c 0 which
inhibits core dumps ). When you detect that OSD grows too much kill it
with
kill -SEGV $pid
and upload the core found in the working directory, together with the
binary in a public place. If the osd binary is compiled with -g but
without
changing the -O settings, you should have a larger binary file but no
negative impact on performances. Forensics analysis will be made a lot
easier with the debugging symbols.
My 2cts
On 01/31/2013 08:57 PM, Sage Weil wrote:
On Thu, 31 Jan 2013, Sylvain Munaut wrote:
Hi,
I disabled scrubbing using
ceph osd tell \* injectargs '--osd-scrub-min-interval 1000000'
ceph osd tell \* injectargs '--osd-scrub-max-interval 10000000'
and the leak seems to be gone.
See the graph at http://i.imgur.com/A0KmVot.png with the OSD
memory
for the 12 osd processes over the last 3.5 days.
Memory was rising every 24h. I did the change yesterday around 13h00
and OSDs stopped growing. OSD memory even seems to go down slowly by
small blocks.
Of course I assume disabling scrubbing is not a long term solution
and
I should re-enable it ... (how do I do that btw ? what were the
default values for those parameters)
It depends on the exact commit you're on. You can see the defaults
if
you
do
ceph-osd --show-config | grep osd_scrub
Thanks for testing this... I have a few other ideas to try to
reproduce.
sage
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Lo?c Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
