On 08/29/2012 12:21 AM, Gregory Farnum wrote:
On Tue, Aug 28, 2012 at 7:50 AM, Xiaopong Tran <xiaopong.tran@xxxxxxxxx> wrote:
On 08/25/2012 12:28 AM, Sage Weil wrote:
On Fri, 24 Aug 2012, Xiaopong Tran wrote:
Hello,
I've been running the 0.48argonaut on production for over a month
without any issue. and today, I suddenly lost one mon. Taking a look
into the syslog file, I see the following trace log. I just couldn't
see what's wrong from the trace log. However, this event created
a gigantic core file. Here's the size of the core file:
-rw------- 1 root root 16085647360 Aug 24 14:53 core
This happened while we were migrating data from our old storage
to the ceph. We are running about 20 processes, migrating data
into ceph, while there are about 30 more application processes
reading from and writing new data to it.
The following is from syslog:
We've seen these backtraces before too, but haven't figured out what
causes them. (See, for example, http://tracker.newdream.net/issues/2026.)
Was there anything in the mon's log file? In most cases, a crash results
in a stack trace of ceph-mon in the mon log file.
Glad to hear everything recovered nicely afterwards. :)
Thanks!
sage
Ah well, I got two crashes in less than 3 days. I browsed thru the
mon log files, and the ceph log files, and there is nothing suspicious,
no trace dump or anything.
One question I don't get is, after mon has crashed, it's not running
anymore, who is creating that empty mon log? The same question goes
for osd. I had two osd down today, and I also see empty osd log files.
And how does the crash end up generating such a huge core file?
If there's any information I can provide, I'd be happy to do so.
Can you extract the backtrace from the core dump?
Will try to do that, it's a big one though :)
Thanks
Xiaopong
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
