Please first delete the old mds log, then run mds with "debug_mds = 15". Send the whole mds log to us after the mds crashes. Yan, Zheng On Wed, Aug 27, 2014 at 12:12 PM, MinhTien MinhTien < tientienminh080590 at gmail.com> wrote: > Hi Gregory Farmum, > > Thank you for your reply! > This is the log: > > 2014-08-26 16:22:39.103461 7f083752f700 -1 mds/CDir.cc: In function 'void > CDir::_committed(version_t)' thread 7f083752f700 time 2014-08-26 > 16:22:39.075809 > mds/CDir.cc: 2071: FAILED assert(in->is_dirty() || in->last < > ((__u64)(-2))) > > ceph version 0.67.10 (9d446bd416c52cd785ccf048ca67737ceafcdd7f) > 1: (CDir::_committed(unsigned long)+0xc4e) [0x74d9ee] > 2: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe8d) [0x7d09bd] > 3: (MDS::handle_core_message(Message*)+0x987) [0x57c457] > 4: (MDS::_dispatch(Message*)+0x2f) [0x57c50f] > 5: (MDS::ms_dispatch(Message*)+0x19b) [0x57dfbb] > 6: (DispatchQueue::entry()+0x5a2) [0x904732] > 7: (DispatchQueue::DispatchThread::entry()+0xd) [0x8afdbd] > 8: (()+0x79d1) [0x7f083c2979d1] > 9: (clone()+0x6d) [0x7f083afb6b5d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed > to interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 0/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/ 5 hadoop > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > -2/-2 (syslog threshold) > -1/-1 (stderr threshold) > max_recent 10000 > max_new 1000 > log_file /var/log/ceph/ceph-mds.Ceph01-dc5k3u0104.log > --- end dump of recent events --- > 2014-08-26 16:22:39.134173 7f083752f700 -1 *** Caught signal (Aborted) ** > in thread 7f083752f700 > > > > > On Wed, Aug 27, 2014 at 3:09 AM, Gregory Farnum <greg at inktank.com> wrote: > >> I don't think the log messages you're showing are the actual cause of >> the failure. The log file should have a proper stack trace (with >> specific function references and probably a listed assert failure), >> can you find that? >> -Greg >> Software Engineer #42 @ http://inktank.com | http://ceph.com >> >> >> On Tue, Aug 26, 2014 at 9:11 AM, MinhTien MinhTien >> <tientienminh080590 at gmail.com> wrote: >> > Hi all, >> > >> > I have a cluster of 2 nodes on Centos 6.5 with ceph 0.67.10 (replicate >> = 2) >> > >> > When I add the 3rd node in the Ceph Cluster, CEPH perform load >> balancing. >> > >> > I have 3 MDS in 3 nodes,the MDS process is dying after a while with a >> stack >> > trace: >> > >> > >> ------------------------------------------------------------------------------------------------------------------------------------------------------- >> > >> > 2014-08-26 17:08:34.362901 7f1c2c704700 1 -- 10.20.0.21:6800/22154 >> <== >> > osd.10 10.20.0.21:6802/15917 1 ==== osd_op_reply(230 >> 100000003f6.00000000 >> > [tmapup 0~0] ondisk = 0) v4 ==== 119+0+0 (1770421071 0 0) 0x2aece00 con >> > 0x2aa4200 >> > -54> 2014-08-26 17:08:34.362942 7f1c2c704700 1 -- >> 10.20.0.21:6800/22154 >> > <== osd.55 10.20.0.23:6800/2407 10 ==== osd_op_reply(263 >> > 1000000048a.00000000 [getxattr] ack = -2 (No such file or directory)) v4 >> > ==== 119+0+0 (3908997833 0 0) 0x1e63000 con 0x1e7aaa0 >> > -53> 2014-08-26 17:08:34.363001 7f1c2c704700 5 mds.0.log >> submit_entry >> > 427629603~1541 : EUpdate purge_stray truncate [metablob 100, 2 dirs] >> > -52> 2014-08-26 17:08:34.363022 7f1c2c704700 1 -- >> 10.20.0.21:6800/22154 >> > <== osd.37 10.20.0.22:6898/11994 6 ==== osd_op_reply(226 1.00000000 >> [tmapput >> > 0~7664] ondisk = 0) v4 ==== 109+0+0 (1007110430 0 0) 0x1e64800 con >> 0x1e7a7e0 >> > -51> 2014-08-26 17:08:34.363092 7f1c2c704700 5 mds.0.log _expired >> > segment 293601899 2548 events >> > -50> 2014-08-26 17:08:34.363117 7f1c2c704700 1 -- >> 10.20.0.21:6800/22154 >> > <== osd.17 10.20.0.21:6941/17572 9 ==== osd_op_reply(264 >> > 10000000489.00000000 [getxattr] ack = -2 (No such file or directory)) v4 >> > ==== 119+0+0 (1979034473 0 0) 0x1e62200 con 0x1e7b180 >> > -49> 2014-08-26 17:08:34.363177 7f1c2c704700 5 mds.0.log >> submit_entry >> > 427631148~1541 : EUpdate purge_stray truncate [metablob 100, 2 dirs] >> > -48> 2014-08-26 17:08:34.363197 7f1c2c704700 1 -- >> 10.20.0.21:6800/22154 >> > <== osd.1 10.20.0.21:6872/13227 6 ==== osd_op_reply(265 >> 10000000491.00000000 >> > [getxattr] ack = -2 (No such file or directory)) v4 ==== 119+0+0 >> (1231782695 >> > 0 0) 0x1e63400 con 0x1e7ac00 >> > -47> 2014-08-26 17:08:34.363255 7f1c2c704700 5 mds.0.log >> submit_entry >> > 427632693~1541 : EUpdate purge_stray truncate [metablob 100, 2 dirs] >> > -46> 2014-08-26 17:08:34.363274 7f1c2c704700 1 -- >> 10.20.0.21:6800/22154 >> > <== osd.11 10.20.0.21:6884/7018 5 ==== osd_op_reply(266 >> 1000000047d.00000000 >> > [getxattr] ack = -2 (No such file or directory)) v4 ==== 119+0+0 >> (2737916920 >> > 0 0) 0x1e61e00 con 0x1e7bc80 >> > >> > >> --------------------------------------------------------------------------------------------------------------------------------------------- >> > I try to restart MDSs, but after a few seconds in a state of "active", >> MDS >> > switch to state "laggy or crashed". I have a lot of important data on >> it. >> > I do not want to use the command: >> > ceph mds newfs <metadata pool id> <data pool id> --yes-i-really-mean-it >> > >> > :( >> > >> > Tien Bui. >> > >> > >> > >> > -- >> > Bui Minh Tien >> > >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users at lists.ceph.com >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > >> > > > > -- > Bui Minh Tien > > _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140827/713381f9/attachment.htm>
 |