Hi Gregory Farmum, Thank you for your reply! This is the log: 2014-08-26 16:22:39.103461 7f083752f700 -1 mds/CDir.cc: In function 'void CDir::_committed(version_t)' thread 7f083752f700 time 2014-08-26 16:22:39.075809 mds/CDir.cc: 2071: FAILED assert(in->is_dirty() || in->last < ((__u64)(-2))) ceph version 0.67.10 (9d446bd416c52cd785ccf048ca67737ceafcdd7f) 1: (CDir::_committed(unsigned long)+0xc4e) [0x74d9ee] 2: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe8d) [0x7d09bd] 3: (MDS::handle_core_message(Message*)+0x987) [0x57c457] 4: (MDS::_dispatch(Message*)+0x2f) [0x57c50f] 5: (MDS::ms_dispatch(Message*)+0x19b) [0x57dfbb] 6: (DispatchQueue::entry()+0x5a2) [0x904732] 7: (DispatchQueue::DispatchThread::entry()+0xd) [0x8afdbd] 8: (()+0x79d1) [0x7f083c2979d1] 9: (clone()+0x6d) [0x7f083afb6b5d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 hadoop 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.Ceph01-dc5k3u0104.log --- end dump of recent events --- 2014-08-26 16:22:39.134173 7f083752f700 -1 *** Caught signal (Aborted) ** in thread 7f083752f700 On Wed, Aug 27, 2014 at 3:09 AM, Gregory Farnum <greg at inktank.com> wrote: > I don't think the log messages you're showing are the actual cause of > the failure. The log file should have a proper stack trace (with > specific function references and probably a listed assert failure), > can you find that? > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com > > > On Tue, Aug 26, 2014 at 9:11 AM, MinhTien MinhTien > <tientienminh080590 at gmail.com> wrote: > > Hi all, > > > > I have a cluster of 2 nodes on Centos 6.5 with ceph 0.67.10 (replicate > = 2) > > > > When I add the 3rd node in the Ceph Cluster, CEPH perform load balancing. > > > > I have 3 MDS in 3 nodes,the MDS process is dying after a while with a > stack > > trace: > > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------- > > > > 2014-08-26 17:08:34.362901 7f1c2c704700 1 -- 10.20.0.21:6800/22154 <== > > osd.10 10.20.0.21:6802/15917 1 ==== osd_op_reply(230 > 100000003f6.00000000 > > [tmapup 0~0] ondisk = 0) v4 ==== 119+0+0 (1770421071 0 0) 0x2aece00 con > > 0x2aa4200 > > -54> 2014-08-26 17:08:34.362942 7f1c2c704700 1 -- > 10.20.0.21:6800/22154 > > <== osd.55 10.20.0.23:6800/2407 10 ==== osd_op_reply(263 > > 1000000048a.00000000 [getxattr] ack = -2 (No such file or directory)) v4 > > ==== 119+0+0 (3908997833 0 0) 0x1e63000 con 0x1e7aaa0 > > -53> 2014-08-26 17:08:34.363001 7f1c2c704700 5 mds.0.log submit_entry > > 427629603~1541 : EUpdate purge_stray truncate [metablob 100, 2 dirs] > > -52> 2014-08-26 17:08:34.363022 7f1c2c704700 1 -- > 10.20.0.21:6800/22154 > > <== osd.37 10.20.0.22:6898/11994 6 ==== osd_op_reply(226 1.00000000 > [tmapput > > 0~7664] ondisk = 0) v4 ==== 109+0+0 (1007110430 0 0) 0x1e64800 con > 0x1e7a7e0 > > -51> 2014-08-26 17:08:34.363092 7f1c2c704700 5 mds.0.log _expired > > segment 293601899 2548 events > > -50> 2014-08-26 17:08:34.363117 7f1c2c704700 1 -- > 10.20.0.21:6800/22154 > > <== osd.17 10.20.0.21:6941/17572 9 ==== osd_op_reply(264 > > 10000000489.00000000 [getxattr] ack = -2 (No such file or directory)) v4 > > ==== 119+0+0 (1979034473 0 0) 0x1e62200 con 0x1e7b180 > > -49> 2014-08-26 17:08:34.363177 7f1c2c704700 5 mds.0.log submit_entry > > 427631148~1541 : EUpdate purge_stray truncate [metablob 100, 2 dirs] > > -48> 2014-08-26 17:08:34.363197 7f1c2c704700 1 -- > 10.20.0.21:6800/22154 > > <== osd.1 10.20.0.21:6872/13227 6 ==== osd_op_reply(265 > 10000000491.00000000 > > [getxattr] ack = -2 (No such file or directory)) v4 ==== 119+0+0 > (1231782695 > > 0 0) 0x1e63400 con 0x1e7ac00 > > -47> 2014-08-26 17:08:34.363255 7f1c2c704700 5 mds.0.log submit_entry > > 427632693~1541 : EUpdate purge_stray truncate [metablob 100, 2 dirs] > > -46> 2014-08-26 17:08:34.363274 7f1c2c704700 1 -- > 10.20.0.21:6800/22154 > > <== osd.11 10.20.0.21:6884/7018 5 ==== osd_op_reply(266 > 1000000047d.00000000 > > [getxattr] ack = -2 (No such file or directory)) v4 ==== 119+0+0 > (2737916920 > > 0 0) 0x1e61e00 con 0x1e7bc80 > > > > > --------------------------------------------------------------------------------------------------------------------------------------------- > > I try to restart MDSs, but after a few seconds in a state of "active", > MDS > > switch to state "laggy or crashed". I have a lot of important data on it. > > I do not want to use the command: > > ceph mds newfs <metadata pool id> <data pool id> --yes-i-really-mean-it > > > > :( > > > > Tien Bui. > > > > > > > > -- > > Bui Minh Tien > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users at lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > -- Bui Minh Tien -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140827/5099e7d6/attachment.htm>
 |