Hi all, I have a cluster of 2 nodes on Centos 6.5 with ceph 0.67.10 (replicate = 2) When I add the 3rd node in the Ceph Cluster, CEPH perform load balancing. I have 3 MDS in 3 nodes,the MDS process is dying after a while with a stack trace: ------------------------------------------------------------------------------------------------------------------------------------------------------- 2014-08-26 17:08:34.362901 7f1c2c704700 1 -- 10.20.0.21:6800/22154 <== osd.10 10.20.0.21:6802/15917 1 ==== osd_op_reply(230 100000003f6.00000000 [tmapup 0~0] ondisk = 0) v4 ==== 119+0+0 (1770421071 0 0) 0x2aece00 con 0x2aa4200 -54> 2014-08-26 17:08:34.362942 7f1c2c704700 1 -- 10.20.0.21:6800/22154 <== osd.55 10.20.0.23:6800/2407 10 ==== osd_op_reply(263 1000000048a.00000000 [getxattr] ack = -2 (No such file or directory)) v4 ==== 119+0+0 (3908997833 0 0) 0x1e63000 con 0x1e7aaa0 -53> 2014-08-26 17:08:34.363001 7f1c2c704700 5 mds.0.log submit_entry 427629603~1541 : EUpdate purge_stray truncate [metablob 100, 2 dirs] -52> 2014-08-26 17:08:34.363022 7f1c2c704700 1 -- 10.20.0.21:6800/22154 <== osd.37 10.20.0.22:6898/11994 6 ==== osd_op_reply(226 1.00000000 [tmapput 0~7664] ondisk = 0) v4 ==== 109+0+0 (1007110430 0 0) 0x1e64800 con 0x1e7a7e0 -51> 2014-08-26 17:08:34.363092 7f1c2c704700 5 mds.0.log _expired segment 293601899 2548 events -50> 2014-08-26 17:08:34.363117 7f1c2c704700 1 -- 10.20.0.21:6800/22154 <== osd.17 10.20.0.21:6941/17572 9 ==== osd_op_reply(264 10000000489.00000000 [getxattr] ack = -2 (No such file or directory)) v4 ==== 119+0+0 (1979034473 0 0) 0x1e62200 con 0x1e7b180 -49> 2014-08-26 17:08:34.363177 7f1c2c704700 5 mds.0.log submit_entry 427631148~1541 : EUpdate purge_stray truncate [metablob 100, 2 dirs] -48> 2014-08-26 17:08:34.363197 7f1c2c704700 1 -- 10.20.0.21:6800/22154 <== osd.1 10.20.0.21:6872/13227 6 ==== osd_op_reply(265 10000000491.00000000 [getxattr] ack = -2 (No such file or directory)) v4 ==== 119+0+0 (1231782695 0 0) 0x1e63400 con 0x1e7ac00 -47> 2014-08-26 17:08:34.363255 7f1c2c704700 5 mds.0.log submit_entry 427632693~1541 : EUpdate purge_stray truncate [metablob 100, 2 dirs] -46> 2014-08-26 17:08:34.363274 7f1c2c704700 1 -- 10.20.0.21:6800/22154 <== osd.11 10.20.0.21:6884/7018 5 ==== osd_op_reply(266 1000000047d.00000000 [getxattr] ack = -2 (No such file or directory)) v4 ==== 119+0+0 (2737916920 0 0) 0x1e61e00 con 0x1e7bc80 --------------------------------------------------------------------------------------------------------------------------------------------- I try to restart MDSs, but after a few seconds in a state of "active", MDS switch to state "laggy or crashed". I have a lot of important data on it. I do not want to use the command: ceph mds newfs <metadata pool id> <data pool id> --yes-i-really-mean-it :( Tien Bui. -- Bui Minh Tien -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140826/f2cb7e6f/attachment.htm>
 |