> Did you run the command I suggested before or after you executed `rmfailed` below? I run “rmfailed” before reading your mail. Then I got MON crashed. I fixed the crash by setting max_mds=2. Then I tried the command you suggested. By reading the code[1], I think I really need to undo the “rmfailed” to get my MDS out of standby state. > I will compile an addfailed command in a branch but you'll need to download the packages and run it. Recompile can be hard, I’m not familiar with the procedure. Now I’m going to modify the logic of[2] a little bit with gdb to insert the failed rank. > Please be careful running hidden/debugging commands. I will definitely be more careful in the future. Thanks again for your help. [1]: https://github.com/ceph/ceph/blob/v16.2.6/src/mon/MDSMonitor.cc#L2238 [2]: https://github.com/ceph/ceph/blob/v16.2.6/src/mds/FSMap.cc#L1031 > What was the crash? ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f7e9e32cb20] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f7e9d94509b] 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f7e9d94b53c] 6: /lib64/libstdc++.so.6(+0x96597) [0x7f7e9d94b597] 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f7e9d94b7f8] 8: /lib64/libstdc++.so.6(+0x9204b) [0x7f7e9d94704b] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xb7f) [0x558c2f6d72ff] 10: (MDSMonitor::tick()+0x161) [0x558c2f6d9cd1] 11: (MDSMonitor::on_active()+0x2c) [0x558c2f6c34bc] 12: (PaxosService::_active()+0x1f5) [0x558c2f5fd865] 13: (Context::complete(int)+0xd) [0x558c2f4eaead] 14: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(ceph::common::CephContext*, … 15: (Paxos::finish_round()+0x169) [0x558c2f5f4139] 16: (Paxos::commit_finish()+0x8c0) [0x558c2f5f6c60] 17: (C_Committed::finish(int)+0x45) [0x558c2f5fa885] 18: (Context::complete(int)+0xd) [0x558c2f4eaead] 19: (MonitorDBStore::C_DoTransaction::finish(int)+0x98) [0x558c2f5fa5a8] 20: (Context::complete(int)+0xd) [0x558c2f4eaead] 21: (Finisher::finisher_thread_entry()+0x1a5) [0x7f7ea068d6d5] 22: /lib64/libpthread.so.0(+0x814a) [0x7f7e9e32214a] 23: clone() _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |