Hi Ben, This looks a lot like the issue I had. Do you know what caused this crashing? In my case, I issued “ceph mds rmfailed”. Did you also issued this command? Here is what I did to prevent the crash, if I recall it correctly. In gdb, before running the mon: (gdb) b MDSMonitor::maybe_resize_cluster (gdb) commands return false cont end (gdb) set pagination off (gdb) run If you manage to prevent the mon from crashing, you can attach the output of “ceph fs dump” to help identify the problem. Weiwen Hu 发件人: Ben Timby<mailto:btimby@xxxxxxxxxxxxx> 发送时间: 2021年10月10日 16:20 收件人: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> 主题: Cluster inaccessible A colleague asked me to take a look at a Ceph cluster that has stopped working. The "ceph -s" command (any ceph command) just times out. Of three monitors two are crashing with: (gdb) bt #0 0x00007fffee17b7bb in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fffee166535 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007fffee530983 in __gnu_cxx::__verbose_terminate_handler () at ../../../../src/libstdc++-v3/libsupc++/vterminate.cc:95 #3 0x00007fffee5368c6 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../src/libstdc++-v3/libsupc++/eh_terminate.cc:47 #4 0x00007fffee536901 in std::terminate () at ../../../../src/libstdc++-v3/libsupc++/eh_terminate.cc:57 #5 0x00007fffee536b34 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x555557d2be50, tinfo=0x7fffee61a9b8 <typeinfo for std::out_of_range>, dest=0x7fffee54b680 <std::out_of_range::~out_of_range()>) at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:95 #6 0x00007fffee53286b in std::__throw_out_of_range (__s=__s@entry=0x5555563cce7f "map::at") from /lib/x86_64-linux-gnu/libstdc++.so.6 #7 0x0000555555f025be in std::map<int, mds_gid_t, std::less<int>, std::allocator<std::pair<int const, mds_gid_t> > >::at (__k=<synthetic pointer>: <optimized out>, this=0x555557dc7a50) at ./src/mon/MDSMonitor.cc:2010 #8 MDSMap::get_info (m=2, this=0x555557dc7818) at ./src/mds/MDSMap.h:448 #9 MDSMonitor::maybe_resize_cluster (this=0x5555580f4580, fsmap=..., fscid=<optimized out>) at ./src/mon/MDSMonitor.cc:2006 #10 0x0000555555f04d51 in MDSMonitor::tick (this=0x5555580f4580) at /usr/include/c++/8/bits/shared_ptr_base.h:1018 #11 0x0000555555eef2f8 in MDSMonitor::on_active (this=0x5555580f4580) at ./src/mon/MDSMonitor.cc:913 #12 0x0000555555e3213d in PaxosService::_active (this=0x5555580f4580) at ./src/mon/PaxosService.cc:356 #13 0x0000555555d29539 in Context::complete (this=0x555556f2d450, r=<optimized out>) at ./src/include/Context.h:99 #14 0x0000555555d54e2d in finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > > (cct=0x555556fe2000, Python Exception <class 'ValueError'> Cannot find type class std::__cxx11::list<Context*, std::allocator<Context*> >::_Node: finished=empty std::__cxx11::list, result=result@entry=0) at /usr/include/c++/8/bits/stl_list.h:971 #15 0x0000555555e296d9 in Paxos::finish_round (this=0x555556fceb00) at ./src/mon/Paxos.cc:1083 #16 0x0000555555e2a7f0 in Paxos::handle_last (this=0x555556fceb00, op=...) at ./src/mon/Paxos.cc:589 #17 0x0000555555e2b42f in Paxos::dispatch (this=0x555556fceb00, op=...) at /usr/include/c++/8/bits/atomic_base.h:295 #18 0x0000555555d27413 in Monitor::dispatch_op (this=0x55555810da00, op=...) at /usr/include/c++/8/bits/atomic_base.h:295 #19 0x0000555555d27cce in Monitor::_ms_dispatch (this=0x55555810da00, m=0x55555816c000) at /usr/include/c++/8/bits/atomic_base.h:295 #20 0x0000555555d55a48 in Monitor::ms_dispatch (m=0x55555816c000, this=0x55555810da00) at ./src/mon/Monitor.h:937 #21 Dispatcher::ms_dispatch2 (this=0x55555810da00, m=...) at ./src/msg/Dispatcher.h:124 #22 0x00007fffef783932 in DispatchQueue::entry() () from /usr/lib/ceph/libceph-common.so.2 #23 0x00007fffef82f5dd in DispatchQueue::DispatchThread::entry() () from /usr/lib/ceph/libceph-common.so.2 #24 0x00007fffee68dfa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #25 0x00007fffee23d4cf in clone () from /lib/x86_64-linux-gnu/libc.so.6 >From this I can tell that at this line of code: https://github.com/ceph/ceph/blob/pacific/src/mon/MDSMonitor.cc#L1971 in == 3, thus target == 2 at: https://github.com/ceph/ceph/blob/pacific/src/mon/MDSMonitor.cc#L2006 Which is out of bounds of mds_map. gdb tells me that: mds_map = @0x555557dc7818: {compat = {compat = {mask = 1, names = std::map with 0 elements}, ro_compat = {mask = 1, names = std::map with 0 elements}, incompat = {mask = 1919, names = std::map with 9 elements = {[1] = "base v0.20", [2] = "client writeable ranges", [3] = "default file layouts on dirs", [4] = "dir inode in separate object", [5] = "mds uses versioned encoding", [6] = "dirfrag is stored in omap", [8] = "no anchor table", [9] = "file layout v2", [10] = "snaprealm v2"}}}, epoch = 68021, enabled = true, fs_name = "prod", flags = 18, last_failure = 0, last_failure_osd_epoch = 9953, created = {tv = {tv_sec = 1630197046, tv_nsec = 944134388}}, modified = {tv = {tv_sec = 1633835801, tv_nsec = 859392387}}, tableserver = 0, root = 0, session_timeout = 60, session_autoclose = 300, max_file_size = 1099511627776, required_client_features = {static bits_per_block = 64, _vec = std::vector of length 0, capacity 0}, data_pools = std::vector of length 1, capacity 1 = {5}, cas_pool = -1, metadata_pool = 4, max_mds = 1, old_max_mds = 1, standby_count_wanted = 1, balancer = "", in = std::set with 3 elements = {[0] = 0, [1] = 1, [2] = 2}, failed = std::set with 0 elements, stopped = std::set with 1 element = {[0] = 3}, damaged = std::set with 0 elements, up = std::map with 0 elements, mds_info = std::map with 0 elements, ever_allowed_features = 32 ' ', explicitly_allowed_features = 32 ' ', inline_data_enabled = false, cached_up_features = 0} I observed the metadata servers using strace and they are all in a loop trying to contact monitors (which are not running) and receiving ECONNREFUSED. I am not sure what I need to change to get the monitors to start, I did find this thread which mentions maybe_resize_cluster(), but I am unsure if this information is relevant. https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/KQ5A5OWRIUEOJBC7VILBGDIKPQGJQIWN/ Thank you for any help you may be able to provide. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |