Ceph 17.2.3 (dockerized in Ubuntu 20.04)
The subject says it. The MDS process always crashes after evicting. ceph -w
shows:
2022-09-22T13:26:23.305527+0200 mds.ksz-cephfs2.ceph00.kqjdwe [INF]
Evicting (and blocklisting) client session 5181680 (
10.149.12.21:0/3369570791)
2022-09-22T13:26:35.729317+0200 mon.ceph00 [INF] daemon
mds.ksz-cephfs2.ceph03.vsyrbk restarted
2022-09-22T13:26:36.039678+0200 mon.ceph00 [INF] daemon
mds.ksz-cephfs2.ceph01.xybiqv restarted
2022-09-22T13:29:21.000392+0200 mds.ksz-cephfs2.ceph04.ekmqio [INF]
Evicting (and blocklisting) client session 5249349 (
10.149.12.22:0/2459302619)
2022-09-22T13:29:32.069656+0200 mon.ceph00 [INF] daemon
mds.ksz-cephfs2.ceph01.xybiqv restarted
2022-09-22T13:30:00.000101+0200 mon.ceph00 [INF] overall HEALTH_OK
2022-09-22T13:30:20.710271+0200 mon.ceph00 [WRN] Health check failed: 1
daemons have recently crashed (RECENT_CRASH)
The crash info of the crashed MDS is:
# ceph crash info
2022-09-22T11:26:24.013274Z_b005f3fc-7704-4cfc-96c5-f2a9c993f166
{
"assert_condition": "!mds->is_any_replay()",
"assert_file":
"/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.3/rpm/el8/BUILD/ceph-17.2.3/src/mds/MDLog.cc",
"assert_func": "void MDLog::_submit_entry(LogEvent*,
MDSLogContextBase*)",
"assert_line": 283,
"assert_msg":
"/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.3/rpm/el8/BUILD/ceph-17.2.3/src/mds/MDLog.cc:
In function 'void MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)'
thread 7f76fa8f6700 time
2022-09-22T11:26:23.992050+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.3/rpm/el8/BUILD/ceph-17.2.3/src/mds/MDLog.cc:
283: FAILED ceph_assert(!mds->is_any_replay())\n",
"assert_thread_name": "ms_dispatch",
"backtrace": [
"/lib64/libpthread.so.0(+0x12ce0) [0x7f770231bce0]",
"gsignal()",
"abort()",
"(ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x1b0) [0x7f770333bcd2]",
"/usr/lib64/ceph/libceph-common.so.2(+0x283e95) [0x7f770333be95]",
"(MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0x3f)
[0x55991905efdf]",
"(Server::journal_close_session(Session*, int, Context*)+0x78c)
[0x559918d7d63c]",
"(Server::kill_session(Session*, Context*)+0x212) [0x559918d7dd92]",
"(Server::apply_blocklist()+0x10d) [0x559918d7e04d]",
"(MDSRank::apply_blocklist(std::set<entity_addr_t,
std::less<entity_addr_t>, std::allocator<entity_addr_t> > const&, unsigned
int)+0x34) [0x559918d39d74]",
"(MDSRankDispatcher::handle_osd_map()+0xf6) [0x559918d3a0b6]",
"(MDSDaemon::handle_core_message(boost::intrusive_ptr<Message const>
const&)+0x39b) [0x559918d2330b]",
"(MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message>
const&)+0xc3) [0x559918d23cc3]",
"(DispatchQueue::entry()+0x14fa) [0x7f77035c240a]",
"(DispatchQueue::DispatchThread::entry()+0x11) [0x7f7703679481]",
"/lib64/libpthread.so.0(+0x81ca) [0x7f77023111ca]",
"clone()"
],
"ceph_version": "17.2.3",
"crash_id":
"2022-09-22T11:26:24.013274Z_b005f3fc-7704-4cfc-96c5-f2a9c993f166",
"entity_name": "mds.ksz-cephfs2.ceph03.vsyrbk",
"os_id": "centos",
"os_name": "CentOS Stream",
"os_version": "8",
"os_version_id": "8",
"process_name": "ceph-mds",
"stack_sig":
"b75e46941b5f6b7c05a037f9af5d42bb19d82ab7fc6a3c168533fc31a42b4de8",
"timestamp": "2022-09-22T11:26:24.013274Z",
"utsname_hostname": "ceph03",
"utsname_machine": "x86_64",
"utsname_release": "5.4.0-125-generic",
"utsname_sysname": "Linux",
"utsname_version": "#141-Ubuntu SMP Wed Aug 10 13:42:03 UTC 2022"
}
(Don't be confused by the time information, "ceph -w" is UTC+2, "crash
info" is UTC)
Should I report this a bug or did I miss something which caused the error?
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
