For some unknown reason, I’m getting mds failures. Running pacific 16.2.4. Crash logs:
LOG:
--- begin dump of recent events ---
-371> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command assert hook 0x55761e800050
-370> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command abort hook 0x55761e800050
-369> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command leak_some_memory hook 0x55761e800050
-368> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perfcounters_dump hook 0x55761e800050
-367> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command 1 hook 0x55761e800050
-366> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf dump hook 0x55761e800050
-365> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perfcounters_schema hook 0x55761e800050
-364> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf histogram dump hook 0x55761e800050
-363> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command 2 hook 0x55761e800050
-362> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf schema hook 0x55761e800050
-361> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf histogram schema hook 0x55761e800050
-360> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf reset hook 0x55761e800050
-359> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config show hook 0x55761e800050
-358> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config help hook 0x55761e800050
-357> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config set hook 0x55761e800050
-356> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config unset hook 0x55761e800050
-355> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config get hook 0x55761e800050
-354> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config diff hook 0x55761e800050
-353> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config diff get hook 0x55761e800050
-352> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command injectargs hook 0x55761e800050
-351> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command log flush hook 0x55761e800050
-350> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command log dump hook 0x55761e800050
-349> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command log reopen hook 0x55761e800050
-348> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_mempools hook 0x55761e8c6068
-347> 2021-12-23T03:54:44.718+0000 7fc8db144780 10 monclient: get_monmap_and_config
-346> 2021-12-23T03:54:44.718+0000 7fc8db144780 10 monclient: build_initial_monmap
-345> 2021-12-23T03:54:44.718+0000 7fc8db144780 10 monclient: monmap:
epoch 0
fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d
last_changed 2021-12-23T03:54:44.719460+0000
created 2021-12-23T03:54:44.719460+0000
min_mon_release 0 (unknown)
election_strategy: 1
0: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.noname-a
1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.noname-b
2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.noname-c
3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.noname-d
4: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.noname-e
-344> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx
-343> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx
-342> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx
-341> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: none
-340> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure
-339> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc
-338> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure
-337> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc
-336> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure
-335> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc
-334> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc
-333> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure
-332> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc
-331> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure
-330> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc
-329> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure
-328> 2021-12-23T03:54:44.718+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring
-327> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient: init
-326> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding auth protocol: cephx
-325> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding auth protocol: cephx
-324> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding auth protocol: cephx
-323> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding auth protocol: none
-322> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure
-321> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc
-320> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure
-319> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc
-318> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure
-317> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc
-316> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc
-315> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure
-314> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc
-313> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure
-312> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc
-311> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure
-310> 2021-12-23T03:54:44.719+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring
-309> 2021-12-23T03:54:44.719+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring
-308> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient: _reopen_session rank -1
-307> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient: _add_conns ranks=[1,0,4,2,3]
-306> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-b con 0x55761e8f8800 addr [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0]
-305> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-a con 0x55761e8f8c00 addr [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0]
-304> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-e con 0x55761e8f9000 addr [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0]
-303> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): start opening mon connection
-302> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): start opening mon connection
-301> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): start opening mon connection
-300> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): _renew_subs
-299> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): authenticate will time out at 2021-12-23T03:59:44.720414+0000
-298> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): get_auth_request con 0x55761e8f8800 auth_method 0
-297> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1]
-296> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): _init_auth method 2
-295> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): _init_auth creating new auth
-294> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): get_auth_request con 0x55761e8f8c00 auth_method 0
-293> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1]
-292> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): _init_auth method 2
-291> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): _init_auth creating new auth
-290> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): get_auth_request con 0x55761e8f9000 auth_method 0
-289> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1]
-288> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): _init_auth method 2
-287> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): _init_auth creating new auth
-286> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more payload 9
-285> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more payload_len 9
-284> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes
-283> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more payload 9
-282> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more payload_len 9
-281> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes
-280> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more payload 9
-279> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more payload_len 9
-278> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes
-277> 2021-12-23T03:54:44.720+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_done global_id 10847251 payload 290
-276> 2021-12-23T03:54:44.720+0000 7fc8cc07d700 10 monclient: _finish_hunting 0
-275> 2021-12-23T03:54:44.720+0000 7fc8cc07d700 1 monclient: found mon.noname-b
-274> 2021-12-23T03:54:44.720+0000 7fc8cc07d700 10 monclient: _send_mon_message to mon.noname-b at v2:192.168.30.12:3300/0
-273> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: handle_monmap mon_map magic: 0 v1
-272> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: got monmap 69 from mon.noname-b (according to old e69)
-271> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: dump:
epoch 69
fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d
last_changed 2021-06-24T11:48:32.597666+0000
created 2021-05-29T06:47:46.644056+0000
min_mon_release 16 (pacific)
election_strategy: 1
0: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.cn05
1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.cn02
2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.cn03
3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.cn04
4: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.cn01
-270> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _finish_auth 0
-269> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2021-12-23T03:54:14.721727+0000)
-268> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _send_mon_message to mon.cn02 at v2:192.168.30.12:3300/0
-267> 2021-12-23T03:54:44.720+0000 7fc8db144780 5 monclient: authenticate success, global_id 10847251
-266> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: handle_config config(4 keys) v1
-265> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: handle_monmap mon_map magic: 0 v1
-264> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: got monmap 69 from mon.cn02 (according to old e69)
-263> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: dump:
epoch 69
fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d
last_changed 2021-06-24T11:48:32.597666+0000
created 2021-05-29T06:47:46.644056+0000
min_mon_release 16 (pacific)
election_strategy: 1
0: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.cn05
1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.cn02
2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.cn03
3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.cn04
4: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.cn01
-262> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 monclient: get_monmap_and_config success
-261> 2021-12-23T03:54:44.720+0000 7fc8db144780 4 set_mon_vals no callback set
-260> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 set_mon_vals container_image = docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949
-259> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 set_mon_vals mds_join_fs = blumfrub
-258> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 set_mon_vals mon_allow_pool_delete = true
-257> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 set_mon_vals rbd_move_to_trash_on_remove = true
-256> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _finish_auth 0
-255> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2021-12-23T03:54:14.722014+0000)
-254> 2021-12-23T03:54:44.721+0000 7fc8db144780 10 monclient: shutdown
-253> 2021-12-23T03:54:44.721+0000 7fc8db144780 0 set uid:gid to 167:167 (ceph:ceph)
-252> 2021-12-23T03:54:44.721+0000 7fc8db144780 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mds, pid 8
-251> 2021-12-23T03:54:44.721+0000 7fc8db144780 1 main not setting numa affinity
-250> 2021-12-23T03:54:44.721+0000 7fc8db144780 0 pidfile_write: ignore empty --pid-file
-249> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) init /var/run/ceph/ceph-mds.blumfrub.cn06.gauawm.asok
-248> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) bind_and_listen /var/run/ceph/ceph-mds.blumfrub.cn06.gauawm.asok
-247> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command 0 hook 0x55761e862200
-246> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command version hook 0x55761e862200
-245> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command git_version hook 0x55761e862200
-244> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command help hook 0x55761e8005d0
-243> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command get_command_descriptions hook 0x55761e8005e0
-242> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx
-241> 2021-12-23T03:54:44.721+0000 7fc8cb07b700 5 asok(0x55761e85a000) entry start
-240> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx
-239> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx
-238> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: none
-237> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure
-236> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc
-235> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure
-234> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc
-233> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure
-232> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc
-231> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc
-230> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure
-229> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc
-228> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure
-227> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc
-226> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure
-225> 2021-12-23T03:54:44.722+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring
-224> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: build_initial_monmap
-223> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: monmap:
epoch 0
fsid 00000000-0000-0000-0000-000000000000
last_changed 0.000000
created 0.000000
min_mon_release 0 (unknown)
election_strategy: 1
0: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.noname-a
1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.noname-b
2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.noname-c
3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.noname-d
4: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.noname-e
-222> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: init
-221> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding auth protocol: cephx
-220> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding auth protocol: cephx
-219> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding auth protocol: cephx
-218> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding auth protocol: none
-217> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure
-216> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc
-215> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure
-214> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc
-213> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure
-212> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc
-211> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc
-210> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure
-209> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc
-208> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure
-207> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc
-206> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure
-205> 2021-12-23T03:54:44.724+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring
-204> 2021-12-23T03:54:44.724+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring
-203> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: _reopen_session rank -1
-202> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: _add_conns ranks=[2,3,0,4,1]
-201> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-c con 0x55761e8f8c00 addr [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0]
-200> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-d con 0x55761e8f9000 addr [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0]
-199> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-a con 0x55761e8f9400 addr [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0]
-198> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): start opening mon connection
-197> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): start opening mon connection
-196> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): start opening mon connection
-195> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): _renew_subs
-194> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): get_auth_request con 0x55761e8f8c00 auth_method 0
-193> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1]
-192> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): _init_auth method 2
-191> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): _init_auth creating new auth
-190> 2021-12-23T03:54:44.724+0000 7fc8cc87e700 10 monclient(hunting): get_auth_request con 0x55761e8f9400 auth_method 0
-189> 2021-12-23T03:54:44.724+0000 7fc8cc87e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1]
-188> 2021-12-23T03:54:44.724+0000 7fc8cc87e700 10 monclient(hunting): _init_auth method 2
-187> 2021-12-23T03:54:44.724+0000 7fc8cc87e700 10 monclient(hunting): _init_auth creating new auth
-186> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more payload 9
-185> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more payload_len 9
-184> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes
-183> 2021-12-23T03:54:44.724+0000 7fc8cb87c700 10 monclient(hunting): get_auth_request con 0x55761e8f9000 auth_method 0
-182> 2021-12-23T03:54:44.724+0000 7fc8cb87c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1]
-181> 2021-12-23T03:54:44.724+0000 7fc8cb87c700 10 monclient(hunting): _init_auth method 2
-180> 2021-12-23T03:54:44.724+0000 7fc8cb87c700 10 monclient(hunting): _init_auth creating new auth
-179> 2021-12-23T03:54:44.725+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more payload 9
-178> 2021-12-23T03:54:44.725+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more payload_len 9
-177> 2021-12-23T03:54:44.725+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes
-176> 2021-12-23T03:54:44.725+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more payload 9
-175> 2021-12-23T03:54:44.725+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more payload_len 9
-174> 2021-12-23T03:54:44.725+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes
-173> 2021-12-23T03:54:44.725+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_done global_id 10896482 payload 1091
-172> 2021-12-23T03:54:44.725+0000 7fc8cc07d700 10 monclient: _finish_hunting 0
-171> 2021-12-23T03:54:44.725+0000 7fc8cc07d700 1 monclient: found mon.noname-c
-170> 2021-12-23T03:54:44.725+0000 7fc8cc07d700 10 monclient: _send_mon_message to mon.noname-c at v2:192.168.30.13:3300/0
-169> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: handle_monmap mon_map magic: 0 v1
-168> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: got monmap 69 from mon.noname-c (according to old e69)
-167> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: dump:
epoch 69
fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d
last_changed 2021-06-24T11:48:32.597666+0000
created 2021-05-29T06:47:46.644056+0000
min_mon_release 16 (pacific)
election_strategy: 1
0: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.cn05
1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.cn02
2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.cn03
3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.cn04
4: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.cn01
-166> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _finish_auth 0
-165> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2021-12-23T03:54:14.726804+0000)
-164> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0
-163> 2021-12-23T03:54:44.725+0000 7fc8db144780 5 monclient: authenticate success, global_id 10896482
-162> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: handle_config config(4 keys) v1
-161> 2021-12-23T03:54:44.725+0000 7fc8db144780 10 monclient: wait_auth_rotating waiting for 30
-160> 2021-12-23T03:54:44.725+0000 7fc8ca079700 4 set_mon_vals no callback set
-159> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: handle_monmap mon_map magic: 0 v1
-158> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: got monmap 69 from mon.cn03 (according to old e69)
-157> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: dump:
epoch 69
fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d
last_changed 2021-06-24T11:48:32.597666+0000
created 2021-05-29T06:47:46.644056+0000
min_mon_release 16 (pacific)
election_strategy: 1
0: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.cn05
1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.cn02
2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.cn03
3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.cn04
4: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.cn01
-156> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _finish_auth 0
-155> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2021-12-23T03:54:14.727073+0000)
-154> 2021-12-23T03:54:44.725+0000 7fc8db144780 10 monclient: wait_auth_rotating done
-153> 2021-12-23T03:54:44.725+0000 7fc8db144780 10 monclient: _renew_subs
-152> 2021-12-23T03:54:44.725+0000 7fc8db144780 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0
-151> 2021-12-23T03:54:44.725+0000 7fc8db144780 5 asok(0x55761e85a000) register_command status hook 0x55761e8002d0
-150> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_ops_in_flight hook 0x55761e8002d0
-149> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command ops hook 0x55761e8002d0
-148> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_blocked_ops hook 0x55761e8002d0
-147> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_historic_ops hook 0x55761e8002d0
-146> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_historic_ops_by_duration hook 0x55761e8002d0
-145> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub_path hook 0x55761e8002d0
-144> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub start hook 0x55761e8002d0
-143> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub abort hook 0x55761e8002d0
-142> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub pause hook 0x55761e8002d0
-141> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub resume hook 0x55761e8002d0
-140> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub status hook 0x55761e8002d0
-139> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command tag path hook 0x55761e8002d0
-138> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command flush_path hook 0x55761e8002d0
-137> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command export dir hook 0x55761e8002d0
-136> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump cache hook 0x55761e8002d0
-135> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command cache drop hook 0x55761e8002d0
-134> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command cache status hook 0x55761e8002d0
-133> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump tree hook 0x55761e8002d0
-132> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump loads hook 0x55761e8002d0
-131> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump snaps hook 0x55761e8002d0
-130> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session ls hook 0x55761e8002d0
-129> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command client ls hook 0x55761e8002d0
-128> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session evict hook 0x55761e8002d0
-127> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command client evict hook 0x55761e8002d0
-126> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session kill hook 0x55761e8002d0
-125> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session ls hook 0x55761e8002d0
-124> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session config hook 0x55761e8002d0
-123> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command client config hook 0x55761e8002d0
-122> 2021-12-23T03:54:44.726+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Updating MDS map to version 52916 from mon.2
-121> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command damage ls hook 0x55761e8002d0
-120> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command damage rm hook 0x55761e8002d0
-119> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command osdmap barrier hook 0x55761e8002d0
-118> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command flush journal hook 0x55761e8002d0
-117> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command force_readonly hook 0x55761e8002d0
-116> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command get subtrees hook 0x55761e8002d0
-115> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dirfrag split hook 0x55761e8002d0
-114> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dirfrag merge hook 0x55761e8002d0
-113> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dirfrag ls hook 0x55761e8002d0
-112> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command openfiles ls hook 0x55761e8002d0
-111> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump inode hook 0x55761e8002d0
-110> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command exit hook 0x55761e8002d0
-109> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command respawn hook 0x55761e8002d0
-108> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command heap hook 0x55761e8002d0
-107> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command cpu_profiler hook 0x55761e8002d0
-106> 2021-12-23T03:54:44.726+0000 7fc8c7073700 5 mds.beacon.blumfrub.cn06.gauawm Sending beacon up:boot seq 1
-105> 2021-12-23T03:54:44.738+0000 7fc8c7073700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0
-104> 2021-12-23T03:54:45.383+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Updating MDS map to version 52917 from mon.2
-103> 2021-12-23T03:54:45.383+0000 7fc8c9077700 10 monclient: _renew_subs
-102> 2021-12-23T03:54:45.383+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0
-101> 2021-12-23T03:54:45.383+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Monitors have assigned me to become a standby.
-100> 2021-12-23T03:54:45.383+0000 7fc8c9077700 5 mds.beacon.blumfrub.cn06.gauawm set_want_state: up:boot -> up:standby
-99> 2021-12-23T03:54:45.385+0000 7fc8cc07d700 5 mds.beacon.blumfrub.cn06.gauawm received beacon reply up:boot seq 1 rtt 0.659011
-98> 2021-12-23T03:54:45.387+0000 7fc8c9077700 4 mgrc handle_mgr_map Got map version 515
-97> 2021-12-23T03:54:45.387+0000 7fc8c9077700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/3684058018,v1:192.168.30.12:6849/3684058018]
-96> 2021-12-23T03:54:45.387+0000 7fc8c9077700 4 mgrc reconnect Starting new session with [v2:192.168.30.12:6848/3684058018,v1:192.168.30.12:6849/3684058018]
-95> 2021-12-23T03:54:45.387+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x55761e8f9800 auth_method 0
-94> 2021-12-23T03:54:45.389+0000 7fc8c9077700 4 mgrc handle_mgr_configure stats_period=5
-93> 2021-12-23T03:54:45.389+0000 7fc8c9077700 4 mgrc handle_mgr_configure updated stats threshold: 5
-92> 2021-12-23T03:54:45.516+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Updating MDS map to version 52918 from mon.2
-91> 2021-12-23T03:54:45.516+0000 7fc8c9077700 4 mds.0.purge_queue operator(): data pool 19 not found in OSDMap
-90> 2021-12-23T03:54:45.516+0000 7fc8c9077700 5 asok(0x55761e85a000) register_command objecter_requests hook 0x55761e800130
-89> 2021-12-23T03:54:45.517+0000 7fc8c9077700 10 monclient: _renew_subs
-88> 2021-12-23T03:54:45.517+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0
-87> 2021-12-23T03:54:45.517+0000 7fc8c9077700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
-86> 2021-12-23T03:54:45.517+0000 7fc8c9077700 4 mds.0.purge_queue operator(): data pool 19 not found in OSDMap
-85> 2021-12-23T03:54:45.517+0000 7fc8c9077700 4 mds.0.0 handle_osd_map epoch 0, 0 new blocklist entries
-84> 2021-12-23T03:54:45.517+0000 7fc8c9077700 1 mds.0.52918 handle_mds_map i am now mds.0.52918
-83> 2021-12-23T03:54:45.517+0000 7fc8c9077700 1 mds.0.52918 handle_mds_map state change up:boot --> up:replay
-82> 2021-12-23T03:54:45.517+0000 7fc8c9077700 5 mds.beacon.blumfrub.cn06.gauawm set_want_state: up:standby -> up:replay
-81> 2021-12-23T03:54:45.517+0000 7fc8c9077700 1 mds.0.52918 replay_start
-80> 2021-12-23T03:54:45.517+0000 7fc8c9077700 1 mds.0.52918 waiting for osdmap 19523 (which blocklists prior instance)
-79> 2021-12-23T03:54:45.517+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0
-78> 2021-12-23T03:54:45.520+0000 7fc8c9077700 4 mds.0.52918 handle_osd_map epoch 19523, 0 new blocklist entries
-77> 2021-12-23T03:54:45.520+0000 7fc8c9077700 10 monclient: _renew_subs
-76> 2021-12-23T03:54:45.520+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0
-75> 2021-12-23T03:54:45.520+0000 7fc8c9077700 10 monclient: handle_get_version_reply finishing 1 version 19523
-74> 2021-12-23T03:54:45.520+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening inotable
-73> 2021-12-23T03:54:45.520+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening sessionmap
-72> 2021-12-23T03:54:45.520+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x55761f656000 auth_method 0
-71> 2021-12-23T03:54:45.520+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening mds log
-70> 2021-12-23T03:54:45.520+0000 7fc8c306b700 5 mds.0.log open discovering log bounds
-69> 2021-12-23T03:54:45.521+0000 7fc8c286a700 4 mds.0.journalpointer Reading journal pointer '400.00000000'
-68> 2021-12-23T03:54:45.521+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening purge queue (async)
-67> 2021-12-23T03:54:45.521+0000 7fc8c306b700 4 mds.0.purge_queue open: opening
-66> 2021-12-23T03:54:45.521+0000 7fc8c306b700 1 mds.0.journaler.pq(ro) recover start
-65> 2021-12-23T03:54:45.521+0000 7fc8c306b700 1 mds.0.journaler.pq(ro) read_head
-64> 2021-12-23T03:54:45.521+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: loading open file table (async)
-63> 2021-12-23T03:54:45.521+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x55761f656800 auth_method 0
-62> 2021-12-23T03:54:45.521+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening snap table
-61> 2021-12-23T03:54:45.521+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x55761f657800 auth_method 0
-60> 2021-12-23T03:54:45.521+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x55761f657000 auth_method 0
-59> 2021-12-23T03:54:45.521+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x55761f676000 auth_method 0
-58> 2021-12-23T03:54:45.521+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x55761f676800 auth_method 0
-57> 2021-12-23T03:54:45.522+0000 7fc8c286a700 1 mds.0.journaler.mdlog(ro) recover start
-56> 2021-12-23T03:54:45.522+0000 7fc8c286a700 1 mds.0.journaler.mdlog(ro) read_head
-55> 2021-12-23T03:54:45.522+0000 7fc8c286a700 4 mds.0.log Waiting for journal 0x200 to recover...
-54> 2021-12-23T03:54:45.522+0000 7fc8c406d700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 37748736, expire 38447747, write 38447747, stream_format 1). probing for end of log (from 38447747)...
-53> 2021-12-23T03:54:45.522+0000 7fc8c406d700 1 mds.0.journaler.pq(ro) probing for end of the log
-52> 2021-12-23T03:54:45.522+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x55761f656400 auth_method 0
-51> 2021-12-23T03:54:45.522+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x55761f657c00 auth_method 0
-50> 2021-12-23T03:54:45.523+0000 7fc8c406d700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 38447747 (header had 38447747). recovered.
-49> 2021-12-23T03:54:45.523+0000 7fc8c406d700 4 mds.0.purge_queue operator(): open complete
-48> 2021-12-23T03:54:45.523+0000 7fc8c406d700 1 mds.0.journaler.pq(ro) set_writeable
-47> 2021-12-23T03:54:45.537+0000 7fc8c306b700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 4492099584, expire 4492102817, write 4668403448, stream_format 1). probing for end of log (from 4668403448)...
-46> 2021-12-23T03:54:45.537+0000 7fc8c306b700 1 mds.0.journaler.mdlog(ro) probing for end of the log
-45> 2021-12-23T03:54:45.537+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x55761f676400 auth_method 0
-44> 2021-12-23T03:54:45.537+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x55761f677000 auth_method 0
-43> 2021-12-23T03:54:45.538+0000 7fc8c306b700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 4668403564 (header had 4668403448). recovered.
-42> 2021-12-23T03:54:45.538+0000 7fc8c286a700 4 mds.0.log Journal 0x200 recovered.
-41> 2021-12-23T03:54:45.538+0000 7fc8c286a700 4 mds.0.log Recovered journal 0x200 in format 1
-40> 2021-12-23T03:54:45.538+0000 7fc8c286a700 2 mds.0.52918 Booting: 1: loading/discovering base inodes
-39> 2021-12-23T03:54:45.538+0000 7fc8c286a700 0 mds.0.cache creating system inode with ino:0x100
-38> 2021-12-23T03:54:45.538+0000 7fc8c286a700 0 mds.0.cache creating system inode with ino:0x1
-37> 2021-12-23T03:54:45.539+0000 7fc8c306b700 2 mds.0.52918 Booting: 2: replaying mds log
-36> 2021-12-23T03:54:45.539+0000 7fc8c306b700 2 mds.0.52918 Booting: 2: waiting for purge queue recovered
-35> 2021-12-23T03:54:45.539+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x55761f657400 auth_method 0
-34> 2021-12-23T03:54:46.128+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x557621715800 auth_method 0
-33> 2021-12-23T03:54:46.374+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x557620b6f400 auth_method 0
-32> 2021-12-23T03:54:46.917+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x557620b6f000 auth_method 0
-31> 2021-12-23T03:54:46.974+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x557620230400 auth_method 0
-30> 2021-12-23T03:54:47.083+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x557620230800 auth_method 0
-29> 2021-12-23T03:54:47.391+0000 7fc8c1868700 1 mds.0.52918 Finished replaying journal
-28> 2021-12-23T03:54:47.391+0000 7fc8c1868700 1 mds.0.52918 making mds journal writeable
-27> 2021-12-23T03:54:47.391+0000 7fc8c1868700 1 mds.0.journaler.mdlog(ro) set_writeable
-26> 2021-12-23T03:54:47.391+0000 7fc8c1868700 2 mds.0.52918 i am alone, moving to state reconnect
-25> 2021-12-23T03:54:47.391+0000 7fc8c1868700 3 mds.0.52918 request_state up:reconnect
-24> 2021-12-23T03:54:47.391+0000 7fc8c1868700 5 mds.beacon.blumfrub.cn06.gauawm set_want_state: up:replay -> up:reconnect
-23> 2021-12-23T03:54:47.391+0000 7fc8c1868700 5 mds.beacon.blumfrub.cn06.gauawm Sending beacon up:reconnect seq 2
-22> 2021-12-23T03:54:47.391+0000 7fc8c1868700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0
-21> 2021-12-23T03:54:47.392+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x5576205c9c00 auth_method 0
-20> 2021-12-23T03:54:47.724+0000 7fc8c8075700 10 monclient: tick
-19> 2021-12-23T03:54:47.724+0000 7fc8c8075700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2021-12-23T03:54:17.725595+0000)
-18> 2021-12-23T03:54:47.896+0000 7fc8cb87c700 10 monclient: handle_auth_request added challenge on 0x5576205c8400
-17> 2021-12-23T03:54:47.897+0000 7fc8c9077700 3 mds.0.server not active yet, waiting
-16> 2021-12-23T03:54:47.897+0000 7fc8c9077700 3 mds.0.server not active yet, waiting
-15> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Updating MDS map to version 52919 from mon.2
-14> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.52918 handle_mds_map i am now mds.0.52918
-13> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.52918 handle_mds_map state change up:replay --> up:reconnect
-12> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.52918 reconnect_start
-11> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.52918 reopen_log
-10> 2021-12-23T03:54:47.900+0000 7fc8c9077700 4 mds.0.52918 reconnect_start: killed 0 blocklisted sessions (179 blocklist entries, 1)
-9> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.server reconnect_clients -- 1 sessions
-8> 2021-12-23T03:54:47.900+0000 7fc8c9077700 0 log_channel(cluster) log [DBG] : reconnect by client.10266701 v1:192.168.70.10:0/2935197590 after 0
-7> 2021-12-23T03:54:47.902+0000 7fc8c9077700 1 mds.0.52918 reconnect_done
-6> 2021-12-23T03:54:47.902+0000 7fc8c9077700 3 mds.0.52918 request_state up:rejoin
-5> 2021-12-23T03:54:47.902+0000 7fc8c9077700 5 mds.beacon.blumfrub.cn06.gauawm set_want_state: up:reconnect -> up:rejoin
-4> 2021-12-23T03:54:47.902+0000 7fc8c9077700 5 mds.beacon.blumfrub.cn06.gauawm Sending beacon up:rejoin seq 3
-3> 2021-12-23T03:54:47.902+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0
-2> 2021-12-23T03:54:47.902+0000 7fc8cc07d700 5 mds.beacon.blumfrub.cn06.gauawm received beacon reply up:reconnect seq 2 rtt 0.511009
-1> 2021-12-23T03:54:48.708+0000 7fc8c9077700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h: In function 'std::ostream& operator<<(std::ostream&, const ClientMetricType&)' thread 7fc8c9077700 time 2021-12-23T03:54:48.708567+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h: 56: ceph_abort_msg("abort() called")
ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7fc8d1cd8cdc]
2: (operator<<(std::ostream&, ClientMetricType const&)+0x10e) [0x7fc8d1f5d42e]
3: (MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fc8d1f5d601]
4: (DispatchQueue::entry()+0x1be2) [0x7fc8d1f13492]
5: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc8d1fc2b71]
6: /lib64/libpthread.so.0(+0x814a) [0x7fc8d0a7a14a]
7: clone()
0> 2021-12-23T03:54:48.709+0000 7fc8c9077700 -1 *** Caught signal (Aborted) **
in thread 7fc8c9077700 thread_name:ms_dispatch
ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
1: /lib64/libpthread.so.0(+0x12b20) [0x7fc8d0a84b20]
2: gsignal()
3: abort()
4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x7fc8d1cd8dad]
5: (operator<<(std::ostream&, ClientMetricType const&)+0x10e) [0x7fc8d1f5d42e]
6: (MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fc8d1f5d601]
7: (DispatchQueue::entry()+0x1be2) [0x7fc8d1f13492]
8: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc8d1fc2b71]
9: /lib64/libpthread.so.0(+0x814a) [0x7fc8d0a7a14a]
10: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 rbd_pwl
0/ 5 journaler
0/ 5 objectcacher
0/ 5 immutable_obj_cache
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
1/ 5 prioritycache
0/ 5 test
0/ 5 cephfs_mirror
0/ 5 cephsqlite
-2/-2 (syslog threshold)
99/99 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
140500217005824 /
140500233791232 /
140500242183936 / MR_Finisher
140500258969344 / PQ_Finisher
140500309325568 / ceph-mds
140500326110976 / safe_timer
140500342896384 / ms_dispatch
140500359681792 / io_context_pool
140500376467200 / admin_socket
140500384859904 / msgr-worker-2
140500393252608 / msgr-worker-1
140500401645312 / msgr-worker-0
140500645726080 / ceph-mds
max_recent 10000
max_new 10000
log_file /var/lib/ceph/crash/2021-12-23T03:54:48.710208Z_c342e59b-e67a-4f0a-970a-252b5b76ea3b/log
--- end dump of recent events —
META:
{
"crash_id": "2021-12-23T03:54:48.710208Z_c342e59b-e67a-4f0a-970a-252b5b76ea3b",
"timestamp": "2021-12-23T03:54:48.710208Z",
"process_name": "ceph-mds",
"entity_name": "mds.blumfrub.cn06.gauawm",
"ceph_version": "16.2.4",
"utsname_hostname": "cn06.ceph.la1.blumfrub.corp",
"utsname_sysname": "Linux",
"utsname_release": "5.15.8-1.el8.elrepo.x86_64",
"utsname_version": "#1 SMP Mon Dec 13 19:41:26 EST 2021",
"utsname_machine": "x86_64",
"os_name": "CentOS Linux",
"os_id": "centos",
"os_version_id": "8",
"os_version": "8",
"assert_condition": "abort",
"assert_func": "std::ostream& operator<<(std::ostream&, const ClientMetricType&)",
"assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h",
"assert_line": 56,
"assert_thread_name": "ms_dispatch",
"assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h: In function 'std::ostream& operator<<(std::ostream&, const ClientMetricType&)' thread 7fc8c9077700 time 2021-12-23T03:54:48.708567+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h: 56: ceph_abort_msg(\"abort() called\")\n",
"backtrace": [
"/lib64/libpthread.so.0(+0x12b20) [0x7fc8d0a84b20]",
"gsignal()",
"abort()",
"(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x7fc8d1cd8dad]",
"(operator<<(std::ostream&, ClientMetricType const&)+0x10e) [0x7fc8d1f5d42e]",
"(MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fc8d1f5d601]",
"(DispatchQueue::entry()+0x1be2) [0x7fc8d1f13492]",
"(DispatchQueue::DispatchThread::entry()+0x11) [0x7fc8d1fc2b71]",
"/lib64/libpthread.so.0(+0x814a) [0x7fc8d0a7a14a]",
"clone()"
]
}
I have a feeling somehow a version mismatch happened. I noticed upgrades to local packages going to 16.2.5 at some point by blindly doing a yum upgrade. I downgraded the packages but perhaps the damage was already done? I had plans to go to 16.2.7, but I’d like to understand why I’m experiencing these failures.
I jacked up the service count to four for each mds service so just provide a higher level of redundancy until I understand what is going on. I don’t know if that was a bad move but two separate filesystems are having mds failures.
Thanks
-jeremy
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
