So yeah, it’s a mess: cexecs ceph: "cephadm version" ************************ ceph ************************ --------- cn01.ceph.la1.blumfrub.corp--------- Using recent ceph image docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) --------- cn02.ceph.la1.blumfrub.corp--------- Using recent ceph image docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) --------- cn03.ceph.la1.blumfrub.corp--------- Using recent ceph image docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) --------- cn04.ceph.la1.blumfrub.corp--------- Using recent ceph image docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) --------- cn05.ceph.la1.blumfrub.corp--------- Using recent ceph image quay.io/ceph/ceph@sha256:5755c3a5c197ef186b8186212e023565f15b799f1ed411207f2c3fcd4a80ab45 ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) --------- cn06.ceph.la1.blumfrub.corp--------- Using recent ceph image docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) What’s the best next step to fix this? This is the containers but the OS level version for cephadm, ceph-common packages, etc, is all 16.2.4. -jeremy > On Wednesday, Dec 22, 2021 at 8:15 PM, Jeremy Hansen <jeremy@xxxxxxxxxx (mailto:jeremy@xxxxxxxxxx)> wrote: > For some unknown reason, I’m getting mds failures. Running pacific 16.2.4. Crash logs: > > > LOG: > > --- begin dump of recent events --- > -371> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command assert hook 0x55761e800050 > -370> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command abort hook 0x55761e800050 > -369> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command leak_some_memory hook 0x55761e800050 > -368> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perfcounters_dump hook 0x55761e800050 > -367> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command 1 hook 0x55761e800050 > -366> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf dump hook 0x55761e800050 > -365> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perfcounters_schema hook 0x55761e800050 > -364> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf histogram dump hook 0x55761e800050 > -363> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command 2 hook 0x55761e800050 > -362> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf schema hook 0x55761e800050 > -361> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf histogram schema hook 0x55761e800050 > -360> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf reset hook 0x55761e800050 > -359> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config show hook 0x55761e800050 > -358> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config help hook 0x55761e800050 > -357> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config set hook 0x55761e800050 > -356> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config unset hook 0x55761e800050 > -355> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config get hook 0x55761e800050 > -354> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config diff hook 0x55761e800050 > -353> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config diff get hook 0x55761e800050 > -352> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command injectargs hook 0x55761e800050 > -351> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command log flush hook 0x55761e800050 > -350> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command log dump hook 0x55761e800050 > -349> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command log reopen hook 0x55761e800050 > -348> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_mempools hook 0x55761e8c6068 > -347> 2021-12-23T03:54:44.718+0000 7fc8db144780 10 monclient: get_monmap_and_config > -346> 2021-12-23T03:54:44.718+0000 7fc8db144780 10 monclient: build_initial_monmap > -345> 2021-12-23T03:54:44.718+0000 7fc8db144780 10 monclient: monmap: > epoch 0 > fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d > last_changed 2021-12-23T03:54:44.719460+0000 > created 2021-12-23T03:54:44.719460+0000 > min_mon_release 0 (unknown) > election_strategy: 1 > 0: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.noname-a > 1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.noname-b > 2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.noname-c > 3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.noname-d > 4: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.noname-e > > -344> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx > -343> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx > -342> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx > -341> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: none > -340> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > -339> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > -338> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > -337> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > -336> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > -335> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > -334> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > -333> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > -332> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > -331> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > -330> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > -329> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > -328> 2021-12-23T03:54:44.718+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring > -327> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient: init > -326> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding auth protocol: cephx > -325> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding auth protocol: cephx > -324> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding auth protocol: cephx > -323> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding auth protocol: none > -322> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure > -321> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc > -320> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure > -319> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc > -318> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure > -317> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc > -316> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc > -315> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure > -314> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc > -313> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure > -312> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc > -311> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure > -310> 2021-12-23T03:54:44.719+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring > -309> 2021-12-23T03:54:44.719+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring > -308> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient: _reopen_session rank -1 > -307> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient: _add_conns ranks=[1,0,4,2,3] > -306> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-b con 0x55761e8f8800 addr [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] > -305> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-a con 0x55761e8f8c00 addr [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] > -304> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-e con 0x55761e8f9000 addr [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] > -303> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): start opening mon connection > -302> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): start opening mon connection > -301> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): start opening mon connection > -300> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): _renew_subs > -299> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): authenticate will time out at 2021-12-23T03:59:44.720414+0000 > -298> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): get_auth_request con 0x55761e8f8800 auth_method 0 > -297> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] > -296> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): _init_auth method 2 > -295> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): _init_auth creating new auth > -294> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): get_auth_request con 0x55761e8f8c00 auth_method 0 > -293> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] > -292> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): _init_auth method 2 > -291> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): _init_auth creating new auth > -290> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): get_auth_request con 0x55761e8f9000 auth_method 0 > -289> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] > -288> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): _init_auth method 2 > -287> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): _init_auth creating new auth > -286> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more payload 9 > -285> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 > -284> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes > -283> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more payload 9 > -282> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 > -281> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes > -280> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more payload 9 > -279> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 > -278> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes > -277> 2021-12-23T03:54:44.720+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_done global_id 10847251 payload 290 > -276> 2021-12-23T03:54:44.720+0000 7fc8cc07d700 10 monclient: _finish_hunting 0 > -275> 2021-12-23T03:54:44.720+0000 7fc8cc07d700 1 monclient: found mon.noname-b > -274> 2021-12-23T03:54:44.720+0000 7fc8cc07d700 10 monclient: _send_mon_message to mon.noname-b at v2:192.168.30.12:3300/0 > -273> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: handle_monmap mon_map magic: 0 v1 > -272> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: got monmap 69 from mon.noname-b (according to old e69) > -271> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: dump: > epoch 69 > fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d > last_changed 2021-06-24T11:48:32.597666+0000 > created 2021-05-29T06:47:46.644056+0000 > min_mon_release 16 (pacific) > election_strategy: 1 > 0: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.cn05 > 1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.cn02 > 2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.cn03 > 3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.cn04 > 4: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.cn01 > > -270> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _finish_auth 0 > -269> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2021-12-23T03:54:14.721727+0000) > -268> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _send_mon_message to mon.cn02 at v2:192.168.30.12:3300/0 > -267> 2021-12-23T03:54:44.720+0000 7fc8db144780 5 monclient: authenticate success, global_id 10847251 > -266> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: handle_config config(4 keys) v1 > -265> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: handle_monmap mon_map magic: 0 v1 > -264> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: got monmap 69 from mon.cn02 (according to old e69) > -263> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: dump: > epoch 69 > fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d > last_changed 2021-06-24T11:48:32.597666+0000 > created 2021-05-29T06:47:46.644056+0000 > min_mon_release 16 (pacific) > election_strategy: 1 > 0: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.cn05 > 1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.cn02 > 2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.cn03 > 3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.cn04 > 4: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.cn01 > > -262> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 monclient: get_monmap_and_config success > -261> 2021-12-23T03:54:44.720+0000 7fc8db144780 4 set_mon_vals no callback set > -260> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 set_mon_vals container_image = docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 > -259> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 set_mon_vals mds_join_fs = blumfrub > -258> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 set_mon_vals mon_allow_pool_delete = true > -257> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 set_mon_vals rbd_move_to_trash_on_remove = true > -256> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _finish_auth 0 > -255> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2021-12-23T03:54:14.722014+0000) > -254> 2021-12-23T03:54:44.721+0000 7fc8db144780 10 monclient: shutdown > -253> 2021-12-23T03:54:44.721+0000 7fc8db144780 0 set uid:gid to 167:167 (ceph:ceph) > -252> 2021-12-23T03:54:44.721+0000 7fc8db144780 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mds, pid 8 > -251> 2021-12-23T03:54:44.721+0000 7fc8db144780 1 main not setting numa affinity > -250> 2021-12-23T03:54:44.721+0000 7fc8db144780 0 pidfile_write: ignore empty --pid-file > -249> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) init /var/run/ceph/ceph-mds.blumfrub.cn06.gauawm.asok > -248> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) bind_and_listen /var/run/ceph/ceph-mds.blumfrub.cn06.gauawm.asok > -247> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command 0 hook 0x55761e862200 > -246> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command version hook 0x55761e862200 > -245> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command git_version hook 0x55761e862200 > -244> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command help hook 0x55761e8005d0 > -243> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command get_command_descriptions hook 0x55761e8005e0 > -242> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx > -241> 2021-12-23T03:54:44.721+0000 7fc8cb07b700 5 asok(0x55761e85a000) entry start > -240> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx > -239> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx > -238> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: none > -237> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > -236> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > -235> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > -234> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > -233> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > -232> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > -231> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > -230> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > -229> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > -228> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > -227> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > -226> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > -225> 2021-12-23T03:54:44.722+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring > -224> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: build_initial_monmap > -223> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: monmap: > epoch 0 > fsid 00000000-0000-0000-0000-000000000000 > last_changed 0.000000 > created 0.000000 > min_mon_release 0 (unknown) > election_strategy: 1 > 0: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.noname-a > 1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.noname-b > 2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.noname-c > 3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.noname-d > 4: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.noname-e > > -222> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: init > -221> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding auth protocol: cephx > -220> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding auth protocol: cephx > -219> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding auth protocol: cephx > -218> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding auth protocol: none > -217> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure > -216> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc > -215> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure > -214> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc > -213> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure > -212> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc > -211> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc > -210> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure > -209> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc > -208> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure > -207> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc > -206> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure > -205> 2021-12-23T03:54:44.724+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring > -204> 2021-12-23T03:54:44.724+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring > -203> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: _reopen_session rank -1 > -202> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: _add_conns ranks=[2,3,0,4,1] > -201> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-c con 0x55761e8f8c00 addr [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] > -200> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-d con 0x55761e8f9000 addr [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] > -199> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-a con 0x55761e8f9400 addr [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] > -198> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): start opening mon connection > -197> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): start opening mon connection > -196> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): start opening mon connection > -195> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): _renew_subs > -194> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): get_auth_request con 0x55761e8f8c00 auth_method 0 > -193> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] > -192> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): _init_auth method 2 > -191> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): _init_auth creating new auth > -190> 2021-12-23T03:54:44.724+0000 7fc8cc87e700 10 monclient(hunting): get_auth_request con 0x55761e8f9400 auth_method 0 > -189> 2021-12-23T03:54:44.724+0000 7fc8cc87e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] > -188> 2021-12-23T03:54:44.724+0000 7fc8cc87e700 10 monclient(hunting): _init_auth method 2 > -187> 2021-12-23T03:54:44.724+0000 7fc8cc87e700 10 monclient(hunting): _init_auth creating new auth > -186> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more payload 9 > -185> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 > -184> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes > -183> 2021-12-23T03:54:44.724+0000 7fc8cb87c700 10 monclient(hunting): get_auth_request con 0x55761e8f9000 auth_method 0 > -182> 2021-12-23T03:54:44.724+0000 7fc8cb87c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] > -181> 2021-12-23T03:54:44.724+0000 7fc8cb87c700 10 monclient(hunting): _init_auth method 2 > -180> 2021-12-23T03:54:44.724+0000 7fc8cb87c700 10 monclient(hunting): _init_auth creating new auth > -179> 2021-12-23T03:54:44.725+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more payload 9 > -178> 2021-12-23T03:54:44.725+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 > -177> 2021-12-23T03:54:44.725+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes > -176> 2021-12-23T03:54:44.725+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more payload 9 > -175> 2021-12-23T03:54:44.725+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 > -174> 2021-12-23T03:54:44.725+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes > -173> 2021-12-23T03:54:44.725+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_done global_id 10896482 payload 1091 > -172> 2021-12-23T03:54:44.725+0000 7fc8cc07d700 10 monclient: _finish_hunting 0 > -171> 2021-12-23T03:54:44.725+0000 7fc8cc07d700 1 monclient: found mon.noname-c > -170> 2021-12-23T03:54:44.725+0000 7fc8cc07d700 10 monclient: _send_mon_message to mon.noname-c at v2:192.168.30.13:3300/0 > -169> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: handle_monmap mon_map magic: 0 v1 > -168> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: got monmap 69 from mon.noname-c (according to old e69) > -167> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: dump: > epoch 69 > fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d > last_changed 2021-06-24T11:48:32.597666+0000 > created 2021-05-29T06:47:46.644056+0000 > min_mon_release 16 (pacific) > election_strategy: 1 > 0: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.cn05 > 1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.cn02 > 2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.cn03 > 3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.cn04 > 4: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.cn01 > > -166> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _finish_auth 0 > -165> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2021-12-23T03:54:14.726804+0000) > -164> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > -163> 2021-12-23T03:54:44.725+0000 7fc8db144780 5 monclient: authenticate success, global_id 10896482 > -162> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: handle_config config(4 keys) v1 > -161> 2021-12-23T03:54:44.725+0000 7fc8db144780 10 monclient: wait_auth_rotating waiting for 30 > -160> 2021-12-23T03:54:44.725+0000 7fc8ca079700 4 set_mon_vals no callback set > -159> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: handle_monmap mon_map magic: 0 v1 > -158> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: got monmap 69 from mon.cn03 (according to old e69) > -157> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: dump: > epoch 69 > fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d > last_changed 2021-06-24T11:48:32.597666+0000 > created 2021-05-29T06:47:46.644056+0000 > min_mon_release 16 (pacific) > election_strategy: 1 > 0: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.cn05 > 1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.cn02 > 2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.cn03 > 3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.cn04 > 4: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.cn01 > > -156> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _finish_auth 0 > -155> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2021-12-23T03:54:14.727073+0000) > -154> 2021-12-23T03:54:44.725+0000 7fc8db144780 10 monclient: wait_auth_rotating done > -153> 2021-12-23T03:54:44.725+0000 7fc8db144780 10 monclient: _renew_subs > -152> 2021-12-23T03:54:44.725+0000 7fc8db144780 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > -151> 2021-12-23T03:54:44.725+0000 7fc8db144780 5 asok(0x55761e85a000) register_command status hook 0x55761e8002d0 > -150> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_ops_in_flight hook 0x55761e8002d0 > -149> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command ops hook 0x55761e8002d0 > -148> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_blocked_ops hook 0x55761e8002d0 > -147> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_historic_ops hook 0x55761e8002d0 > -146> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_historic_ops_by_duration hook 0x55761e8002d0 > -145> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub_path hook 0x55761e8002d0 > -144> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub start hook 0x55761e8002d0 > -143> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub abort hook 0x55761e8002d0 > -142> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub pause hook 0x55761e8002d0 > -141> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub resume hook 0x55761e8002d0 > -140> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub status hook 0x55761e8002d0 > -139> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command tag path hook 0x55761e8002d0 > -138> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command flush_path hook 0x55761e8002d0 > -137> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command export dir hook 0x55761e8002d0 > -136> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump cache hook 0x55761e8002d0 > -135> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command cache drop hook 0x55761e8002d0 > -134> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command cache status hook 0x55761e8002d0 > -133> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump tree hook 0x55761e8002d0 > -132> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump loads hook 0x55761e8002d0 > -131> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump snaps hook 0x55761e8002d0 > -130> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session ls hook 0x55761e8002d0 > -129> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command client ls hook 0x55761e8002d0 > -128> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session evict hook 0x55761e8002d0 > -127> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command client evict hook 0x55761e8002d0 > -126> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session kill hook 0x55761e8002d0 > -125> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session ls hook 0x55761e8002d0 > -124> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session config hook 0x55761e8002d0 > -123> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command client config hook 0x55761e8002d0 > -122> 2021-12-23T03:54:44.726+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Updating MDS map to version 52916 from mon.2 > -121> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command damage ls hook 0x55761e8002d0 > -120> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command damage rm hook 0x55761e8002d0 > -119> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command osdmap barrier hook 0x55761e8002d0 > -118> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command flush journal hook 0x55761e8002d0 > -117> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command force_readonly hook 0x55761e8002d0 > -116> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command get subtrees hook 0x55761e8002d0 > -115> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dirfrag split hook 0x55761e8002d0 > -114> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dirfrag merge hook 0x55761e8002d0 > -113> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dirfrag ls hook 0x55761e8002d0 > -112> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command openfiles ls hook 0x55761e8002d0 > -111> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump inode hook 0x55761e8002d0 > -110> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command exit hook 0x55761e8002d0 > -109> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command respawn hook 0x55761e8002d0 > -108> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command heap hook 0x55761e8002d0 > -107> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command cpu_profiler hook 0x55761e8002d0 > -106> 2021-12-23T03:54:44.726+0000 7fc8c7073700 5 mds.beacon.blumfrub.cn06.gauawm Sending beacon up:boot seq 1 > -105> 2021-12-23T03:54:44.738+0000 7fc8c7073700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > -104> 2021-12-23T03:54:45.383+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Updating MDS map to version 52917 from mon.2 > -103> 2021-12-23T03:54:45.383+0000 7fc8c9077700 10 monclient: _renew_subs > -102> 2021-12-23T03:54:45.383+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > -101> 2021-12-23T03:54:45.383+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Monitors have assigned me to become a standby. > -100> 2021-12-23T03:54:45.383+0000 7fc8c9077700 5 mds.beacon.blumfrub.cn06.gauawm set_want_state: up:boot -> up:standby > -99> 2021-12-23T03:54:45.385+0000 7fc8cc07d700 5 mds.beacon.blumfrub.cn06.gauawm received beacon reply up:boot seq 1 rtt 0.659011 > -98> 2021-12-23T03:54:45.387+0000 7fc8c9077700 4 mgrc handle_mgr_map Got map version 515 > -97> 2021-12-23T03:54:45.387+0000 7fc8c9077700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/3684058018,v1:192.168.30.12:6849/3684058018] > -96> 2021-12-23T03:54:45.387+0000 7fc8c9077700 4 mgrc reconnect Starting new session with [v2:192.168.30.12:6848/3684058018,v1:192.168.30.12:6849/3684058018] > -95> 2021-12-23T03:54:45.387+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x55761e8f9800 auth_method 0 > -94> 2021-12-23T03:54:45.389+0000 7fc8c9077700 4 mgrc handle_mgr_configure stats_period=5 > -93> 2021-12-23T03:54:45.389+0000 7fc8c9077700 4 mgrc handle_mgr_configure updated stats threshold: 5 > -92> 2021-12-23T03:54:45.516+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Updating MDS map to version 52918 from mon.2 > -91> 2021-12-23T03:54:45.516+0000 7fc8c9077700 4 mds.0.purge_queue operator(): data pool 19 not found in OSDMap > -90> 2021-12-23T03:54:45.516+0000 7fc8c9077700 5 asok(0x55761e85a000) register_command objecter_requests hook 0x55761e800130 > -89> 2021-12-23T03:54:45.517+0000 7fc8c9077700 10 monclient: _renew_subs > -88> 2021-12-23T03:54:45.517+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > -87> 2021-12-23T03:54:45.517+0000 7fc8c9077700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) > -86> 2021-12-23T03:54:45.517+0000 7fc8c9077700 4 mds.0.purge_queue operator(): data pool 19 not found in OSDMap > -85> 2021-12-23T03:54:45.517+0000 7fc8c9077700 4 mds.0.0 handle_osd_map epoch 0, 0 new blocklist entries > -84> 2021-12-23T03:54:45.517+0000 7fc8c9077700 1 mds.0.52918 handle_mds_map i am now mds.0.52918 > -83> 2021-12-23T03:54:45.517+0000 7fc8c9077700 1 mds.0.52918 handle_mds_map state change up:boot --> up:replay > -82> 2021-12-23T03:54:45.517+0000 7fc8c9077700 5 mds.beacon.blumfrub.cn06.gauawm set_want_state: up:standby -> up:replay > -81> 2021-12-23T03:54:45.517+0000 7fc8c9077700 1 mds.0.52918 replay_start > -80> 2021-12-23T03:54:45.517+0000 7fc8c9077700 1 mds.0.52918 waiting for osdmap 19523 (which blocklists prior instance) > -79> 2021-12-23T03:54:45.517+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > -78> 2021-12-23T03:54:45.520+0000 7fc8c9077700 4 mds.0.52918 handle_osd_map epoch 19523, 0 new blocklist entries > -77> 2021-12-23T03:54:45.520+0000 7fc8c9077700 10 monclient: _renew_subs > -76> 2021-12-23T03:54:45.520+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > -75> 2021-12-23T03:54:45.520+0000 7fc8c9077700 10 monclient: handle_get_version_reply finishing 1 version 19523 > -74> 2021-12-23T03:54:45.520+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening inotable > -73> 2021-12-23T03:54:45.520+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening sessionmap > -72> 2021-12-23T03:54:45.520+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x55761f656000 auth_method 0 > -71> 2021-12-23T03:54:45.520+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening mds log > -70> 2021-12-23T03:54:45.520+0000 7fc8c306b700 5 mds.0.log open discovering log bounds > -69> 2021-12-23T03:54:45.521+0000 7fc8c286a700 4 mds.0.journalpointer Reading journal pointer '400.00000000' > -68> 2021-12-23T03:54:45.521+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening purge queue (async) > -67> 2021-12-23T03:54:45.521+0000 7fc8c306b700 4 mds.0.purge_queue open: opening > -66> 2021-12-23T03:54:45.521+0000 7fc8c306b700 1 mds.0.journaler.pq(ro) recover start > -65> 2021-12-23T03:54:45.521+0000 7fc8c306b700 1 mds.0.journaler.pq(ro) read_head > -64> 2021-12-23T03:54:45.521+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: loading open file table (async) > -63> 2021-12-23T03:54:45.521+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x55761f656800 auth_method 0 > -62> 2021-12-23T03:54:45.521+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening snap table > -61> 2021-12-23T03:54:45.521+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x55761f657800 auth_method 0 > -60> 2021-12-23T03:54:45.521+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x55761f657000 auth_method 0 > -59> 2021-12-23T03:54:45.521+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x55761f676000 auth_method 0 > -58> 2021-12-23T03:54:45.521+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x55761f676800 auth_method 0 > -57> 2021-12-23T03:54:45.522+0000 7fc8c286a700 1 mds.0.journaler.mdlog(ro) recover start > -56> 2021-12-23T03:54:45.522+0000 7fc8c286a700 1 mds.0.journaler.mdlog(ro) read_head > -55> 2021-12-23T03:54:45.522+0000 7fc8c286a700 4 mds.0.log Waiting for journal 0x200 to recover... > -54> 2021-12-23T03:54:45.522+0000 7fc8c406d700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 37748736, expire 38447747, write 38447747, stream_format 1). probing for end of log (from 38447747)... > -53> 2021-12-23T03:54:45.522+0000 7fc8c406d700 1 mds.0.journaler.pq(ro) probing for end of the log > -52> 2021-12-23T03:54:45.522+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x55761f656400 auth_method 0 > -51> 2021-12-23T03:54:45.522+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x55761f657c00 auth_method 0 > -50> 2021-12-23T03:54:45.523+0000 7fc8c406d700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 38447747 (header had 38447747). recovered. > -49> 2021-12-23T03:54:45.523+0000 7fc8c406d700 4 mds.0.purge_queue operator(): open complete > -48> 2021-12-23T03:54:45.523+0000 7fc8c406d700 1 mds.0.journaler.pq(ro) set_writeable > -47> 2021-12-23T03:54:45.537+0000 7fc8c306b700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 4492099584, expire 4492102817, write 4668403448, stream_format 1). probing for end of log (from 4668403448)... > -46> 2021-12-23T03:54:45.537+0000 7fc8c306b700 1 mds.0.journaler.mdlog(ro) probing for end of the log > -45> 2021-12-23T03:54:45.537+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x55761f676400 auth_method 0 > -44> 2021-12-23T03:54:45.537+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x55761f677000 auth_method 0 > -43> 2021-12-23T03:54:45.538+0000 7fc8c306b700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 4668403564 (header had 4668403448). recovered. > -42> 2021-12-23T03:54:45.538+0000 7fc8c286a700 4 mds.0.log Journal 0x200 recovered. > -41> 2021-12-23T03:54:45.538+0000 7fc8c286a700 4 mds.0.log Recovered journal 0x200 in format 1 > -40> 2021-12-23T03:54:45.538+0000 7fc8c286a700 2 mds.0.52918 Booting: 1: loading/discovering base inodes > -39> 2021-12-23T03:54:45.538+0000 7fc8c286a700 0 mds.0.cache creating system inode with ino:0x100 > -38> 2021-12-23T03:54:45.538+0000 7fc8c286a700 0 mds.0.cache creating system inode with ino:0x1 > -37> 2021-12-23T03:54:45.539+0000 7fc8c306b700 2 mds.0.52918 Booting: 2: replaying mds log > -36> 2021-12-23T03:54:45.539+0000 7fc8c306b700 2 mds.0.52918 Booting: 2: waiting for purge queue recovered > -35> 2021-12-23T03:54:45.539+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x55761f657400 auth_method 0 > -34> 2021-12-23T03:54:46.128+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x557621715800 auth_method 0 > -33> 2021-12-23T03:54:46.374+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x557620b6f400 auth_method 0 > -32> 2021-12-23T03:54:46.917+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x557620b6f000 auth_method 0 > -31> 2021-12-23T03:54:46.974+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x557620230400 auth_method 0 > -30> 2021-12-23T03:54:47.083+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x557620230800 auth_method 0 > -29> 2021-12-23T03:54:47.391+0000 7fc8c1868700 1 mds.0.52918 Finished replaying journal > -28> 2021-12-23T03:54:47.391+0000 7fc8c1868700 1 mds.0.52918 making mds journal writeable > -27> 2021-12-23T03:54:47.391+0000 7fc8c1868700 1 mds.0.journaler.mdlog(ro) set_writeable > -26> 2021-12-23T03:54:47.391+0000 7fc8c1868700 2 mds.0.52918 i am alone, moving to state reconnect > -25> 2021-12-23T03:54:47.391+0000 7fc8c1868700 3 mds.0.52918 request_state up:reconnect > -24> 2021-12-23T03:54:47.391+0000 7fc8c1868700 5 mds.beacon.blumfrub.cn06.gauawm set_want_state: up:replay -> up:reconnect > -23> 2021-12-23T03:54:47.391+0000 7fc8c1868700 5 mds.beacon.blumfrub.cn06.gauawm Sending beacon up:reconnect seq 2 > -22> 2021-12-23T03:54:47.391+0000 7fc8c1868700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > -21> 2021-12-23T03:54:47.392+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x5576205c9c00 auth_method 0 > -20> 2021-12-23T03:54:47.724+0000 7fc8c8075700 10 monclient: tick > -19> 2021-12-23T03:54:47.724+0000 7fc8c8075700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2021-12-23T03:54:17.725595+0000) > -18> 2021-12-23T03:54:47.896+0000 7fc8cb87c700 10 monclient: handle_auth_request added challenge on 0x5576205c8400 > -17> 2021-12-23T03:54:47.897+0000 7fc8c9077700 3 mds.0.server not active yet, waiting > -16> 2021-12-23T03:54:47.897+0000 7fc8c9077700 3 mds.0.server not active yet, waiting > -15> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Updating MDS map to version 52919 from mon.2 > -14> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.52918 handle_mds_map i am now mds.0.52918 > -13> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.52918 handle_mds_map state change up:replay --> up:reconnect > -12> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.52918 reconnect_start > -11> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.52918 reopen_log > -10> 2021-12-23T03:54:47.900+0000 7fc8c9077700 4 mds.0.52918 reconnect_start: killed 0 blocklisted sessions (179 blocklist entries, 1) > -9> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.server reconnect_clients -- 1 sessions > -8> 2021-12-23T03:54:47.900+0000 7fc8c9077700 0 log_channel(cluster) log [DBG] : reconnect by client.10266701 v1:192.168.70.10:0/2935197590 after 0 > -7> 2021-12-23T03:54:47.902+0000 7fc8c9077700 1 mds.0.52918 reconnect_done > -6> 2021-12-23T03:54:47.902+0000 7fc8c9077700 3 mds.0.52918 request_state up:rejoin > -5> 2021-12-23T03:54:47.902+0000 7fc8c9077700 5 mds.beacon.blumfrub.cn06.gauawm set_want_state: up:reconnect -> up:rejoin > -4> 2021-12-23T03:54:47.902+0000 7fc8c9077700 5 mds.beacon.blumfrub.cn06.gauawm Sending beacon up:rejoin seq 3 > -3> 2021-12-23T03:54:47.902+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > -2> 2021-12-23T03:54:47.902+0000 7fc8cc07d700 5 mds.beacon.blumfrub.cn06.gauawm received beacon reply up:reconnect seq 2 rtt 0.511009 > -1> 2021-12-23T03:54:48.708+0000 7fc8c9077700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h: In function 'std::ostream& operator<<(std::ostream&, const ClientMetricType&)' thread 7fc8c9077700 time 2021-12-23T03:54:48.708567+0000 > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h: 56: ceph_abort_msg("abort() called") > > ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) > 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7fc8d1cd8cdc] > 2: (operator<<(std::ostream&, ClientMetricType const&)+0x10e) [0x7fc8d1f5d42e] > 3: (MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fc8d1f5d601] > 4: (DispatchQueue::entry()+0x1be2) [0x7fc8d1f13492] > 5: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc8d1fc2b71] > 6: /lib64/libpthread.so.0(+0x814a) [0x7fc8d0a7a14a] > 7: clone() > > 0> 2021-12-23T03:54:48.709+0000 7fc8c9077700 -1 *** Caught signal (Aborted) ** > in thread 7fc8c9077700 thread_name:ms_dispatch > > ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) > 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc8d0a84b20] > 2: gsignal() > 3: abort() > 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x7fc8d1cd8dad] > 5: (operator<<(std::ostream&, ClientMetricType const&)+0x10e) [0x7fc8d1f5d42e] > 6: (MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fc8d1f5d601] > 7: (DispatchQueue::entry()+0x1be2) [0x7fc8d1f13492] > 8: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc8d1fc2b71] > 9: /lib64/libpthread.so.0(+0x814a) [0x7fc8d0a7a14a] > 10: clone() > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 rbd_mirror > 0/ 5 rbd_replay > 0/ 5 rbd_pwl > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 immutable_obj_cache > 0/ 5 client > 1/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 0 ms > 1/ 5 mon > 0/10 monc > 1/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 1 reserver > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/ 5 rgw_sync > 1/10 civetweb > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > 0/ 0 refs > 1/ 5 compressor > 1/ 5 bluestore > 1/ 5 bluefs > 1/ 3 bdev > 1/ 5 kstore > 4/ 5 rocksdb > 4/ 5 leveldb > 4/ 5 memdb > 1/ 5 fuse > 1/ 5 mgr > 1/ 5 mgrc > 1/ 5 dpdk > 1/ 5 eventtrace > 1/ 5 prioritycache > 0/ 5 test > 0/ 5 cephfs_mirror > 0/ 5 cephsqlite > -2/-2 (syslog threshold) > 99/99 (stderr threshold) > --- pthread ID / name mapping for recent threads --- > 140500217005824 / > 140500233791232 / > 140500242183936 / MR_Finisher > 140500258969344 / PQ_Finisher > 140500309325568 / ceph-mds > 140500326110976 / safe_timer > 140500342896384 / ms_dispatch > 140500359681792 / io_context_pool > 140500376467200 / admin_socket > 140500384859904 / msgr-worker-2 > 140500393252608 / msgr-worker-1 > 140500401645312 / msgr-worker-0 > 140500645726080 / ceph-mds > max_recent 10000 > max_new 10000 > log_file /var/lib/ceph/crash/2021-12-23T03:54:48.710208Z_c342e59b-e67a-4f0a-970a-252b5b76ea3b/log > --- end dump of recent events — > > > META: > > { > "crash_id": "2021-12-23T03:54:48.710208Z_c342e59b-e67a-4f0a-970a-252b5b76ea3b", > "timestamp": "2021-12-23T03:54:48.710208Z", > "process_name": "ceph-mds", > "entity_name": "mds.blumfrub.cn06.gauawm", > "ceph_version": "16.2.4", > "utsname_hostname": "cn06.ceph.la1.blumfrub.corp", > "utsname_sysname": "Linux", > "utsname_release": "5.15.8-1.el8.elrepo.x86_64", > "utsname_version": "#1 SMP Mon Dec 13 19:41:26 EST 2021", > "utsname_machine": "x86_64", > "os_name": "CentOS Linux", > "os_id": "centos", > "os_version_id": "8", > "os_version": "8", > "assert_condition": "abort", > "assert_func": "std::ostream& operator<<(std::ostream&, const ClientMetricType&)", > "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h", > "assert_line": 56, > "assert_thread_name": "ms_dispatch", > "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h: In function 'std::ostream& operator<<(std::ostream&, const ClientMetricType&)' thread 7fc8c9077700 time 2021-12-23T03:54:48.708567+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h: 56: ceph_abort_msg(\"abort() called\")\n", > "backtrace": [ > "/lib64/libpthread.so.0(+0x12b20) [0x7fc8d0a84b20]", > "gsignal()", > "abort()", > "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x7fc8d1cd8dad]", > "(operator<<(std::ostream&, ClientMetricType const&)+0x10e) [0x7fc8d1f5d42e]", > "(MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fc8d1f5d601]", > "(DispatchQueue::entry()+0x1be2) [0x7fc8d1f13492]", > "(DispatchQueue::DispatchThread::entry()+0x11) [0x7fc8d1fc2b71]", > "/lib64/libpthread.so.0(+0x814a) [0x7fc8d0a7a14a]", > "clone()" > ] > } > > > > > I have a feeling somehow a version mismatch happened. I noticed upgrades to local packages going to 16.2.5 at some point by blindly doing a yum upgrade. I downgraded the packages but perhaps the damage was already done? I had plans to go to 16.2.7, but I’d like to understand why I’m experiencing these failures. > > I jacked up the service count to four for each mds service so just provide a higher level of redundancy until I understand what is going on. I don’t know if that was a bad move but two separate filesystems are having mds failures. > > Thanks > -jeremy >
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
