Downgraded via the orchestrator and all is good. -jeremy > On Wednesday, Dec 22, 2021 at 8:33 PM, Jeremy Hansen <jeremy@xxxxxxxxxx (mailto:jeremy@xxxxxxxxxx)> wrote: > So yeah, it’s a mess: > > cexecs ceph: "cephadm version" > ************************ ceph ************************ > --------- cn01.ceph.la1.blumfrub.corp--------- > Using recent ceph image docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 > ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) > --------- cn02.ceph.la1.blumfrub.corp--------- > Using recent ceph image docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb > ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) > --------- cn03.ceph.la1.blumfrub.corp--------- > Using recent ceph image docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 > ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) > --------- cn04.ceph.la1.blumfrub.corp--------- > Using recent ceph image docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 > ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) > --------- cn05.ceph.la1.blumfrub.corp--------- > Using recent ceph image quay.io/ceph/ceph@sha256:5755c3a5c197ef186b8186212e023565f15b799f1ed411207f2c3fcd4a80ab45 > ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) > --------- cn06.ceph.la1.blumfrub.corp--------- > Using recent ceph image docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb > ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) > > > What’s the best next step to fix this? This is the containers but the OS level version for cephadm, ceph-common packages, etc, is all 16.2.4. > > -jeremy > > > > > On Wednesday, Dec 22, 2021 at 8:15 PM, Jeremy Hansen <jeremy@xxxxxxxxxx (mailto:jeremy@xxxxxxxxxx)> wrote: > > For some unknown reason, I’m getting mds failures. Running pacific 16.2.4. Crash logs: > > > > > > LOG: > > > > --- begin dump of recent events --- > > -371> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command assert hook 0x55761e800050 > > -370> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command abort hook 0x55761e800050 > > -369> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command leak_some_memory hook 0x55761e800050 > > -368> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perfcounters_dump hook 0x55761e800050 > > -367> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command 1 hook 0x55761e800050 > > -366> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf dump hook 0x55761e800050 > > -365> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perfcounters_schema hook 0x55761e800050 > > -364> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf histogram dump hook 0x55761e800050 > > -363> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command 2 hook 0x55761e800050 > > -362> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf schema hook 0x55761e800050 > > -361> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf histogram schema hook 0x55761e800050 > > -360> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command perf reset hook 0x55761e800050 > > -359> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config show hook 0x55761e800050 > > -358> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config help hook 0x55761e800050 > > -357> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config set hook 0x55761e800050 > > -356> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config unset hook 0x55761e800050 > > -355> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config get hook 0x55761e800050 > > -354> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config diff hook 0x55761e800050 > > -353> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command config diff get hook 0x55761e800050 > > -352> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command injectargs hook 0x55761e800050 > > -351> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command log flush hook 0x55761e800050 > > -350> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command log dump hook 0x55761e800050 > > -349> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command log reopen hook 0x55761e800050 > > -348> 2021-12-23T03:54:44.712+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_mempools hook 0x55761e8c6068 > > -347> 2021-12-23T03:54:44.718+0000 7fc8db144780 10 monclient: get_monmap_and_config > > -346> 2021-12-23T03:54:44.718+0000 7fc8db144780 10 monclient: build_initial_monmap > > -345> 2021-12-23T03:54:44.718+0000 7fc8db144780 10 monclient: monmap: > > epoch 0 > > fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d > > last_changed 2021-12-23T03:54:44.719460+0000 > > created 2021-12-23T03:54:44.719460+0000 > > min_mon_release 0 (unknown) > > election_strategy: 1 > > 0: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.noname-a > > 1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.noname-b > > 2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.noname-c > > 3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.noname-d > > 4: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.noname-e > > > > -344> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx > > -343> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx > > -342> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx > > -341> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: none > > -340> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > > -339> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > > -338> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > > -337> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > > -336> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > > -335> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > > -334> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > > -333> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > > -332> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > > -331> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > > -330> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > > -329> 2021-12-23T03:54:44.718+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > > -328> 2021-12-23T03:54:44.718+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring > > -327> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient: init > > -326> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding auth protocol: cephx > > -325> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding auth protocol: cephx > > -324> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding auth protocol: cephx > > -323> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding auth protocol: none > > -322> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure > > -321> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc > > -320> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure > > -319> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc > > -318> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure > > -317> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc > > -316> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc > > -315> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure > > -314> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc > > -313> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure > > -312> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: crc > > -311> 2021-12-23T03:54:44.719+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e81bb0) adding con mode: secure > > -310> 2021-12-23T03:54:44.719+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring > > -309> 2021-12-23T03:54:44.719+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring > > -308> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient: _reopen_session rank -1 > > -307> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient: _add_conns ranks=[1,0,4,2,3] > > -306> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-b con 0x55761e8f8800 addr [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] > > -305> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-a con 0x55761e8f8c00 addr [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] > > -304> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-e con 0x55761e8f9000 addr [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] > > -303> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): start opening mon connection > > -302> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): start opening mon connection > > -301> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): start opening mon connection > > -300> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): _renew_subs > > -299> 2021-12-23T03:54:44.719+0000 7fc8db144780 10 monclient(hunting): authenticate will time out at 2021-12-23T03:59:44.720414+0000 > > -298> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): get_auth_request con 0x55761e8f8800 auth_method 0 > > -297> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] > > -296> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): _init_auth method 2 > > -295> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): _init_auth creating new auth > > -294> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): get_auth_request con 0x55761e8f8c00 auth_method 0 > > -293> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] > > -292> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): _init_auth method 2 > > -291> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): _init_auth creating new auth > > -290> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): get_auth_request con 0x55761e8f9000 auth_method 0 > > -289> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] > > -288> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): _init_auth method 2 > > -287> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): _init_auth creating new auth > > -286> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more payload 9 > > -285> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 > > -284> 2021-12-23T03:54:44.719+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes > > -283> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more payload 9 > > -282> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 > > -281> 2021-12-23T03:54:44.719+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes > > -280> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more payload 9 > > -279> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 > > -278> 2021-12-23T03:54:44.719+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes > > -277> 2021-12-23T03:54:44.720+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_done global_id 10847251 payload 290 > > -276> 2021-12-23T03:54:44.720+0000 7fc8cc07d700 10 monclient: _finish_hunting 0 > > -275> 2021-12-23T03:54:44.720+0000 7fc8cc07d700 1 monclient: found mon.noname-b > > -274> 2021-12-23T03:54:44.720+0000 7fc8cc07d700 10 monclient: _send_mon_message to mon.noname-b at v2:192.168.30.12:3300/0 > > -273> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: handle_monmap mon_map magic: 0 v1 > > -272> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: got monmap 69 from mon.noname-b (according to old e69) > > -271> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: dump: > > epoch 69 > > fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d > > last_changed 2021-06-24T11:48:32.597666+0000 > > created 2021-05-29T06:47:46.644056+0000 > > min_mon_release 16 (pacific) > > election_strategy: 1 > > 0: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.cn05 > > 1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.cn02 > > 2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.cn03 > > 3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.cn04 > > 4: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.cn01 > > > > -270> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _finish_auth 0 > > -269> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2021-12-23T03:54:14.721727+0000) > > -268> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _send_mon_message to mon.cn02 at v2:192.168.30.12:3300/0 > > -267> 2021-12-23T03:54:44.720+0000 7fc8db144780 5 monclient: authenticate success, global_id 10847251 > > -266> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: handle_config config(4 keys) v1 > > -265> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: handle_monmap mon_map magic: 0 v1 > > -264> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: got monmap 69 from mon.cn02 (according to old e69) > > -263> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: dump: > > epoch 69 > > fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d > > last_changed 2021-06-24T11:48:32.597666+0000 > > created 2021-05-29T06:47:46.644056+0000 > > min_mon_release 16 (pacific) > > election_strategy: 1 > > 0: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.cn05 > > 1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.cn02 > > 2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.cn03 > > 3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.cn04 > > 4: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.cn01 > > > > -262> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 monclient: get_monmap_and_config success > > -261> 2021-12-23T03:54:44.720+0000 7fc8db144780 4 set_mon_vals no callback set > > -260> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 set_mon_vals container_image = docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 > > -259> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 set_mon_vals mds_join_fs = blumfrub > > -258> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 set_mon_vals mon_allow_pool_delete = true > > -257> 2021-12-23T03:54:44.720+0000 7fc8db144780 10 set_mon_vals rbd_move_to_trash_on_remove = true > > -256> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _finish_auth 0 > > -255> 2021-12-23T03:54:44.720+0000 7fc8cb07b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2021-12-23T03:54:14.722014+0000) > > -254> 2021-12-23T03:54:44.721+0000 7fc8db144780 10 monclient: shutdown > > -253> 2021-12-23T03:54:44.721+0000 7fc8db144780 0 set uid:gid to 167:167 (ceph:ceph) > > -252> 2021-12-23T03:54:44.721+0000 7fc8db144780 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mds, pid 8 > > -251> 2021-12-23T03:54:44.721+0000 7fc8db144780 1 main not setting numa affinity > > -250> 2021-12-23T03:54:44.721+0000 7fc8db144780 0 pidfile_write: ignore empty --pid-file > > -249> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) init /var/run/ceph/ceph-mds.blumfrub.cn06.gauawm.asok > > -248> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) bind_and_listen /var/run/ceph/ceph-mds.blumfrub.cn06.gauawm.asok > > -247> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command 0 hook 0x55761e862200 > > -246> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command version hook 0x55761e862200 > > -245> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command git_version hook 0x55761e862200 > > -244> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command help hook 0x55761e8005d0 > > -243> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 asok(0x55761e85a000) register_command get_command_descriptions hook 0x55761e8005e0 > > -242> 2021-12-23T03:54:44.721+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx > > -241> 2021-12-23T03:54:44.721+0000 7fc8cb07b700 5 asok(0x55761e85a000) entry start > > -240> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx > > -239> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: cephx > > -238> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding auth protocol: none > > -237> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > > -236> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > > -235> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > > -234> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > > -233> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > > -232> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > > -231> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > > -230> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > > -229> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > > -228> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > > -227> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: crc > > -226> 2021-12-23T03:54:44.722+0000 7fc8db144780 5 AuthRegistry(0x55761e8a2a40) adding con mode: secure > > -225> 2021-12-23T03:54:44.722+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring > > -224> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: build_initial_monmap > > -223> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: monmap: > > epoch 0 > > fsid 00000000-0000-0000-0000-000000000000 > > last_changed 0.000000 > > created 0.000000 > > min_mon_release 0 (unknown) > > election_strategy: 1 > > 0: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.noname-a > > 1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.noname-b > > 2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.noname-c > > 3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.noname-d > > 4: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.noname-e > > > > -222> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: init > > -221> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding auth protocol: cephx > > -220> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding auth protocol: cephx > > -219> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding auth protocol: cephx > > -218> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding auth protocol: none > > -217> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure > > -216> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc > > -215> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure > > -214> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc > > -213> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure > > -212> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc > > -211> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc > > -210> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure > > -209> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc > > -208> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure > > -207> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: crc > > -206> 2021-12-23T03:54:44.724+0000 7fc8db144780 5 AuthRegistry(0x7ffd65e835b0) adding con mode: secure > > -205> 2021-12-23T03:54:44.724+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring > > -204> 2021-12-23T03:54:44.724+0000 7fc8db144780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-blumfrub.cn06.gauawm/keyring > > -203> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: _reopen_session rank -1 > > -202> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient: _add_conns ranks=[2,3,0,4,1] > > -201> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-c con 0x55761e8f8c00 addr [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] > > -200> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-d con 0x55761e8f9000 addr [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] > > -199> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): picked mon.noname-a con 0x55761e8f9400 addr [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] > > -198> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): start opening mon connection > > -197> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): start opening mon connection > > -196> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): start opening mon connection > > -195> 2021-12-23T03:54:44.724+0000 7fc8db144780 10 monclient(hunting): _renew_subs > > -194> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): get_auth_request con 0x55761e8f8c00 auth_method 0 > > -193> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] > > -192> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): _init_auth method 2 > > -191> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): _init_auth creating new auth > > -190> 2021-12-23T03:54:44.724+0000 7fc8cc87e700 10 monclient(hunting): get_auth_request con 0x55761e8f9400 auth_method 0 > > -189> 2021-12-23T03:54:44.724+0000 7fc8cc87e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] > > -188> 2021-12-23T03:54:44.724+0000 7fc8cc87e700 10 monclient(hunting): _init_auth method 2 > > -187> 2021-12-23T03:54:44.724+0000 7fc8cc87e700 10 monclient(hunting): _init_auth creating new auth > > -186> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more payload 9 > > -185> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 > > -184> 2021-12-23T03:54:44.724+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes > > -183> 2021-12-23T03:54:44.724+0000 7fc8cb87c700 10 monclient(hunting): get_auth_request con 0x55761e8f9000 auth_method 0 > > -182> 2021-12-23T03:54:44.724+0000 7fc8cb87c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] > > -181> 2021-12-23T03:54:44.724+0000 7fc8cb87c700 10 monclient(hunting): _init_auth method 2 > > -180> 2021-12-23T03:54:44.724+0000 7fc8cb87c700 10 monclient(hunting): _init_auth creating new auth > > -179> 2021-12-23T03:54:44.725+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more payload 9 > > -178> 2021-12-23T03:54:44.725+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 > > -177> 2021-12-23T03:54:44.725+0000 7fc8cc87e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes > > -176> 2021-12-23T03:54:44.725+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more payload 9 > > -175> 2021-12-23T03:54:44.725+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 > > -174> 2021-12-23T03:54:44.725+0000 7fc8cb87c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes > > -173> 2021-12-23T03:54:44.725+0000 7fc8cc07d700 10 monclient(hunting): handle_auth_done global_id 10896482 payload 1091 > > -172> 2021-12-23T03:54:44.725+0000 7fc8cc07d700 10 monclient: _finish_hunting 0 > > -171> 2021-12-23T03:54:44.725+0000 7fc8cc07d700 1 monclient: found mon.noname-c > > -170> 2021-12-23T03:54:44.725+0000 7fc8cc07d700 10 monclient: _send_mon_message to mon.noname-c at v2:192.168.30.13:3300/0 > > -169> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: handle_monmap mon_map magic: 0 v1 > > -168> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: got monmap 69 from mon.noname-c (according to old e69) > > -167> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: dump: > > epoch 69 > > fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d > > last_changed 2021-06-24T11:48:32.597666+0000 > > created 2021-05-29T06:47:46.644056+0000 > > min_mon_release 16 (pacific) > > election_strategy: 1 > > 0: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.cn05 > > 1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.cn02 > > 2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.cn03 > > 3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.cn04 > > 4: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.cn01 > > > > -166> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _finish_auth 0 > > -165> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2021-12-23T03:54:14.726804+0000) > > -164> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > > -163> 2021-12-23T03:54:44.725+0000 7fc8db144780 5 monclient: authenticate success, global_id 10896482 > > -162> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: handle_config config(4 keys) v1 > > -161> 2021-12-23T03:54:44.725+0000 7fc8db144780 10 monclient: wait_auth_rotating waiting for 30 > > -160> 2021-12-23T03:54:44.725+0000 7fc8ca079700 4 set_mon_vals no callback set > > -159> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: handle_monmap mon_map magic: 0 v1 > > -158> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: got monmap 69 from mon.cn03 (according to old e69) > > -157> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: dump: > > epoch 69 > > fsid bfa2ad58-c049-11eb-9098-3c8cf8ed728d > > last_changed 2021-06-24T11:48:32.597666+0000 > > created 2021-05-29T06:47:46.644056+0000 > > min_mon_release 16 (pacific) > > election_strategy: 1 > > 0: [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0] mon.cn05 > > 1: [v2:192.168.30.12:3300/0,v1:192.168.30.12:6789/0] mon.cn02 > > 2: [v2:192.168.30.13:3300/0,v1:192.168.30.13:6789/0] mon.cn03 > > 3: [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0] mon.cn04 > > 4: [v2:192.168.30.11:3300/0,v1:192.168.30.11:6789/0] mon.cn01 > > > > -156> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _finish_auth 0 > > -155> 2021-12-23T03:54:44.725+0000 7fc8c9077700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2021-12-23T03:54:14.727073+0000) > > -154> 2021-12-23T03:54:44.725+0000 7fc8db144780 10 monclient: wait_auth_rotating done > > -153> 2021-12-23T03:54:44.725+0000 7fc8db144780 10 monclient: _renew_subs > > -152> 2021-12-23T03:54:44.725+0000 7fc8db144780 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > > -151> 2021-12-23T03:54:44.725+0000 7fc8db144780 5 asok(0x55761e85a000) register_command status hook 0x55761e8002d0 > > -150> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_ops_in_flight hook 0x55761e8002d0 > > -149> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command ops hook 0x55761e8002d0 > > -148> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_blocked_ops hook 0x55761e8002d0 > > -147> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_historic_ops hook 0x55761e8002d0 > > -146> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump_historic_ops_by_duration hook 0x55761e8002d0 > > -145> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub_path hook 0x55761e8002d0 > > -144> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub start hook 0x55761e8002d0 > > -143> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub abort hook 0x55761e8002d0 > > -142> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub pause hook 0x55761e8002d0 > > -141> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub resume hook 0x55761e8002d0 > > -140> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command scrub status hook 0x55761e8002d0 > > -139> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command tag path hook 0x55761e8002d0 > > -138> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command flush_path hook 0x55761e8002d0 > > -137> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command export dir hook 0x55761e8002d0 > > -136> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump cache hook 0x55761e8002d0 > > -135> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command cache drop hook 0x55761e8002d0 > > -134> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command cache status hook 0x55761e8002d0 > > -133> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump tree hook 0x55761e8002d0 > > -132> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump loads hook 0x55761e8002d0 > > -131> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump snaps hook 0x55761e8002d0 > > -130> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session ls hook 0x55761e8002d0 > > -129> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command client ls hook 0x55761e8002d0 > > -128> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session evict hook 0x55761e8002d0 > > -127> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command client evict hook 0x55761e8002d0 > > -126> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session kill hook 0x55761e8002d0 > > -125> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session ls hook 0x55761e8002d0 > > -124> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command session config hook 0x55761e8002d0 > > -123> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command client config hook 0x55761e8002d0 > > -122> 2021-12-23T03:54:44.726+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Updating MDS map to version 52916 from mon.2 > > -121> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command damage ls hook 0x55761e8002d0 > > -120> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command damage rm hook 0x55761e8002d0 > > -119> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command osdmap barrier hook 0x55761e8002d0 > > -118> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command flush journal hook 0x55761e8002d0 > > -117> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command force_readonly hook 0x55761e8002d0 > > -116> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command get subtrees hook 0x55761e8002d0 > > -115> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dirfrag split hook 0x55761e8002d0 > > -114> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dirfrag merge hook 0x55761e8002d0 > > -113> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dirfrag ls hook 0x55761e8002d0 > > -112> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command openfiles ls hook 0x55761e8002d0 > > -111> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command dump inode hook 0x55761e8002d0 > > -110> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command exit hook 0x55761e8002d0 > > -109> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command respawn hook 0x55761e8002d0 > > -108> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command heap hook 0x55761e8002d0 > > -107> 2021-12-23T03:54:44.726+0000 7fc8db144780 5 asok(0x55761e85a000) register_command cpu_profiler hook 0x55761e8002d0 > > -106> 2021-12-23T03:54:44.726+0000 7fc8c7073700 5 mds.beacon.blumfrub.cn06.gauawm Sending beacon up:boot seq 1 > > -105> 2021-12-23T03:54:44.738+0000 7fc8c7073700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > > -104> 2021-12-23T03:54:45.383+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Updating MDS map to version 52917 from mon.2 > > -103> 2021-12-23T03:54:45.383+0000 7fc8c9077700 10 monclient: _renew_subs > > -102> 2021-12-23T03:54:45.383+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > > -101> 2021-12-23T03:54:45.383+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Monitors have assigned me to become a standby. > > -100> 2021-12-23T03:54:45.383+0000 7fc8c9077700 5 mds.beacon.blumfrub.cn06.gauawm set_want_state: up:boot -> up:standby > > -99> 2021-12-23T03:54:45.385+0000 7fc8cc07d700 5 mds.beacon.blumfrub.cn06.gauawm received beacon reply up:boot seq 1 rtt 0.659011 > > -98> 2021-12-23T03:54:45.387+0000 7fc8c9077700 4 mgrc handle_mgr_map Got map version 515 > > -97> 2021-12-23T03:54:45.387+0000 7fc8c9077700 4 mgrc handle_mgr_map Active mgr is now [v2:192.168.30.12:6848/3684058018,v1:192.168.30.12:6849/3684058018] > > -96> 2021-12-23T03:54:45.387+0000 7fc8c9077700 4 mgrc reconnect Starting new session with [v2:192.168.30.12:6848/3684058018,v1:192.168.30.12:6849/3684058018] > > -95> 2021-12-23T03:54:45.387+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x55761e8f9800 auth_method 0 > > -94> 2021-12-23T03:54:45.389+0000 7fc8c9077700 4 mgrc handle_mgr_configure stats_period=5 > > -93> 2021-12-23T03:54:45.389+0000 7fc8c9077700 4 mgrc handle_mgr_configure updated stats threshold: 5 > > -92> 2021-12-23T03:54:45.516+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Updating MDS map to version 52918 from mon.2 > > -91> 2021-12-23T03:54:45.516+0000 7fc8c9077700 4 mds.0.purge_queue operator(): data pool 19 not found in OSDMap > > -90> 2021-12-23T03:54:45.516+0000 7fc8c9077700 5 asok(0x55761e85a000) register_command objecter_requests hook 0x55761e800130 > > -89> 2021-12-23T03:54:45.517+0000 7fc8c9077700 10 monclient: _renew_subs > > -88> 2021-12-23T03:54:45.517+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > > -87> 2021-12-23T03:54:45.517+0000 7fc8c9077700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) > > -86> 2021-12-23T03:54:45.517+0000 7fc8c9077700 4 mds.0.purge_queue operator(): data pool 19 not found in OSDMap > > -85> 2021-12-23T03:54:45.517+0000 7fc8c9077700 4 mds.0.0 handle_osd_map epoch 0, 0 new blocklist entries > > -84> 2021-12-23T03:54:45.517+0000 7fc8c9077700 1 mds.0.52918 handle_mds_map i am now mds.0.52918 > > -83> 2021-12-23T03:54:45.517+0000 7fc8c9077700 1 mds.0.52918 handle_mds_map state change up:boot --> up:replay > > -82> 2021-12-23T03:54:45.517+0000 7fc8c9077700 5 mds.beacon.blumfrub.cn06.gauawm set_want_state: up:standby -> up:replay > > -81> 2021-12-23T03:54:45.517+0000 7fc8c9077700 1 mds.0.52918 replay_start > > -80> 2021-12-23T03:54:45.517+0000 7fc8c9077700 1 mds.0.52918 waiting for osdmap 19523 (which blocklists prior instance) > > -79> 2021-12-23T03:54:45.517+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > > -78> 2021-12-23T03:54:45.520+0000 7fc8c9077700 4 mds.0.52918 handle_osd_map epoch 19523, 0 new blocklist entries > > -77> 2021-12-23T03:54:45.520+0000 7fc8c9077700 10 monclient: _renew_subs > > -76> 2021-12-23T03:54:45.520+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > > -75> 2021-12-23T03:54:45.520+0000 7fc8c9077700 10 monclient: handle_get_version_reply finishing 1 version 19523 > > -74> 2021-12-23T03:54:45.520+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening inotable > > -73> 2021-12-23T03:54:45.520+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening sessionmap > > -72> 2021-12-23T03:54:45.520+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x55761f656000 auth_method 0 > > -71> 2021-12-23T03:54:45.520+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening mds log > > -70> 2021-12-23T03:54:45.520+0000 7fc8c306b700 5 mds.0.log open discovering log bounds > > -69> 2021-12-23T03:54:45.521+0000 7fc8c286a700 4 mds.0.journalpointer Reading journal pointer '400.00000000' > > -68> 2021-12-23T03:54:45.521+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening purge queue (async) > > -67> 2021-12-23T03:54:45.521+0000 7fc8c306b700 4 mds.0.purge_queue open: opening > > -66> 2021-12-23T03:54:45.521+0000 7fc8c306b700 1 mds.0.journaler.pq(ro) recover start > > -65> 2021-12-23T03:54:45.521+0000 7fc8c306b700 1 mds.0.journaler.pq(ro) read_head > > -64> 2021-12-23T03:54:45.521+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: loading open file table (async) > > -63> 2021-12-23T03:54:45.521+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x55761f656800 auth_method 0 > > -62> 2021-12-23T03:54:45.521+0000 7fc8c306b700 2 mds.0.52918 Booting: 0: opening snap table > > -61> 2021-12-23T03:54:45.521+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x55761f657800 auth_method 0 > > -60> 2021-12-23T03:54:45.521+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x55761f657000 auth_method 0 > > -59> 2021-12-23T03:54:45.521+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x55761f676000 auth_method 0 > > -58> 2021-12-23T03:54:45.521+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x55761f676800 auth_method 0 > > -57> 2021-12-23T03:54:45.522+0000 7fc8c286a700 1 mds.0.journaler.mdlog(ro) recover start > > -56> 2021-12-23T03:54:45.522+0000 7fc8c286a700 1 mds.0.journaler.mdlog(ro) read_head > > -55> 2021-12-23T03:54:45.522+0000 7fc8c286a700 4 mds.0.log Waiting for journal 0x200 to recover... > > -54> 2021-12-23T03:54:45.522+0000 7fc8c406d700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 37748736, expire 38447747, write 38447747, stream_format 1). probing for end of log (from 38447747)... > > -53> 2021-12-23T03:54:45.522+0000 7fc8c406d700 1 mds.0.journaler.pq(ro) probing for end of the log > > -52> 2021-12-23T03:54:45.522+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x55761f656400 auth_method 0 > > -51> 2021-12-23T03:54:45.522+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x55761f657c00 auth_method 0 > > -50> 2021-12-23T03:54:45.523+0000 7fc8c406d700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 38447747 (header had 38447747). recovered. > > -49> 2021-12-23T03:54:45.523+0000 7fc8c406d700 4 mds.0.purge_queue operator(): open complete > > -48> 2021-12-23T03:54:45.523+0000 7fc8c406d700 1 mds.0.journaler.pq(ro) set_writeable > > -47> 2021-12-23T03:54:45.537+0000 7fc8c306b700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 4492099584, expire 4492102817, write 4668403448, stream_format 1). probing for end of log (from 4668403448)... > > -46> 2021-12-23T03:54:45.537+0000 7fc8c306b700 1 mds.0.journaler.mdlog(ro) probing for end of the log > > -45> 2021-12-23T03:54:45.537+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x55761f676400 auth_method 0 > > -44> 2021-12-23T03:54:45.537+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x55761f677000 auth_method 0 > > -43> 2021-12-23T03:54:45.538+0000 7fc8c306b700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 4668403564 (header had 4668403448). recovered. > > -42> 2021-12-23T03:54:45.538+0000 7fc8c286a700 4 mds.0.log Journal 0x200 recovered. > > -41> 2021-12-23T03:54:45.538+0000 7fc8c286a700 4 mds.0.log Recovered journal 0x200 in format 1 > > -40> 2021-12-23T03:54:45.538+0000 7fc8c286a700 2 mds.0.52918 Booting: 1: loading/discovering base inodes > > -39> 2021-12-23T03:54:45.538+0000 7fc8c286a700 0 mds.0.cache creating system inode with ino:0x100 > > -38> 2021-12-23T03:54:45.538+0000 7fc8c286a700 0 mds.0.cache creating system inode with ino:0x1 > > -37> 2021-12-23T03:54:45.539+0000 7fc8c306b700 2 mds.0.52918 Booting: 2: replaying mds log > > -36> 2021-12-23T03:54:45.539+0000 7fc8c306b700 2 mds.0.52918 Booting: 2: waiting for purge queue recovered > > -35> 2021-12-23T03:54:45.539+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x55761f657400 auth_method 0 > > -34> 2021-12-23T03:54:46.128+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x557621715800 auth_method 0 > > -33> 2021-12-23T03:54:46.374+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x557620b6f400 auth_method 0 > > -32> 2021-12-23T03:54:46.917+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x557620b6f000 auth_method 0 > > -31> 2021-12-23T03:54:46.974+0000 7fc8cb87c700 10 monclient: get_auth_request con 0x557620230400 auth_method 0 > > -30> 2021-12-23T03:54:47.083+0000 7fc8cc87e700 10 monclient: get_auth_request con 0x557620230800 auth_method 0 > > -29> 2021-12-23T03:54:47.391+0000 7fc8c1868700 1 mds.0.52918 Finished replaying journal > > -28> 2021-12-23T03:54:47.391+0000 7fc8c1868700 1 mds.0.52918 making mds journal writeable > > -27> 2021-12-23T03:54:47.391+0000 7fc8c1868700 1 mds.0.journaler.mdlog(ro) set_writeable > > -26> 2021-12-23T03:54:47.391+0000 7fc8c1868700 2 mds.0.52918 i am alone, moving to state reconnect > > -25> 2021-12-23T03:54:47.391+0000 7fc8c1868700 3 mds.0.52918 request_state up:reconnect > > -24> 2021-12-23T03:54:47.391+0000 7fc8c1868700 5 mds.beacon.blumfrub.cn06.gauawm set_want_state: up:replay -> up:reconnect > > -23> 2021-12-23T03:54:47.391+0000 7fc8c1868700 5 mds.beacon.blumfrub.cn06.gauawm Sending beacon up:reconnect seq 2 > > -22> 2021-12-23T03:54:47.391+0000 7fc8c1868700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > > -21> 2021-12-23T03:54:47.392+0000 7fc8cc07d700 10 monclient: get_auth_request con 0x5576205c9c00 auth_method 0 > > -20> 2021-12-23T03:54:47.724+0000 7fc8c8075700 10 monclient: tick > > -19> 2021-12-23T03:54:47.724+0000 7fc8c8075700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2021-12-23T03:54:17.725595+0000) > > -18> 2021-12-23T03:54:47.896+0000 7fc8cb87c700 10 monclient: handle_auth_request added challenge on 0x5576205c8400 > > -17> 2021-12-23T03:54:47.897+0000 7fc8c9077700 3 mds.0.server not active yet, waiting > > -16> 2021-12-23T03:54:47.897+0000 7fc8c9077700 3 mds.0.server not active yet, waiting > > -15> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.blumfrub.cn06.gauawm Updating MDS map to version 52919 from mon.2 > > -14> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.52918 handle_mds_map i am now mds.0.52918 > > -13> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.52918 handle_mds_map state change up:replay --> up:reconnect > > -12> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.52918 reconnect_start > > -11> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.52918 reopen_log > > -10> 2021-12-23T03:54:47.900+0000 7fc8c9077700 4 mds.0.52918 reconnect_start: killed 0 blocklisted sessions (179 blocklist entries, 1) > > -9> 2021-12-23T03:54:47.900+0000 7fc8c9077700 1 mds.0.server reconnect_clients -- 1 sessions > > -8> 2021-12-23T03:54:47.900+0000 7fc8c9077700 0 log_channel(cluster) log [DBG] : reconnect by client.10266701 v1:192.168.70.10:0/2935197590 after 0 > > -7> 2021-12-23T03:54:47.902+0000 7fc8c9077700 1 mds.0.52918 reconnect_done > > -6> 2021-12-23T03:54:47.902+0000 7fc8c9077700 3 mds.0.52918 request_state up:rejoin > > -5> 2021-12-23T03:54:47.902+0000 7fc8c9077700 5 mds.beacon.blumfrub.cn06.gauawm set_want_state: up:reconnect -> up:rejoin > > -4> 2021-12-23T03:54:47.902+0000 7fc8c9077700 5 mds.beacon.blumfrub.cn06.gauawm Sending beacon up:rejoin seq 3 > > -3> 2021-12-23T03:54:47.902+0000 7fc8c9077700 10 monclient: _send_mon_message to mon.cn03 at v2:192.168.30.13:3300/0 > > -2> 2021-12-23T03:54:47.902+0000 7fc8cc07d700 5 mds.beacon.blumfrub.cn06.gauawm received beacon reply up:reconnect seq 2 rtt 0.511009 > > -1> 2021-12-23T03:54:48.708+0000 7fc8c9077700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h: In function 'std::ostream& operator<<(std::ostream&, const ClientMetricType&)' thread 7fc8c9077700 time 2021-12-23T03:54:48.708567+0000 > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h: 56: ceph_abort_msg("abort() called") > > > > ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) > > 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7fc8d1cd8cdc] > > 2: (operator<<(std::ostream&, ClientMetricType const&)+0x10e) [0x7fc8d1f5d42e] > > 3: (MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fc8d1f5d601] > > 4: (DispatchQueue::entry()+0x1be2) [0x7fc8d1f13492] > > 5: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc8d1fc2b71] > > 6: /lib64/libpthread.so.0(+0x814a) [0x7fc8d0a7a14a] > > 7: clone() > > > > 0> 2021-12-23T03:54:48.709+0000 7fc8c9077700 -1 *** Caught signal (Aborted) ** > > in thread 7fc8c9077700 thread_name:ms_dispatch > > > > ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) > > 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc8d0a84b20] > > 2: gsignal() > > 3: abort() > > 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x7fc8d1cd8dad] > > 5: (operator<<(std::ostream&, ClientMetricType const&)+0x10e) [0x7fc8d1f5d42e] > > 6: (MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fc8d1f5d601] > > 7: (DispatchQueue::entry()+0x1be2) [0x7fc8d1f13492] > > 8: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc8d1fc2b71] > > 9: /lib64/libpthread.so.0(+0x814a) [0x7fc8d0a7a14a] > > 10: clone() > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > > > --- logging levels --- > > 0/ 5 none > > 0/ 1 lockdep > > 0/ 1 context > > 1/ 1 crush > > 1/ 5 mds > > 1/ 5 mds_balancer > > 1/ 5 mds_locker > > 1/ 5 mds_log > > 1/ 5 mds_log_expire > > 1/ 5 mds_migrator > > 0/ 1 buffer > > 0/ 1 timer > > 0/ 1 filer > > 0/ 1 striper > > 0/ 1 objecter > > 0/ 5 rados > > 0/ 5 rbd > > 0/ 5 rbd_mirror > > 0/ 5 rbd_replay > > 0/ 5 rbd_pwl > > 0/ 5 journaler > > 0/ 5 objectcacher > > 0/ 5 immutable_obj_cache > > 0/ 5 client > > 1/ 5 osd > > 0/ 5 optracker > > 0/ 5 objclass > > 1/ 3 filestore > > 1/ 3 journal > > 0/ 0 ms > > 1/ 5 mon > > 0/10 monc > > 1/ 5 paxos > > 0/ 5 tp > > 1/ 5 auth > > 1/ 5 crypto > > 1/ 1 finisher > > 1/ 1 reserver > > 1/ 5 heartbeatmap > > 1/ 5 perfcounter > > 1/ 5 rgw > > 1/ 5 rgw_sync > > 1/10 civetweb > > 1/ 5 javaclient > > 1/ 5 asok > > 1/ 1 throttle > > 0/ 0 refs > > 1/ 5 compressor > > 1/ 5 bluestore > > 1/ 5 bluefs > > 1/ 3 bdev > > 1/ 5 kstore > > 4/ 5 rocksdb > > 4/ 5 leveldb > > 4/ 5 memdb > > 1/ 5 fuse > > 1/ 5 mgr > > 1/ 5 mgrc > > 1/ 5 dpdk > > 1/ 5 eventtrace > > 1/ 5 prioritycache > > 0/ 5 test > > 0/ 5 cephfs_mirror > > 0/ 5 cephsqlite > > -2/-2 (syslog threshold) > > 99/99 (stderr threshold) > > --- pthread ID / name mapping for recent threads --- > > 140500217005824 / > > 140500233791232 / > > 140500242183936 / MR_Finisher > > 140500258969344 / PQ_Finisher > > 140500309325568 / ceph-mds > > 140500326110976 / safe_timer > > 140500342896384 / ms_dispatch > > 140500359681792 / io_context_pool > > 140500376467200 / admin_socket > > 140500384859904 / msgr-worker-2 > > 140500393252608 / msgr-worker-1 > > 140500401645312 / msgr-worker-0 > > 140500645726080 / ceph-mds > > max_recent 10000 > > max_new 10000 > > log_file /var/lib/ceph/crash/2021-12-23T03:54:48.710208Z_c342e59b-e67a-4f0a-970a-252b5b76ea3b/log > > --- end dump of recent events — > > > > > > META: > > > > { > > "crash_id": "2021-12-23T03:54:48.710208Z_c342e59b-e67a-4f0a-970a-252b5b76ea3b", > > "timestamp": "2021-12-23T03:54:48.710208Z", > > "process_name": "ceph-mds", > > "entity_name": "mds.blumfrub.cn06.gauawm", > > "ceph_version": "16.2.4", > > "utsname_hostname": "cn06.ceph.la1.blumfrub.corp", > > "utsname_sysname": "Linux", > > "utsname_release": "5.15.8-1.el8.elrepo.x86_64", > > "utsname_version": "#1 SMP Mon Dec 13 19:41:26 EST 2021", > > "utsname_machine": "x86_64", > > "os_name": "CentOS Linux", > > "os_id": "centos", > > "os_version_id": "8", > > "os_version": "8", > > "assert_condition": "abort", > > "assert_func": "std::ostream& operator<<(std::ostream&, const ClientMetricType&)", > > "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h", > > "assert_line": 56, > > "assert_thread_name": "ms_dispatch", > > "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h: In function 'std::ostream& operator<<(std::ostream&, const ClientMetricType&)' thread 7fc8c9077700 time 2021-12-23T03:54:48.708567+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.4/rpm/el8/BUILD/ceph-16.2.4/src/include/cephfs/metrics/Types.h: 56: ceph_abort_msg(\"abort() called\")\n", > > "backtrace": [ > > "/lib64/libpthread.so.0(+0x12b20) [0x7fc8d0a84b20]", > > "gsignal()", > > "abort()", > > "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x7fc8d1cd8dad]", > > "(operator<<(std::ostream&, ClientMetricType const&)+0x10e) [0x7fc8d1f5d42e]", > > "(MClientMetrics::print(std::ostream&) const+0x1a1) [0x7fc8d1f5d601]", > > "(DispatchQueue::entry()+0x1be2) [0x7fc8d1f13492]", > > "(DispatchQueue::DispatchThread::entry()+0x11) [0x7fc8d1fc2b71]", > > "/lib64/libpthread.so.0(+0x814a) [0x7fc8d0a7a14a]", > > "clone()" > > ] > > } > > > > > > > > > > I have a feeling somehow a version mismatch happened. I noticed upgrades to local packages going to 16.2.5 at some point by blindly doing a yum upgrade. I downgraded the packages but perhaps the damage was already done? I had plans to go to 16.2.7, but I’d like to understand why I’m experiencing these failures. > > > > I jacked up the service count to four for each mds service so just provide a higher level of redundancy until I understand what is going on. I don’t know if that was a bad move but two separate filesystems are having mds failures. > > > > Thanks > > -jeremy > >
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
