Re: lost all monitors at the same time

[Eugen Block <eblock@xxxxxx>]

I still find it strange that a power outage can break a cluster, we've  
had multiple outages this year and the cluster recovered sucessfully  
every time. Although I should add that it's not containerized yet,  
it's still running on Nautilus.
Anyway, did you verify that all directories are there  
(/var/lib/ceph/<FSID>/) and belong to ceph? Are the keyrings present,  
are the mon stores present? Do all hosts have their IPs and can they  
reach each other?
The monmap should be there if the mon store is intact, I believe.

Zitat von Daniel Brunner <daniel@brunner.ninja>:

> Hi,
>
> I accidently lost power on all nodes at the same time and now my monitors
> do not get in quorum anymore. I cannot mount my cephfs anymore, I cannot
> see the status anymore, seems like all data is lost.
>
> I used cephadm to deploy all nodes, I only see the monitor docker container
> on 2 hosts,
> and both only log "e18 handle_auth_request failed to assign global_id".
>
> Most ceph commands are just stuck, but using the admin socket I at least
> get the status from those 2 monitors.
>
> Do I need to rebuild my monmap from the OSDs now or is my data gone? I
> don't understand the exact issue here. ceph orch commands are stuck and I
> cannot see if any services are missing
>
>
> sudo ceph --admin-daemon
> /run/ceph/69f993e2-637e-11ed-8ca1-22aca0408523/ceph-mon.flucky-server.asok
> mon_status
>
> {
>     "name": "flucky-server",
>     "rank": 0,
>     "state": "probing",
>     "election_epoch": 0,
>     "quorum": [],
>     "features": {
>         "required_con": "2449958755906961412",
>         "required_mon": [
>             "kraken",
>             "luminous",
>             "mimic",
>             "osdmap-prune",
>             "nautilus",
>             "octopus",
>             "pacific",
>             "elector-pinging",
>             "quincy"
>         ],
>         "quorum_con": "0",
>         "quorum_mon": []
>     },
>     "outside_quorum": [
>         "flucky-server",
>         "matrixpi"
>     ],
>     "extra_probe_peers": [],
>     "sync_provider": [],
>     "monmap": {
>         "epoch": 18,
>         "fsid": "69f993e2-637e-11ed-8ca1-22aca0408523",
>         "modified": "2022-11-15T22:36:40.883664Z",
>         "created": "2022-11-13T18:10:17.638340Z",
>         "min_mon_release": 17,
>         "min_mon_release_name": "quincy",
>         "election_strategy": 1,
>         "disallowed_leaders: ": "",
>         "stretch_mode": false,
>         "tiebreaker_mon": "",
>         "features": {
>             "persistent": [
>                 "kraken",
>                 "luminous",
>                 "mimic",
>                 "osdmap-prune",
>                 "nautilus",
>                 "octopus",
>                 "pacific",
>                 "elector-pinging",
>                 "quincy"
>             ],
>             "optional": []
>         },
>         "mons": [
>             {
>                 "rank": 0,
>                 "name": "flucky-server",
>                 "public_addrs": {
>                     "addrvec": [
>                         {
>                             "type": "v2",
>                             "addr": "192.168.0.2:3300",
>                             "nonce": 0
>                         },
>                         {
>                             "type": "v1",
>                             "addr": "192.168.0.2:6789",
>                             "nonce": 0
>                         }
>                     ]
>                 },
>                 "addr": "192.168.0.2:6789/0",
>                 "public_addr": "192.168.0.2:6789/0",
>                 "priority": 0,
>                 "weight": 0,
>                 "crush_location": "{}"
>             },
>             {
>                 "rank": 1,
>                 "name": "matrixpi",
>                 "public_addrs": {
>                     "addrvec": [
>                         {
>                             "type": "v2",
>                             "addr": "192.168.0.3:3300",
>                             "nonce": 0
>                         },
>                         {
>                             "type": "v1",
>                             "addr": "192.168.0.3:6789",
>                             "nonce": 0
>                         }
>                     ]
>                 },
>                 "addr": "192.168.0.3:6789/0",
>                 "public_addr": "192.168.0.3:6789/0",
>                 "priority": 0,
>                 "weight": 0,
>                 "crush_location": "{}"
>             },
>             {
>                 "rank": 2,
>                 "name": "raspberrypi3-09",
>                 "public_addrs": {
>                     "addrvec": [
>                         {
>                             "type": "v2",
>                             "addr": "192.168.0.29:3300",
>                             "nonce": 0
>                         },
>                         {
>                             "type": "v1",
>                             "addr": "192.168.0.29:6789",
>                             "nonce": 0
>                         }
>                     ]
>                 },
>                 "addr": "192.168.0.29:6789/0",
>                 "public_addr": "192.168.0.29:6789/0",
>                 "priority": 0,
>                 "weight": 0,
>                 "crush_location": "{}"
>             },
>             {
>                 "rank": 3,
>                 "name": "raspberrypi3-02",
>                 "public_addrs": {
>                     "addrvec": [
>                         {
>                             "type": "v2",
>                             "addr": "192.168.0.22:3300",
>                             "nonce": 0
>                         },
>                         {
>                             "type": "v1",
>                             "addr": "192.168.0.22:6789",
>                             "nonce": 0
>                         }
>                     ]
>                 },
>                 "addr": "192.168.0.22:6789/0",
>                 "public_addr": "192.168.0.22:6789/0",
>                 "priority": 0,
>                 "weight": 0,
>                 "crush_location": "{}"
>             }
>         ]
>     },
>     "feature_map": {
>         "mon": [
>             {
>                 "features": "0x3f01cfbf7ffdffff",
>                 "release": "luminous",
>                 "num": 1
>             }
>         ]
>     },
>     "stretch_mode": false
> }
>
>
>
> [feedc0de@matrixpi ~]$ sudo ceph --admin-daemon
> /run/ceph/69f993e2-637e-11ed-8ca1-22aca0408523/ceph-mon.matrixpi.asok
> mon_status
>
> {
>     "name": "matrixpi",
>     "rank": 1,
>     "state": "probing",
>     "election_epoch": 0,
>     "quorum": [],
>     "features": {
>         "required_con": "2449958755906961412",
>         "required_mon": [
>             "kraken",
>             "luminous",
>             "mimic",
>             "osdmap-prune",
>             "nautilus",
>             "octopus",
>             "pacific",
>             "elector-pinging",
>             "quincy"
>         ],
>         "quorum_con": "0",
>         "quorum_mon": []
>     },
>     "outside_quorum": [
>         "flucky-server",
>         "matrixpi"
>     ],
>     "extra_probe_peers": [],
>     "sync_provider": [],
>     "monmap": {
>         "epoch": 18,
>         "fsid": "69f993e2-637e-11ed-8ca1-22aca0408523",
>         "modified": "2022-11-15T22:36:40.883664Z",
>         "created": "2022-11-13T18:10:17.638340Z",
>         "min_mon_release": 17,
>         "min_mon_release_name": "quincy",
>         "election_strategy": 1,
>         "disallowed_leaders: ": "",
>         "stretch_mode": false,
>         "tiebreaker_mon": "",
>         "features": {
>             "persistent": [
>                 "kraken",
>                 "luminous",
>                 "mimic",
>                 "osdmap-prune",
>                 "nautilus",
>                 "octopus",
>                 "pacific",
>                 "elector-pinging",
>                 "quincy"
>             ],
>             "optional": []
>         },
>         "mons": [
>             {
>                 "rank": 0,
>                 "name": "flucky-server",
>                 "public_addrs": {
>                     "addrvec": [
>                         {
>                             "type": "v2",
>                             "addr": "192.168.0.2:3300",
>                             "nonce": 0
>                         },
>                         {
>                             "type": "v1",
>                             "addr": "192.168.0.2:6789",
>                             "nonce": 0
>                         }
>                     ]
>                 },
>                 "addr": "192.168.0.2:6789/0",
>                 "public_addr": "192.168.0.2:6789/0",
>                 "priority": 0,
>                 "weight": 0,
>                 "crush_location": "{}"
>             },
>             {
>                 "rank": 1,
>                 "name": "matrixpi",
>                 "public_addrs": {
>                     "addrvec": [
>                         {
>                             "type": "v2",
>                             "addr": "192.168.0.3:3300",
>                             "nonce": 0
>                         },
>                         {
>                             "type": "v1",
>                             "addr": "192.168.0.3:6789",
>                             "nonce": 0
>                         }
>                     ]
>                 },
>                 "addr": "192.168.0.3:6789/0",
>                 "public_addr": "192.168.0.3:6789/0",
>                 "priority": 0,
>                 "weight": 0,
>                 "crush_location": "{}"
>             },
>             {
>                 "rank": 2,
>                 "name": "raspberrypi3-09",
>                 "public_addrs": {
>                     "addrvec": [
>                         {
>                             "type": "v2",
>                             "addr": "192.168.0.29:3300",
>                             "nonce": 0
>                         },
>                         {
>                             "type": "v1",
>                             "addr": "192.168.0.29:6789",
>                             "nonce": 0
>                         }
>                     ]
>                 },
>                 "addr": "192.168.0.29:6789/0",
>                 "public_addr": "192.168.0.29:6789/0",
>                 "priority": 0,
>                 "weight": 0,
>                 "crush_location": "{}"
>             },
>             {
>                 "rank": 3,
>                 "name": "raspberrypi3-02",
>                 "public_addrs": {
>                     "addrvec": [
>                         {
>                             "type": "v2",
>                             "addr": "192.168.0.22:3300",
>                             "nonce": 0
>                         },
>                         {
>                             "type": "v1",
>                             "addr": "192.168.0.22:6789",
>                             "nonce": 0
>                         }
>                     ]
>                 },
>                 "addr": "192.168.0.22:6789/0",
>                 "public_addr": "192.168.0.22:6789/0",
>                 "priority": 0,
>                 "weight": 0,
>                 "crush_location": "{}"
>             }
>         ]
>     },
>     "feature_map": {
>         "mon": [
>             {
>                 "features": "0x3f01cfbf7ffdffff",
>                 "release": "luminous",
>                 "num": 1
>             }
>         ],
>         "client": [
>             {
>                 "features": "0x2f018fb87aa4aafe",
>                 "release": "luminous",
>                 "num": 1
>             }
>         ]
>     },
>     "stretch_mode": false
> }
>
>
>  feedc0de@flucky-server:~$ docker logs -f
> ceph-69f993e2-637e-11ed-8ca1-22aca0408523-mon-flucky-server
> debug 2022-11-15T23:36:06.582+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:07.079+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:07.179+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:07.289+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:07.429+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:07.899+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:08.632+0000 7f0b211f7700 -1  
> mon.flucky-server@0(probing)
> e18 get_health_metrics reporting 2 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:09.785+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:10.282+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:10.632+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:11.152+0000 7f0b249fe700  0 log_channel(audit) log
> [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]:
> dispatch
> debug 2022-11-15T23:36:11.152+0000 7f0b249fe700  0 log_channel(audit) log
> [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]:
> finished
> debug 2022-11-15T23:36:11.642+0000 7f0b1c9ee700  0 cephx: verify_authorizer
> could not decrypt ticket info: error: bad magic in decode_decrypt,
> 16288580011235018902 != 18374858748799134293
> debug 2022-11-15T23:36:13.632+0000 7f0b211f7700 -1  
> mon.flucky-server@0(probing)
> e18 get_health_metrics reporting 2 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:16.192+0000 7f0b1d1ef700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:16.689+0000 7f0b1d1ef700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:17.039+0000 7f0b1d1ef700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:18.632+0000 7f0b211f7700 -1  
> mon.flucky-server@0(probing)
> e18 get_health_metrics reporting 2 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:22.233+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:22.349+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:22.959+0000 7f0b1d1ef700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:23.633+0000 7f0b211f7700 -1  
> mon.flucky-server@0(probing)
> e18 get_health_metrics reporting 2 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:26.659+0000 7f0b1c9ee700  0 cephx: verify_authorizer
> could not decrypt ticket info: error: bad magic in decode_decrypt,
> 16288580011235018902 != 18374858748799134293
> debug 2022-11-15T23:36:28.633+0000 7f0b211f7700 -1  
> mon.flucky-server@0(probing)
> e18 get_health_metrics reporting 2 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:29.006+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:29.503+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:29.853+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:33.633+0000 7f0b211f7700 -1  
> mon.flucky-server@0(probing)
> e18 get_health_metrics reporting 2 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:37.286+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:37.410+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:38.020+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:38.633+0000 7f0b211f7700 -1  
> mon.flucky-server@0(probing)
> e18 get_health_metrics reporting 2 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:41.667+0000 7f0b1c9ee700  0 cephx: verify_authorizer
> could not decrypt ticket info: error: bad magic in decode_decrypt,
> 16288580011235018902 != 18374858748799134293
> debug 2022-11-15T23:36:43.633+0000 7f0b211f7700 -1  
> mon.flucky-server@0(probing)
> e18 get_health_metrics reporting 4 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:44.020+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:44.517+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:44.867+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:48.633+0000 7f0b211f7700 -1  
> mon.flucky-server@0(probing)
> e18 get_health_metrics reporting 4 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:52.340+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:52.460+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:53.077+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:53.634+0000 7f0b211f7700 -1  
> mon.flucky-server@0(probing)
> e18 get_health_metrics reporting 4 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:56.687+0000 7f0b1c9ee700  0 cephx: verify_authorizer
> could not decrypt ticket info: error: bad magic in decode_decrypt,
> 16288580011235018902 != 18374858748799134293
> debug 2022-11-15T23:36:58.634+0000 7f0b211f7700 -1  
> mon.flucky-server@0(probing)
> e18 get_health_metrics reporting 4 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:59.037+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:59.534+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:59.881+0000 7f0b1c9ee700  1  
> mon.flucky-server@0(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:37:03.634+0000 7f0b211f7700 -1  
> mon.flucky-server@0(probing)
> e18 get_health_metrics reporting 4 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
>
>
>
> [feedc0de@matrixpi ~]$ docker logs -f
> ceph-69f993e2-637e-11ed-8ca1-22aca0408523-mon-matrixpi
> debug 2022-11-15T23:36:11.467+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:16.081+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:16.468+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:16.690+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:21.468+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:22.369+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:22.390+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:22.476+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:22.575+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:22.718+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:22.820+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:22.825+0000 ffff9fe6d080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:26.468+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:28.894+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:29.502+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:31.468+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:36.469+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:37.391+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:37.445+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:37.528+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:37.624+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:37.771+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:37.857+0000 ffff9fe6d080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:37.869+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:41.469+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:43.909+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:44.518+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:46.469+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:51.469+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:52.450+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:52.502+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:52.577+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:52.684+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:52.821+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:52.916+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:52.918+0000 ffff9fe6d080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:56.470+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:36:58.926+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:36:59.533+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:37:01.470+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:37:06.470+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:37:07.501+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:37:07.546+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:37:07.635+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:37:07.743+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:37:07.875+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:37:07.975+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:37:07.977+0000 ffff9fe6d080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:37:11.470+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:37:13.938+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:37:14.546+0000 ffff9f66c080  1 mon.matrixpi@1(probing)
> e18 handle_auth_request failed to assign global_id
> debug 2022-11-15T23:37:16.469+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> debug 2022-11-15T23:37:21.470+0000 ffffa3e75080 -1 mon.matrixpi@1(probing)
> e18 get_health_metrics reporting 7 slow ops, oldest is auth(proto 0 34
> bytes epoch 0)
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx



_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx