since quincy i'm randomly getting authentication issues from clients to osds.
symptom is qemu hangs, but when it happens, i can reproduce it using:
> ceph tell osd.\* version
some - but only some - osds will never respond, but only to clients
on _some_ hosts.
the client gets stuck in a loop with this error
> 2023-03-14T10:09:38.492+0100 7f38f5d95700 1 --2- 10.180.10.36:0/329477069 >> [v2:10.180.10.24:6810/697584,v1:10.180.10.24:6811/697584] conn(0x7f38f0107990 0x7f38f0107d60 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)
restarting the affected OSD helps for a few hours.
in the osd log i see only
> 2023-03-14T09:27:27.801+0000 7fb79020a700 10 osd.4 114909 ms_handle_authentication session 0x55880cd58b40 client.admin h
as caps osdcap[grant(*)] 'allow *'
> 2023-03-14T09:27:27.805+0000 7fb781a3c700 2 osd.4 114909 ms_handle_reset con 0x55880a7fec00 session 0x55880cd58b40
searching for this issue gives me people whos mon is dead, but i dont
think "tell" is supposed to go through mon, beyond the initial
listing, which succeeds. but here's the full auth log from mon anyway
if it helps:
2023-03-14T09:34:48.847+0000 7fcc8a5c7700 10 In
get_auth_session_handler for protocol 0
2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 start_session
entity_name=client.admin global_id=6751719 is_new_global_id=1
2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx server
client.admin: start_session server_challenge 20aa2b96857f41cf
2023-03-14T09:34:48.847+0000 7fcc865bf700 10 start_session
entity_name=client.admin global_id=6751722 is_new_global_id=1
2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx server
client.admin: start_session server_challenge 6066dd1200ddc855
2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx server
client.admin: handle_request get_auth_session_key for client.admin
2023-03-14T09:34:48.847+0000 7fcc84dbc700 20 cephx server
client.admin: checking key: req.key=92ed7ea281e9ac0c
expected_key=92ed7ea281e9ac0c
2023-03-14T09:34:48.847+0000 7fcc84dbc700 20 cephx server
client.admin: checking old_ticket: secret_id=0 len=0,
old_ticket_may_be_omitted=0
2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx server
client.admin: new global_id 6751719
2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx:
build_service_ticket_reply encoding 1 tickets with secret REDACTED==
2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx:
build_service_ticket service auth secret_id 160
ticket_info.ticket.name=client.admin ticket.global_id 6751719
2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx keyserverdata:
get_caps: name=client.admin
2023-03-14T09:34:48.847+0000 7fcc84dbc700 10 cephx keyserverdata:
get_secret: num of caps=4
2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx server
client.admin: handle_request get_auth_session_key for client.admin
2023-03-14T09:34:48.847+0000 7fcc865bf700 20 cephx server
client.admin: checking key: req.key=3c1f6182caf84073
expected_key=3c1f6182caf84073
2023-03-14T09:34:48.847+0000 7fcc865bf700 20 cephx server
client.admin: checking old_ticket: secret_id=0 len=0,
old_ticket_may_be_omitted=0
2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx server
client.admin: new global_id 6751722
2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx:
build_service_ticket_reply encoding 1 tickets with secret REDACTED==
2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx:
build_service_ticket service auth secret_id 160
ticket_info.ticket.name=client.admin ticket.global_id 6751722
2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx keyserverdata:
get_caps: name=client.admin
2023-03-14T09:34:48.847+0000 7fcc865bf700 10 cephx keyserverdata:
get_secret: num of caps=4
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 start_session
entity_name=client.admin global_id=6751725 is_new_global_id=1
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server
client.admin: start_session server_challenge 22fa068f8da1fb28
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server
client.admin: handle_request get_auth_session_key for client.admin
2023-03-14T09:34:48.851+0000 7fcc84dbc700 20 cephx server
client.admin: checking key: req.key=fc7fdedb8e669347
expected_key=fc7fdedb8e669347
2023-03-14T09:34:48.851+0000 7fcc84dbc700 20 cephx server
client.admin: checking old_ticket: secret_id=0 len=0,
old_ticket_may_be_omitted=0
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server
client.admin: new global_id 6751725
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx:
build_service_ticket_reply encoding 1 tickets with secret REDACTED==
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx:
build_service_ticket service auth secret_id 160
ticket_info.ticket.name=client.admin ticket.global_id 6751725
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata:
get_caps: name=client.admin
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata:
get_secret: num of caps=4
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server
client.admin: adding key for service mon
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server
client.admin: adding key for service osd
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata:
get_caps: name=client.admin
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata:
get_secret: num of caps=4
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx server
client.admin: adding key for service mgr
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata:
get_caps: name=client.admin
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx keyserverdata:
get_secret: num of caps=4
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx:
build_service_ticket_reply encoding 3 tickets with secret REDACTED==
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx:
build_service_ticket service mon secret_id 11455
ticket_info.ticket.name=client.admin ticket.global_id 6751725
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx:
build_service_ticket service osd secret_id 11455
ticket_info.ticket.name=client.admin ticket.global_id 6751725
2023-03-14T09:34:48.851+0000 7fcc84dbc700 10 cephx:
build_service_ticket service mgr secret_id 11455
ticket_info.ticket.name=client.admin ticket.global_id 6751725
2023-03-14T09:34:48.851+0000 7fcc865bf700 0
mon.mon-yca4ceph@0(leader) e8 handle_command mon_command({"prefix":
"osd ls"} v 0) v1
2023-03-14T09:34:48.851+0000 7fcc865bf700 0 log_channel(audit) log
[DBG] : from='client.? 10.180.10.36:0/329477069' entity='client.admin'
cmd=[{"prefix": "osd ls"}]: dispatch
--
+4916093821054
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
