This is a common error on my system (Pacific).
It appears that there is internal confusion as to where the crash
support stuff lives - whether it's new-style (administered and under
/var/lib/ceph/fsid) or legacy style (/var/lib/ceph). One way to fake it
out was to manually created a minimal crash infrastructure (keyring)
under /var/lib/ceph/crash and assign appropriate access rights.
Tim
On 10/29/24 10:41, mailing-lists wrote:
Hey Cephers,
i was investigating some other issue, when I stumbled across this. I
am not sure, if this is "as intended" or faulty. This is a cephadm
cluster on reef 18.2.4, containerized with docker.
The ceph-crash module states that it cant find its key and that it
cant access RADOS.
Pre-face, this is how its key looks like and what its caps are (dont
worry about leaking the key, its a Test-Cluster):
client.crash.bi-ubu-srv-ceph2-01
key: AQBi5CBnedEKORAAgaTwkiqO7KJ+wzu8+EGXEQ==
caps: [mgr] profile crash
caps: [mon] profile crash
Looks like it should. As per documentation:
https://docs.ceph.com/en/reef/mgr/crash/#automated-collection
The key on the node/directory of this ceph-crash container is the same.
This is the docker logs of the container running ceph-crash.
INFO:ceph-crash:pinging cluster to exercise our key
2024-10-29T13:34:28.261+0000 7f7abb1af640 -1 auth: unable to find a
keyring on
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:
(2) No such file or directory
2024-10-29T13:34:28.261+0000 7f7abb1af640 -1
AuthRegistry(0x7f7ab4067810) no keyring found at
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,
disabling cephx
2024-10-29T13:34:28.265+0000 7f7abb1af640 -1 auth: unable to find a
keyring on
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:
(2) No such file or directory
2024-10-29T13:34:28.265+0000 7f7abb1af640 -1
AuthRegistry(0x7f7abb1ae0c0) no keyring found at
/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,
disabling cephx
2024-10-29T13:34:28.265+0000 7f7ab99ac640 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support [1]
2024-10-29T13:34:28.265+0000 7f7aba1ad640 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support [1]
2024-10-29T13:34:28.265+0000 7f7aba9ae640 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support [1]
2024-10-29T13:34:28.265+0000 7f7abb1af640 -1 monclient: authenticate
NOTE: no keyring found; disabled cephx authentication
[errno 13] RADOS permission denied (error connecting to the cluster)
INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s
As we can see here, it says that it couldn't find its key. After some
online research, I found that some names are not good/interpreted
correctly. Well, just to test, i've changed the unit.run file of
ceph-crash-container and restarted it.
This is the part I've changed:
From
/var/lib/ceph/a12b3ade-2849-11ed-9b46-c5b62beb178a/crash.bi-ubu-srv-ceph2-01/keyring:/etc/ceph/ceph.client.crash.bi-ubu-srv-ceph2-01.keyring
To
/var/lib/ceph/a12b3ade-2849-11ed-9b46-c5b62beb178a/crash.bi-ubu-srv-ceph2-01/keyring:/etc/ceph/keyring
Then the logs look different, but the permissions denied error is
still there.
INFO:ceph-crash:pinging cluster to exercise our key
2024-10-29T13:36:42.273+0000 7f1d04b49640 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support
[2,1]
2024-10-29T13:36:42.277+0000 7f1cff7fe640 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support
[2,1]
2024-10-29T13:36:42.281+0000 7f1cfffff640 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support
[2,1]
[errno 13] RADOS permission denied (error connecting to the cluster)
INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s
Can anybody tell me if this is "normal"?
I think this is suspicious. Because for example, if i do ceph crash
ls, it does not yield anything, while there is definitely something on
a node inside the folder /var/lib/ceph/xyz/crash/
Thanks in advance and best wishes!
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
