That is indeed unexpected, but good for you. ;-) Is the rest of the
cluster healthy now?
Zitat von Gaël THEROND <gael.therond@xxxxxxxxxxxx>:
So!
Here is really mysterious resolution.
The issue vanished at the moment I requested the osd about its slow_ops
history.
I didn’t had time to do anything except to look for the osd ops history
that was actually empty :-)
I’ll keep all your suggestions if it ever came back :-)
Thanks a lot!
Le mer. 23 févr. 2022 à 12:51, Gaël THEROND <gael.therond@xxxxxxxxxxxx> a
écrit :
Thanks a lot Eugene, I dumbly forgot about the rbd block prefix!
I’ll try that this afternoon and told you how it went.
Le mer. 23 févr. 2022 à 11:41, Eugen Block <eblock@xxxxxx> a écrit :
Hi,
> How can I identify which operation this OSD is trying to achieve as
> osd_op() is a bit large ^^ ?
I would start by querying the OSD for historic_slow_ops:
ceph daemon osd.<OSD> dump_historic_slow_ops to see which operation it is.
> How can I identify the related images to this data chunk?
You could go through all rbd images and check for the line containing
block_name_prefix, this could take some time depending on how many
images you have:
block_name_prefix: rbd_data.ca69416b8b4567
I sometimes do that with this for loop:
for i in `rbd -p <POOL> ls`; do if [ $(rbd info <POOL>/$i | grep -c
<PREFIX>) -gt 0 ]; then echo "image: $i"; break; fi; done
So in your case it would look something like this:
for i in `rbd -p <POOL> ls`; do if [ $(rbd info <POOL>/$i | grep -c
89a4a940aba90b -gt 0 ]; then echo "image: $i"; break; fi; done
To see which clients are connected you can check the mon daemon:
ceph daemon mon.<MON> sessions
The mon daemon also has a history of slow ops:
ceph daemon mon.<MON> dump_historic_slow_ops
Regards,
Eugen
Zitat von Gaël THEROND <gael.therond@xxxxxxxxxxxx>:
> Hi everyone, I'm having a really nasty issue since around two days where
> our cluster report a bunch of SLOW_OPS on one of our OSD as:
>
> https://paste.openstack.org/show/b3DkgnJDVx05vL5o4OmY/
>
> Here is the cluster specification:
> * Used to store Openstack related data (VMs/Snaphots/Volumes/Swift).
> * Based on CEPH Nautilus 14.2.8 installed using ceph-ansible.
> * Use an EC based storage profile.
> * We have a separate and dedicated frontend and backend 10Gbps
network.
> * We don't have any network issues observed or reported by our
monitoring
> system.
>
> Here is our current cluster status:
> https://paste.openstack.org/show/biVnkm9Yyog3lmSUn0UK/
> Here is a detailed view of our cluster status:
> https://paste.openstack.org/show/bgKCSVuow0JUZITo2Ndj/
>
> My main issue here is that this health alert is starting to fill the
> Monitor's disk and so trigger a MON_DISK_BIG alert.
>
> I'm worried as I'm having a hard time to identify which osd operation is
> actually slow and especially, which image does it concern and which
client
> is using it.
>
> So far I've try:
> * To match this client ID with any watcher of our stored
> volumes/vms/snaphots by extracting the whole list and then using the
> following command: *rbd status <pool>/<image>*
> Unfortunately none of the watchers is matching my reported client
from
> the OSD on any pool.
>
> * * *To map this reported chunk of data to any of our store image
> using: *ceph
> osd map <pool>/rbd_data.5.89a4a940aba90b.00000000000000a0*
> Unfortunately any pool name existing within our cluster give me
back
> an answer with no image information and a different watcher client ID.
>
> So my questions are:
>
> How can I identify which operation this OSD is trying to achieve as
> osd_op() is a bit large ^^ ?
> Does the *snapc *information part within the log relate to snapshot or
is
> that something totally different?
> How can I identify the related images to this data chunk?
> Is there official documentation about SLOW_OPS operations code
explaining
> how to read the logs like something that explains which block is PG
> number, which is the ID of something etc?
>
> Thanks a lot everyone and feel free to ask for additional information!
> G.
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
