Unexpectedly, everything disappeared and the cluster health went back to its previous state! I think I’ll never have a definitive answer ^^ I’ve been able to find out a really nice way to get the rbd stats/iotop on our prometheus using the mgr plugin too and it’s awesome as we can now better chase projects that are putting pressure on the cluster even if the Openstack platform is having QoS in place all over ah ah :-) Le mer. 23 févr. 2022 à 16:57, Eugen Block <eblock@xxxxxx> a écrit : > That is indeed unexpected, but good for you. ;-) Is the rest of the > cluster healthy now? > > Zitat von Gaël THEROND <gael.therond@xxxxxxxxxxxx>: > > > So! > > > > Here is really mysterious resolution. > > The issue vanished at the moment I requested the osd about its slow_ops > > history. > > > > I didn’t had time to do anything except to look for the osd ops history > > that was actually empty :-) > > > > I’ll keep all your suggestions if it ever came back :-) > > > > Thanks a lot! > > > > Le mer. 23 févr. 2022 à 12:51, Gaël THEROND <gael.therond@xxxxxxxxxxxx> > a > > écrit : > > > >> Thanks a lot Eugene, I dumbly forgot about the rbd block prefix! > >> > >> I’ll try that this afternoon and told you how it went. > >> > >> Le mer. 23 févr. 2022 à 11:41, Eugen Block <eblock@xxxxxx> a écrit : > >> > >>> Hi, > >>> > >>> > How can I identify which operation this OSD is trying to achieve as > >>> > osd_op() is a bit large ^^ ? > >>> > >>> I would start by querying the OSD for historic_slow_ops: > >>> > >>> ceph daemon osd.<OSD> dump_historic_slow_ops to see which operation it > is. > >>> > >>> > How can I identify the related images to this data chunk? > >>> > >>> You could go through all rbd images and check for the line containing > >>> block_name_prefix, this could take some time depending on how many > >>> images you have: > >>> > >>> block_name_prefix: rbd_data.ca69416b8b4567 > >>> > >>> I sometimes do that with this for loop: > >>> > >>> for i in `rbd -p <POOL> ls`; do if [ $(rbd info <POOL>/$i | grep -c > >>> <PREFIX>) -gt 0 ]; then echo "image: $i"; break; fi; done > >>> > >>> So in your case it would look something like this: > >>> > >>> for i in `rbd -p <POOL> ls`; do if [ $(rbd info <POOL>/$i | grep -c > >>> 89a4a940aba90b -gt 0 ]; then echo "image: $i"; break; fi; done > >>> > >>> To see which clients are connected you can check the mon daemon: > >>> > >>> ceph daemon mon.<MON> sessions > >>> > >>> The mon daemon also has a history of slow ops: > >>> > >>> ceph daemon mon.<MON> dump_historic_slow_ops > >>> > >>> Regards, > >>> Eugen > >>> > >>> > >>> Zitat von Gaël THEROND <gael.therond@xxxxxxxxxxxx>: > >>> > >>> > Hi everyone, I'm having a really nasty issue since around two days > where > >>> > our cluster report a bunch of SLOW_OPS on one of our OSD as: > >>> > > >>> > https://paste.openstack.org/show/b3DkgnJDVx05vL5o4OmY/ > >>> > > >>> > Here is the cluster specification: > >>> > * Used to store Openstack related data > (VMs/Snaphots/Volumes/Swift). > >>> > * Based on CEPH Nautilus 14.2.8 installed using ceph-ansible. > >>> > * Use an EC based storage profile. > >>> > * We have a separate and dedicated frontend and backend 10Gbps > >>> network. > >>> > * We don't have any network issues observed or reported by our > >>> monitoring > >>> > system. > >>> > > >>> > Here is our current cluster status: > >>> > https://paste.openstack.org/show/biVnkm9Yyog3lmSUn0UK/ > >>> > Here is a detailed view of our cluster status: > >>> > https://paste.openstack.org/show/bgKCSVuow0JUZITo2Ndj/ > >>> > > >>> > My main issue here is that this health alert is starting to fill the > >>> > Monitor's disk and so trigger a MON_DISK_BIG alert. > >>> > > >>> > I'm worried as I'm having a hard time to identify which osd > operation is > >>> > actually slow and especially, which image does it concern and which > >>> client > >>> > is using it. > >>> > > >>> > So far I've try: > >>> > * To match this client ID with any watcher of our stored > >>> > volumes/vms/snaphots by extracting the whole list and then using the > >>> > following command: *rbd status <pool>/<image>* > >>> > Unfortunately none of the watchers is matching my reported > client > >>> from > >>> > the OSD on any pool. > >>> > > >>> > * * *To map this reported chunk of data to any of our store image > >>> > using: *ceph > >>> > osd map <pool>/rbd_data.5.89a4a940aba90b.00000000000000a0* > >>> > Unfortunately any pool name existing within our cluster give me > >>> back > >>> > an answer with no image information and a different watcher client > ID. > >>> > > >>> > So my questions are: > >>> > > >>> > How can I identify which operation this OSD is trying to achieve as > >>> > osd_op() is a bit large ^^ ? > >>> > Does the *snapc *information part within the log relate to snapshot > or > >>> is > >>> > that something totally different? > >>> > How can I identify the related images to this data chunk? > >>> > Is there official documentation about SLOW_OPS operations code > >>> explaining > >>> > how to read the logs like something that explains which block is PG > >>> > number, which is the ID of something etc? > >>> > > >>> > Thanks a lot everyone and feel free to ask for additional > information! > >>> > G. > >>> > _______________________________________________ > >>> > ceph-users mailing list -- ceph-users@xxxxxxx > >>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > >>> > >>> > >>> > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users@xxxxxxx > >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx > >>> > >> > > > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |