Re: octopus rbd cluster just stopped out of nowhere (>20k slow ops)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Perhaps run "iostat -xtcy <list of OSD devices> 5" on the OSD hosts to
see if any of the drives have weirdly high utilization despite low
iops/requests?


Den tis 6 dec. 2022 kl 10:02 skrev Boris Behrens <bb@xxxxxxxxx>:
>
> Hi Sven,
> I am searching really hard for defect hardware, but I am currently out of
> ideas:
> - checked prometheus stats, but in all that data I don't know what to look
> for (osd apply latency if very low at the mentioned point and went up to
> 40ms after all OSDs were restarted)
> - smartctl shows nothing
> - dmesg show nothing
> - network data shows nothing
> - osd and clusterlogs show nothing
>
> If anybody got a good tip what I can check, that would be awesome. A string
> in the logs (I made a copy from that days logs), or a tool to fire against
> the hardware. I am 100% out of ideas what it could be.
> In a time frame of 20s 2/3 of our OSDs went from "all fine" to "I am
> waiting for the replicas to do their work" (log message 'waiting for sub
> ops'). But there was no alert that any OSD had connection problems to other
> OSDs. Additional the cluster_network is the same interface, switch,
> everything as public_network. Only difference is the VLAN id (I plan to
> remove the cluster_network because it does not provide anything for us).
>
> I am also planning to update all hosts from centos7 to ubuntu 20.04 (newer
> kernel, standardized OS config and so on).
>
> Am Mo., 5. Dez. 2022 um 14:24 Uhr schrieb Sven Kieske <S.Kieske@xxxxxxxxxxx
> >:
>
> > On Sa, 2022-12-03 at 01:54 +0100, Boris Behrens wrote:
> > > hi,
> > > maybe someone here can help me to debug an issue we faced today.
> > >
> > > Today one of our clusters came to a grinding halt with 2/3 of our OSDs
> > > reporting slow ops.
> > > Only option to get it back to work fast, was to restart all OSDs daemons.
> > >
> > > The cluster is an octopus cluster with 150 enterprise SSD OSDs. Last work
> > > on the cluster: synced in a node 4 days ago.
> > >
> > > The only health issue, that was reported, was the SLOW_OPS. No slow pings
> > > on the networks. No restarting OSDs. Nothing.
> > >
> > > I was able to ping it to a 20s timeframe and I read ALL the logs in a 20
> > > minute timeframe around this issue.
> > >
> > > I haven't found any clues.
> > >
> > > Maybe someone encountered this in the past?
> >
> > do you happen to run your rocksdb on a dedicated caching device (nvme ssd)?
> >
> > I observed slow ops in octopus after a faulty nvme ssd was inserted in one
> > ceph server.
> > as was said in other mails, try to isolate your root cause.
> >
> > maybe the node added 4 days ago was the culprit here?
> >
> > we were able to pinpoint the nvme by monitoring the slow osds
> > and the commonality in this case was the same nvme cache device.
> >
> > you should always benchmark new hardware/perform burn-in tests imho, which
> > is not always possible due to environment constraints.
> >
> > --
> > Mit freundlichen Grüßen / Regards
> >
> > Sven Kieske
> > Systementwickler / systems engineer
> >
> >
> > Mittwald CM Service GmbH & Co. KG
> > Königsberger Straße 4-6
> > 32339 Espelkamp
> >
> > Tel.: 05772 / 293-900
> > Fax: 05772 / 293-333
> >
> > https://www.mittwald.de
> >
> > Geschäftsführer: Robert Meyer, Florian Jürgens
> >
> > St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
> > Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
> >
> > Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit
> > gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds abrufbar.
> >
> >
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> groüen Saal.
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx



-- 
May the most significant bit of your life be positive.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux