Re: osd_pg_create causing slow requests in Nautilus

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Paul and others,

while digging deeper, I noticed that when the cluster gets into this
state, osd_map_cache_miss on OSDs starts growing rapidly.. even when
I increased osd map cache size to 500 (which was the default at least
for luminous) it behaves the same..

I think this could be related..

I'll try playing more with cache settings..

BR

nik



On Wed, Mar 11, 2020 at 03:40:04PM +0100, Paul Emmerich wrote:
> Encountered this one again today, I've updated the issue with new
> information: https://tracker.ceph.com/issues/44184
> 
> 
> Paul
> 
> -- 
> Paul Emmerich
> 
> Looking for help with your Ceph cluster? Contact us at https://croit.io
> 
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
> 
> On Sat, Feb 29, 2020 at 10:21 PM Nikola Ciprich
> <nikola.ciprich@xxxxxxxxxxx> wrote:
> >
> > Hi,
> >
> > I just wanted to report we've just hit very similar problem.. on mimic
> > (13.2.6). Any manipulation with OSD (ie restart) causes lot of slow
> > ops caused by waiting for new map. It seems those are slowed by SATA
> > OSDs which keep being 100% busy reading for long time until all ops are gone,
> > blocking OPS on unrelated NVME pools - SATA pools are completely unused now.
> >
> > is this possible that those maps are being requested from slow SATA OSDs
> > and it takes such a long time for some reason? why could it take so long?
> > the cluster is very small with very light load..
> >
> > BR
> >
> > nik
> >
> >
> >
> > On Wed, Feb 19, 2020 at 10:03:35AM +0100, Wido den Hollander wrote:
> > >
> > >
> > > On 2/19/20 9:34 AM, Paul Emmerich wrote:
> > > > On Wed, Feb 19, 2020 at 7:26 AM Wido den Hollander <wido@xxxxxxxx> wrote:
> > > >>
> > > >>
> > > >>
> > > >> On 2/18/20 6:54 PM, Paul Emmerich wrote:
> > > >>> I've also seen this problem on Nautilus with no obvious reason for the
> > > >>> slowness once.
> > > >>
> > > >> Did this resolve itself? Or did you remove the pool?
> > > >
> > > > I've seen this twice on the same cluster, it fixed itself the first
> > > > time (maybe with some OSD restarts?) and the other time I removed the
> > > > pool after a few minutes because the OSDs were running into heartbeat
> > > > timeouts. There unfortunately seems to be no way to reproduce this :(
> > > >
> > >
> > > Yes, that's the problem. I've been trying to reproduce it, but I can't.
> > > It works on all my Nautilus systems except for this one.
> > >
> > > As you saw it, Bryan saw it, I expect others to encounter this at some
> > > point as well.
> > >
> > > I don't have any extensive logging as this cluster is in production and
> > > I can't simply crank up the logging and try again.
> > >
> > > > In this case it wasn't a new pool that caused problems but a very old one.
> > > >
> > > >
> > > > Paul
> > > >
> > > >>
> > > >>> In my case it was a rather old cluster that was upgraded all the way
> > > >>> from firefly
> > > >>>
> > > >>>
> > > >>
> > > >> This cluster has also been installed with Firefly. It was installed in
> > > >> 2015, so a while ago.
> > > >>
> > > >> Wido
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users@xxxxxxx
> > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > >
> >
> > --
> > -------------------------------------
> > Ing. Nikola CIPRICH
> > LinuxBox.cz, s.r.o.
> > 28.rijna 168, 709 00 Ostrava
> >
> > tel.:   +420 591 166 214
> > fax:    +420 596 621 273
> > mobil:  +420 777 093 799
> > www.linuxbox.cz
> >
> > mobil servis: +420 737 238 656
> > email servis: servis@xxxxxxxxxxx
> > -------------------------------------
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@xxxxxxxxxxx
-------------------------------------
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux