no, we dont use it... 803.3ad is LACP or bonding option "mode=4" or "mode=802.3ad" we use mode=1 which is active-backup and have nothing to do with LACP explictly because we want to push all traffic thru one switch (we have a pair of them) to avoid pushing traffic thru link between those switches and only use it in case of either network card failure or dead switch also AFAIK cant do LACP between 1 machine and 2 switches if switches do not support that On Mon, 7 Sep 2015 09:22:19 -0400 (EDT), Shinobu Kinjo <skinjo@xxxxxxxxxx> wrote: > O.k, that's the protocol, 803.ad. > > ----- Original Message ----- > From: "Mariusz Gronczewski" <mariusz.gronczewski@xxxxxxxxxxxx> > To: "Shinobu Kinjo" <skinjo@xxxxxxxxxx> > Cc: "Jan Schermer" <jan@xxxxxxxxxxx>, ceph-users@xxxxxxxxxxxxxx > Sent: Monday, September 7, 2015 10:19:23 PM > Subject: Re: Huge memory usage spike in OSD on hammer/giant > > yes > > On Mon, 7 Sep 2015 09:15:55 -0400 (EDT), Shinobu Kinjo > <skinjo@xxxxxxxxxx> wrote: > > > > master/slave > > > > Meaning that you are using bonding? > > > > ----- Original Message ----- > > From: "Mariusz Gronczewski" <mariusz.gronczewski@xxxxxxxxxxxx> > > To: "Shinobu Kinjo" <skinjo@xxxxxxxxxx> > > Cc: "Jan Schermer" <jan@xxxxxxxxxxx>, ceph-users@xxxxxxxxxxxxxx > > Sent: Monday, September 7, 2015 10:05:23 PM > > Subject: Re: Huge memory usage spike in OSD on hammer/giant > > > > nope, master/slave, that's why on graph there is only traffic on eth2 > > > > On Mon, 7 Sep 2015 09:01:53 -0400 (EDT), Shinobu Kinjo > > <skinjo@xxxxxxxxxx> wrote: > > > > > Are you using lacp in 10g interfaces? > > > > > > ----- Original Message ----- > > > From: "Mariusz Gronczewski" <mariusz.gronczewski@xxxxxxxxxxxx> > > > To: "Shinobu Kinjo" <skinjo@xxxxxxxxxx> > > > Cc: "Jan Schermer" <jan@xxxxxxxxxxx>, ceph-users@xxxxxxxxxxxxxx > > > Sent: Monday, September 7, 2015 9:58:33 PM > > > Subject: Re: Huge memory usage spike in OSD on hammer/giant > > > > > > that was on 10Gbit interface between OSDs, not from outside, traffic > > > from outside was rather low (eth0/1 public - 2/3 cluster) > > > > > > On Mon, 7 Sep 2015 08:42:09 -0400 (EDT), Shinobu Kinjo > > > <skinjo@xxxxxxxxxx> wrote: > > > > > > > How heavy network traffic was? > > > > > > > > Have you tried to capture that traffic between cluster and public network > > > > to see where such a bunch of traffic came from? > > > > > > > > Shinobu > > > > > > > > ----- Original Message ----- > > > > From: "Jan Schermer" <jan@xxxxxxxxxxx> > > > > To: "Mariusz Gronczewski" <mariusz.gronczewski@xxxxxxxxxxxx> > > > > Cc: ceph-users@xxxxxxxxxxxxxx > > > > Sent: Monday, September 7, 2015 9:17:04 PM > > > > Subject: Re: Huge memory usage spike in OSD on hammer/giant > > > > > > > > Hmm, even network traffic went up. > > > > Nothing in logs on the mons which started 9/4 ~6 AM? > > > > > > > > Jan > > > > > > > > > On 07 Sep 2015, at 14:11, Mariusz Gronczewski <mariusz.gronczewski@xxxxxxxxxxxx> wrote: > > > > > > > > > > On Mon, 7 Sep 2015 13:44:55 +0200, Jan Schermer <jan@xxxxxxxxxxx> wrote: > > > > > > > > > >> Maybe some configuration change occured that now takes effect when you start the OSD? > > > > >> Not sure what could affect memory usage though - some ulimit values maybe (stack size), number of OSD threads (compare the number from this OSD to the rest of OSDs), fd cache size. Look in /proc and compare everything. > > > > >> Also look in "ceph osd tree" - didn't someone touch it while you were gone? > > > > >> > > > > >> Jan > > > > >> > > > > > > > > > >> number of OSD threads (compare the number from this OSD to the rest of > > > > > OSDs), > > > > > > > > > > it occured on all OSDs, and it looked like that > > > > > http://imgur.com/IIMIyRG > > > > > > > > > > sadly I was on vacation so I didnt manage to catch it before ;/ but I'm > > > > > sure there was no config change > > > > > > > > > > > > > > >>> On 07 Sep 2015, at 13:40, Mariusz Gronczewski <mariusz.gronczewski@xxxxxxxxxxxx> wrote: > > > > >>> > > > > >>> On Mon, 7 Sep 2015 13:02:38 +0200, Jan Schermer <jan@xxxxxxxxxxx> wrote: > > > > >>> > > > > >>>> Apart from bug causing this, this could be caused by failure of other OSDs (even temporary) that starts backfills. > > > > >>>> > > > > >>>> 1) something fails > > > > >>>> 2) some PGs move to this OSD > > > > >>>> 3) this OSD has to allocate memory for all the PGs > > > > >>>> 4) whatever fails gets back up > > > > >>>> 5) the memory is never released. > > > > >>>> > > > > >>>> A similiar scenario is possible if for example someone confuses "ceph osd crush reweight" with "ceph osd reweight" (yes, this happened to me :-)). > > > > >>>> > > > > >>>> Did you try just restarting the OSD before you upgraded it? > > > > >>> > > > > >>> stopped, upgraded, started. it helped a bit ( <3GB per OSD) but beside > > > > >>> that nothing changed. I've tried to wait till it stops eating CPU then > > > > >>> restart it but it still eats >2GB of memory which means I can't start > > > > >>> all 4 OSDs at same time ;/ > > > > >>> > > > > >>> I've also added noin,nobackfill,norecover flags but that didnt help > > > > >>> > > > > >>> it is suprising for me because before all 4 OSDs total ate less than > > > > >>> 2GBs of memory so I though I have enough headroom, and we did restart > > > > >>> machines and removed/added os to test if recovery/rebalance goes fine > > > > >>> > > > > >>> it also does not have any external traffic at the moment > > > > >>> > > > > >>> > > > > >>>>> On 07 Sep 2015, at 12:58, Mariusz Gronczewski <mariusz.gronczewski@xxxxxxxxxxxx> wrote: > > > > >>>>> > > > > >>>>> Hi, > > > > >>>>> > > > > >>>>> over a weekend (was on vacation so I didnt get exactly what happened) > > > > >>>>> our OSDs started eating in excess of 6GB of RAM (well RSS), which was a > > > > >>>>> problem considering that we had only 8GB of ram for 4 OSDs (about 700 > > > > >>>>> pgs per osd and about 70GB space used. So spam of coredumps and OOMs > > > > >>>>> blocked the osds down to unusabiltity. > > > > >>>>> > > > > >>>>> I then upgraded one of OSDs to hammer which made it a bit better (~2GB > > > > >>>>> per osd) but still much higher usage than before. > > > > >>>>> > > > > >>>>> any ideas what would be a reason for that ? logs are mostly full on > > > > >>>>> OSDs trying to recover and timed out heartbeats > > > > >>>>> > > > > >>>>> -- > > > > >>>>> Mariusz Gronczewski, Administrator > > > > >>>>> > > > > >>>>> Efigence S. A. > > > > >>>>> ul. Wołoska 9a, 02-583 Warszawa > > > > >>>>> T: [+48] 22 380 13 13 > > > > >>>>> F: [+48] 22 380 13 14 > > > > >>>>> E: mariusz.gronczewski@xxxxxxxxxxxx > > > > >>>>> <mailto:mariusz.gronczewski@xxxxxxxxxxxx> > > > > >>>>> _______________________________________________ > > > > >>>>> ceph-users mailing list > > > > >>>>> ceph-users@xxxxxxxxxxxxxx > > > > >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > >>>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> -- > > > > >>> Mariusz Gronczewski, Administrator > > > > >>> > > > > >>> Efigence S. A. > > > > >>> ul. Wołoska 9a, 02-583 Warszawa > > > > >>> T: [+48] 22 380 13 13 > > > > >>> F: [+48] 22 380 13 14 > > > > >>> E: mariusz.gronczewski@xxxxxxxxxxxx > > > > >>> <mailto:mariusz.gronczewski@xxxxxxxxxxxx> > > > > >> > > > > > > > > > > > > > > > > > > > > -- > > > > > Mariusz Gronczewski, Administrator > > > > > > > > > > Efigence S. A. > > > > > ul. Wołoska 9a, 02-583 Warszawa > > > > > T: [+48] 22 380 13 13 > > > > > F: [+48] 22 380 13 14 > > > > > E: mariusz.gronczewski@xxxxxxxxxxxx > > > > > <mailto:mariusz.gronczewski@xxxxxxxxxxxx> > > > > > > > > _______________________________________________ > > > > ceph-users mailing list > > > > ceph-users@xxxxxxxxxxxxxx > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > > > > > > > -- Mariusz Gronczewski, Administrator Efigence S. A. ul. Wołoska 9a, 02-583 Warszawa T: [+48] 22 380 13 13 F: [+48] 22 380 13 14 E: mariusz.gronczewski@xxxxxxxxxxxx <mailto:mariusz.gronczewski@xxxxxxxxxxxx>
Attachment:
pgpdGtjjFpNkB.pgp
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
