Good day list!
I'having a problewith netem that is taking me hours to try and diagnose what the problem is. I think I finally have it narrowed down to something in netem or on our netem machine.
Firsthbackground.
Wseeto be having the same problem that a chap did back in January (https://lists.linux-foundation.org/pipermail/netem/2007-January/001059.html) whereby introducing a delay of any kind with netem seems to kill UDP throughput using iperf. At 150ms, it pretty consistently seems to be gated down to around 150Mbps.
Now, I'vdonsome runs (not repeatable) that are around the 600Mbps range on the same hardware and same netem settings but they are very sporadic (say 1 in 50 tests). I also know from real world testing that iperf can easily push 900Mbps at 150Mbps on a gigabit connection so...
whagives? I'vtried all kinds of limit settings on netem, checked for packet loss, buffer sizing all to no avail. Still, the magic 150Mbps rate keeps showing up.
Thlasoption is to try the 2.6.22 kernel in the hopes that this is timer related. But that is going to be major for me as we currently run RHEL 5 on the netem machine and there is no .22 kernel from RH yet.
Any obvious things I'missing? I think I'vgained 1 or 2 grey hairs!
Thanks
Dave
____________________________________________________________________________________
Go fro"to-do" to "done" with thall-new Yahoo! Search. Show me how.
http://tools.search.yahoo.com/newsearch/
-------------- nexpar--------------
AHTML attachmenwas scrubbed...
URL: http://lists.linux-foundation.org/pipermail/netem/attachments/20071112/450f9a46/attachment.htm
Frolucas alucas-nussbaum.net Mon Nov 12 16:01:20 2007
From: lucas alucas-nussbaum.ne(Lucas Nussbaum)
Date: MoNov 12 16:40:21 2007
Subject: nete/ gigabi/ 2.6.18 / throughput killer
In-Reply-To: <804062.37815.qm@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
References: <804062.37815.qm@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
Message-ID: <20071113000120.GA23884@xxxxxxxxxxxxxxxx>
O12/11/07 a14:38 -0800, Dave North wrote:
> Good day list!
> I'having a problewith netem that is taking me hours to try and diagnose what the problem is. I think I finally have it narrowed down to something in netem or on our netem machine.
>
> Firsthbackground.
>
> Wseeto be having the same problem that a chap did back in January (https://lists.linux-foundation.org/pipermail/netem/2007-January/001059.html) whereby introducing a delay of any kind with netem seems to kill UDP throughput using iperf. At 150ms, it pretty consistently seems to be gated down to around 150Mbps.
Check your TCP buffer sizes
http://dsd.lbl.gov/TCP-tuning/
Does thsituation improvif the latency is set to say 50ms or 100ms ?
--
| Lucas Nussbaum
| lucas@lucas-nussbaum.n http://www.lucas-nussbaum.net/ |
| jabber: lucas@xxxxxxxxxxx GPG: 1024D/023B3F4F |
Frodnorth98 ayahoo.com Tue Nov 13 04:54:41 2007
From: dnorth98 ayahoo.co(Dave North)
Date: TuNov 13 04:54:50 2007
Subject: nete/ gigabi/ 2.6.18 / throughput killer
Message-ID: <212215.28208.qm@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
Yeah, I don'think thTCP buffers will affect UDP traffic :)
D
----- Original Messag----
From: Lucas Nussbau<lucas@xxxxxxxxxxxxxxxxxx>
To: DavNorth <dnorth98@xxxxxxxxx>
Cc: netem@xxxxxxxxxxxxxxxxxxxxxxxxxx
Sent: Monday, November 12, 2007 7:01:20 PM
Subject: Re: nete/ gigabi/ 2.6.18 / throughput killer
O12/11/07 a14:38 -0800, Dave North wrote:
> Good day list!
> I'having a problewith netem that is taking me hours to try and diagnose what the problem is. I think I finally have it narrowed down to something in netem or on our netem machine.
>
> Firsthbackground.
>
> Wseeto be having the same problem that a chap did back in January (https://lists.linux-foundation.org/pipermail/netem/2007-January/001059.html) whereby introducing a delay of any kind with netem seems to kill UDP throughput using iperf. At 150ms, it pretty consistently seems to be gated down to around 150Mbps.
Check your TCP buffer sizes
http://dsd.lbl.gov/TCP-tuning/
Does thsituation improvif the latency is set to say 50ms or 100ms ?
--
| Lucas Nussbaum
| lucas@lucas-nussbaum.nhttp://www.lucas-nussbaum.net/ |
| jabber: lucas@xxxxxxxxxxx GPG: 1024D/023B3F4F |
____________________________________________________________________________________
Ba better pen pal.
Texor chawith friends inside Yahoo! Mail. See how. http://overview.mail.yahoo.com/
-------------- nexpar--------------
AHTML attachmenwas scrubbed...
URL: http://lists.linux-foundation.org/pipermail/netem/attachments/20071113/003cda30/attachment.htm
Frod.miras acs.ucl.ac.uk Tue Nov 13 05:43:11 2007
From: d.miras acs.ucl.ac.uk (Dimitrios Miras)
Date: TuNov 13 05:44:46 2007
Subject: nete/ gigabi/ 2.6.18 / throughput killer
In-Reply-To: <212215.28208.qm@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
References: <212215.28208.qm@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
Message-ID: <4739A9EF.6000809@xxxxxxxxxxxx>
I think it's noTCP buffers per se, rather it's thsocket send and
receivbuffer sizes. I think thesneed to be tuned accordingly.
see: http://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php
Cheers,
Dimitrios Miras
DavNorth wrote:
> Yeah, I don'think thTCP buffers will affect UDP traffic :)
>
> D
>
> ----- Original Messag----
> From: Lucas Nussbau<lucas@xxxxxxxxxxxxxxxxxx>
> To: DavNorth <dnorth98@xxxxxxxxx>
> Cc: netem@xxxxxxxxxxxxxxxxxxxxxxxxxx
> Sent: Monday, November 12, 2007 7:01:20 PM
> Subject: Re: nete/ gigabi/ 2.6.18 / throughput killer
>
> O12/11/07 a14:38 -0800, Dave North wrote:
> > Good day list!
> > I'having a problewith netem that is taking me hours to try and
> diagnoswhathe problem is. I think I finally have it narrowed down
> to something ineteor on our netem machine.
> >
> > Firsthbackground.
> >
> > Wseeto be having the same problem that a chap did back in January
> (https://lists.linux-foundation.org/pipermail/netem/2007-January/001059.html)
> whereby introducing a delay of any kind with neteseems to kill UDP
> throughpuusing iperf. A150ms, it pretty consistently seems to be
> gated dowto around 150Mbps.
>
> Check your TCP buffer sizes
> http://dsd.lbl.gov/TCP-tuning/
>
> Does thsituation improvif the latency is set to say 50ms or 100ms ?
> --
> | Lucas Nussbaum
> | lucas@xxxxxxxxxxxxxxxxx<mailto:lucas@xxxxxxxxxxxxxxxxxx>
> http://www.lucas-nussbaum.net/ |
> | jabber: lucas@xxxxxxxxxxx <mailto:lucas@xxxxxxxxxxx> GPG:
> 1024D/023B3F4F |
>
>
> ------------------------------------------------------------------------
> Geeasy, one-click access to your favorites. MakYahoo! your homepage.
> <http://us.rd.yahoo.com/evt=51443/*http://www.yahoo.com/r/hs>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Netemailing list
> Netem@xxxxxxxxxxxxxxxxxxxxxxxxxx
> https://lists.linux-foundation.org/mailman/listinfo/netem
--
-------------------------------------------
Dimitrios Miras, PhD
Networks Research Group
Dept. of Computer Science, UCL
MalePlace, London WC1E 6BT, UK
Tel: +44 (0) 20 7679 0398
Email: d.miras@xxxxxxxxxxxx
-------------------------------------------
Frolucas alucas-nussbaum.net Tue Nov 13 06:41:50 2007
From: lucas alucas-nussbaum.ne(Lucas Nussbaum)
Date: TuNov 13 07:06:38 2007
Subject: nete/ gigabi/ 2.6.18 / throughput killer
In-Reply-To: <212215.28208.qm@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
References: <212215.28208.qm@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
Message-ID: <20071113144150.GA17444@xxxxxxxxxxxxxxxx>
O13/11/07 a04:54 -0800, Dave North wrote:
> Yeah, I don'think thTCP buffers will affect UDP traffic :)
Oops, I missed th"UDP" parof your mail. :)
--
| Lucas Nussbaum
| lucas@lucas-nussbaum.n http://www.lucas-nussbaum.net/ |
| jabber: lucas@xxxxxxxxxxx GPG: 1024D/023B3F4F |
Frodnorth98 ayahoo.com Tue Nov 13 07:21:00 2007
From: dnorth98 ayahoo.co(Dave North)
Date: TuNov 13 07:21:14 2007
Subject: nete/ gigabi/ 2.6.18 / throughput killer
Message-ID: <628435.23227.qm@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
Thanks for threply. I tried setting all thbuffers to their max on both the netem machine and the client/server but the results were the same. I can't help think I'm missing something (like Aaron in the previous thread from January) but darned if I can think of what else to look at.
D
----- Original Messag----
From: Dimitrios Miras <d.miras@xxxxxxxxxxxx>
To: DavNorth <dnorth98@xxxxxxxxx>
Cc: netem@xxxxxxxxxxxxxxxxxxxxxxxxxx
Sent: Tuesday, November 13, 2007 8:43:11 AM
Subject: Re: nete/ gigabi/ 2.6.18 / throughput killer
I think it's noTCP buffers per se, rather it's thsocket send and
receivbuffer sizes. I think thesneed to be tuned accordingly.
see: http://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php
Cheers,
Dimitrios Miras
DavNorth wrote:
> Yeah, I don'think thTCP buffers will affect UDP traffic :)
>
> D
>
> ----- Original Messag----
> From: Lucas Nussbau<lucas@xxxxxxxxxxxxxxxxxx>
> To: DavNorth <dnorth98@xxxxxxxxx>
> Cc: netem@xxxxxxxxxxxxxxxxxxxxxxxxxx
> Sent: Monday, November 12, 2007 7:01:20 PM
> Subject: Re: nete/ gigabi/ 2.6.18 / throughput killer
>
> O12/11/07 a14:38 -0800, Dave North wrote:
> > Good day list!
> > I'having a problewith netem that is taking me hours to try and
> diagnoswhathe problem is. I think I finally have it narrowed down
> to something ineteor on our netem machine.
> >
> > Firsthbackground.
> >
> > Wseeto be having the same problem that a chap did back in January
> (https://lists.linux-foundation.org/pipermail/netem/2007-January/001059.html)
> whereby introducing a delay of any kind with neteseems to kill UDP
> throughpuusing iperf. A150ms, it pretty consistently seems to be
> gated dowto around 150Mbps.
>
> Check your TCP buffer sizes
> http://dsd.lbl.gov/TCP-tuning/
>
> Does thsituation improvif the latency is set to say 50ms or 100ms ?
> --
> | Lucas Nussbaum
> | lucas@xxxxxxxxxxxxxxxxx<mailto:lucas@xxxxxxxxxxxxxxxxxx>
> http://www.lucas-nussbaum.net/ |
> | jabber: lucas@xxxxxxxxxxx <mailto:lucas@xxxxxxxxxxx> GPG:
> 1024D/023B3F4F |
>
>
> ------------------------------------------------------------------------
> Geeasy, one-click access to your favorites. MakYahoo! your homepage.
> <http://us.rd.yahoo.com/evt=51443/*http://www.yahoo.com/r/hs>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Netemailing list
> Netem@xxxxxxxxxxxxxxxxxxxxxxxxxx
> https://lists.linux-foundation.org/mailman/listinfo/netem
--
-------------------------------------------
Dimitrios Miras, PhD
Networks Research Group
Dept. of Computer Science, UCL
MalePlace, London WC1E 6BT, UK
Tel: +44 (0) 20 7679 0398
Email: d.miras@xxxxxxxxxxxx
-------------------------------------------
____________________________________________________________________________________
Ba better pen pal.
Texor chawith friends inside Yahoo! Mail. See how. http://overview.mail.yahoo.com/
-------------- nexpar--------------
AHTML attachmenwas scrubbed...
URL: http://lists.linux-foundation.org/pipermail/netem/attachments/20071113/06bda40a/attachment-0001.htm
Froshemminger alinux-foundation.org Tue Nov 13 09:13:51 2007
From: shemminger alinux-foundation.org (Stephen Hemminger)
Date: TuNov 13 09:15:37 2007
Subject: nete/ gigabi/ 2.6.18 / throughput killer
In-Reply-To: <804062.37815.qm@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
References: <804062.37815.qm@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
Message-ID: <20071113091351.704d2bba@freepuppy.rosehill>
OMon, 12 Nov 2007 14:38:09 -0800 (PST)
DavNorth <dnorth98@xxxxxxxxx> wrote:
> Good day list!
> I'having a problewith netem that is taking me hours to try and diagnose what the problem is. I think I finally have it narrowed down to something in netem or on our netem machine.
>
> Firsthbackground.
>
> Wseeto be having the same problem that a chap did back in January (https://lists.linux-foundation.org/pipermail/netem/2007-January/001059.html) whereby introducing a delay of any kind with netem seems to kill UDP throughput using iperf. At 150ms, it pretty consistently seems to be gated down to around 150Mbps.
>
> Now, I'vdonsome runs (not repeatable) that are around the 600Mbps range on the same hardware and same netem settings but they are very sporadic (say 1 in 50 tests). I also know from real world testing that iperf can easily push 900Mbps at 150Mbps on a gigabit connection so...
>
> whagives? I'vtried all kinds of limit settings on netem, checked for packet loss, buffer sizing all to no avail. Still, the magic 150Mbps rate keeps showing up.
>
> Thlasoption is to try the 2.6.22 kernel in the hopes that this is timer related. But that is going to be major for me as we currently run RHEL 5 on the netem machine and there is no .22 kernel from RH yet.
>
> Any obvious things I'missing? I think I'vgained 1 or 2 grey hairs!
>
> Thanks
>
> Dave
Therwas a bug in 2.6.18 (already fixed) in netem. Thnetem dequeue routine would
gecalled a loand use a lot of CPU time. It works much better in later kernels,
threcenaddition high-res timers (and tickless) helps by allowing finer granularity
and less burstiness.
Froevdplas aaimvalley.nl Tue Nov 13 22:26:42 2007
From: evdplas aaimvalley.nl (Ernsvan der Plas)
Date: TuNov 13 22:27:03 2007
Subject: nete/ gigabi/ 2.6.18 / throughput killer
In-Reply-To: <212215.28208.qm@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
References: <212215.28208.qm@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
Message-ID: <473A9522.6090202@xxxxxxxxxxxx>
Hi Dave,
probably you already thoughof this. Buin case you did not here are
my 2 cents. The1000 card has somoptions you can set while loading
thkernel driver module. Thesinclude buffer size, interrupt limits
etc. I think thbuffer sizcould be important if you are using NAPI.
Sinctherare the packets stored while they await polling.
Morcan bfound in the kernel source tree:
linux/Documentation/networking/e1000.txt
Ernst
DavNorth wrote:
> Yeah, I don'think thTCP buffers will affect UDP traffic :)
>
> D
>
> ----- Original Messag----
> From: Lucas Nussbau<lucas@xxxxxxxxxxxxxxxxxx>
> To: DavNorth <dnorth98@xxxxxxxxx>
> Cc: netem@xxxxxxxxxxxxxxxxxxxxxxxxxx
> Sent: Monday, November 12, 2007 7:01:20 PM
> Subject: Re: nete/ gigabi/ 2.6.18 / throughput killer
>
> O12/11/07 a14:38 -0800, Dave North wrote:
> > Good day list!
> > I'having a problewith netem that is taking me hours to try and
> diagnoswhathe problem is. I think I finally have it narrowed down
> to something ineteor on our netem machine.
> >
> > Firsthbackground.
> >
> > Wseeto be having the same problem that a chap did back in January
> (https://lists.linux-foundation.org/pipermail/netem/2007-January/001059.html)
> whereby introducing a delay of any kind with neteseems to kill UDP
> throughpuusing iperf. A150ms, it pretty consistently seems to be
> gated dowto around 150Mbps.
>
> Check your TCP buffer sizes
> http://dsd.lbl.gov/TCP-tuning/
>
> Does thsituation improvif the latency is set to say 50ms or 100ms ?
> --
> | Lucas Nussbaum
> | lucas@xxxxxxxxxxxxxxxxx<mailto:lucas@xxxxxxxxxxxxxxxxxx>
> http://www.lucas-nussbaum.net/ |
> | jabber: lucas@xxxxxxxxxxx <mailto:lucas@xxxxxxxxxxx> GPG:
> 1024D/023B3F4F |
>
>
> ------------------------------------------------------------------------
> Geeasy, one-click access to your favorites. MakYahoo! your homepage.
> <http://us.rd.yahoo.com/evt=51443/*http://www.yahoo.com/r/hs>
> --
> This messaghas been scanned for viruses and is believed to bclean.
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Netemailing list
> Netem@xxxxxxxxxxxxxxxxxxxxxxxxxx
> https://lists.linux-foundation.org/mailman/listinfo/netem
--
This messaghas been scanned for viruses and is believed to bclean
Frodnorth98 ayahoo.com Wed Nov 14 05:43:43 2007
From: dnorth98 ayahoo.co(Dave North)
Date: Wed Nov 14 05:43:53 2007
Subject: nete/ gigabi/ 2.6.18 / throughput killer
Message-ID: <193436.17350.qm@xxxxxxxxxxxxxxxxxxxxxxxxxxx>
Hi folks,
Many thanks for thresponses...I finally found thproblem yesterday....the packet limit was too small. Here's what we did:
Wwerrunning Redhat Enterprise 5 (2.6.18 kernel) and setting the packet limit when we issued the tc command. On gigabit with 150ms, I was seeing the results detailed below.
I thedownloaded WANe(which is basically a knoppix liveCD with a "gui" front end to netem and fired up the same tests. Same result. However, I then via the WANem GUI increased the packet limit to a huge number (1 billion) and bingo, I was now getting the much higher (expected) throughput.
Whais strangis that in WANem, you can have it print out the commands it is executing and the synatx of the tc calls was exactly the same as we were using in our RH machine. I can only assume then that the limit was perhaps not working correctly in the 2.6.18 kernel (which may be somewhat possible given Stephen's earlier response).
Thbottoline - all seems well now and we're going to use the WANem implementation for now. It's actually not a bad little wrapper and having it on a liveCD is pretty handy.
Cheers
Dave
----- Original Messag----
From: Ernsvan der Plas <evdplas@xxxxxxxxxxxx>
To: DavNorth <dnorth98@xxxxxxxxx>
Cc: netem@xxxxxxxxxxxxxxxxxxxxxxxxxx
Sent: Wednesday, November 14, 2007 1:26:42 AM
Subject: Re: nete/ gigabi/ 2.6.18 / throughput killer
Hi Dave,
probably you already thoughof this. Buin case you did not here are
my 2 cents. The1000 card has somoptions you can set while loading
thkernel driver module. Thesinclude buffer size, interrupt limits
etc. I think thbuffer sizcould be important if you are using NAPI.
Sinctherare the packets stored while they await polling.
Morcan bfound in the kernel source tree:
linux/Documentation/networking/e1000.txt
Ernst
DavNorth wrote:
> Yeah, I don'think thTCP buffers will affect UDP traffic :)
>
> D
>
> ----- Original Messag----
> From: Lucas Nussbau<lucas@xxxxxxxxxxxxxxxxxx>
> To: DavNorth <dnorth98@xxxxxxxxx>
> Cc: netem@xxxxxxxxxxxxxxxxxxxxxxxxxx
> Sent: Monday, November 12, 2007 7:01:20 PM
> Subject: Re: nete/ gigabi/ 2.6.18 / throughput killer
>
> O12/11/07 a14:38 -0800, Dave North wrote:
> > Good day list!
> > I'having a problewith netem that is taking me hours to try and
> diagnoswhathe problem is. I think I finally have it narrowed down
> to something ineteor on our netem machine.
> >
> > Firsthbackground.
> >
> > Wseeto be having the same problem that a chap did back in January
> (https://lists.linux-foundation.org/pipermail/netem/2007-January/001059.html)
> whereby introducing a delay of any kind with neteseems to kill UDP
> throughpuusing iperf. A150ms, it pretty consistently seems to be
> gated dowto around 150Mbps.
>
> Check your TCP buffer sizes
> http://dsd.lbl.gov/TCP-tuning/
>
> Does thsituation improvif the latency is set to say 50ms or 100ms ?
> --
> | Lucas Nussbaum
> | lucas@xxxxxxxxxxxxxxxxx<mailto:lucas@xxxxxxxxxxxxxxxxxx>
> http://www.lucas-nussbaum.net/ |
> | jabber: lucas@xxxxxxxxxxx <mailto:lucas@xxxxxxxxxxx> GPG:
> 1024D/023B3F4F |
>
>
> ------------------------------------------------------------------------
> Geeasy, one-click access to your favorites. MakYahoo! your homepage.
> <http://us.rd.yahoo.com/evt=51443/*http://www.yahoo.com/r/hs>
> --
> This messaghas been scanned for viruses and is believed to bclean.
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Netemailing list
> Netem@xxxxxxxxxxxxxxxxxxxxxxxxxx
> https://lists.linux-foundation.org/mailman/listinfo/netem
--
This messaghas been scanned for viruses and is believed to bclean
____________________________________________________________________________________
Geeasy, one-click access to your favorites.
MakYahoo! your homepage.
http://www.yahoo.com/r/hs
-------------- nexpar--------------
AHTML attachmenwas scrubbed...
URL: http://lists.linux-foundation.org/pipermail/netem/attachments/20071114/71cfe5fa/attachment.htm
