This was down to the tx buffer size on the network card i was using. It was an Intel 82547EI gigabit Card using the e1000 driver and operating at 100mbit. The tx buffer was set to 256 which caused this huge delay. The minimum the driver lets me reduce the tx buffer size using ethtool is 80. By reducing the tx ring buffer to 80, the delay when there is full link utilisation and a maximum queue of 10 packets was reduced from 30ms to 10ms. The 3com 3c59x vortex driver uses a tx buffer of 16. I reduced the tx to 16 on the e1000 driver, but the max throughput i could achieve on the interface went down. Has anyone experimented with reducing the size of the tx buffer on this card to get a good balance between delay and throughput ? Jonathan On Tue, 2005-10-11 at 22:04 +0100, Jonathan Lynch wrote: > I have a router with 3 network interfaces like in the following ASCII > diagram below. All interfaces are 100mbit. There is tcp traffic being > sent from net1 to net3 and from net2 to net3 and the tcp connections > consume as much bandwidth as possible. There is a pfifo queue on the > egress interface eth0 of the core router with a limit of 10 packets. > > > net1 --> (eth1) router (eth0) -> net3 > (eth2) > ^ > | > net 2 > > I police traffic on the edge of net1 to 48.4375 Mbit and shape the > traffic on exit of net 2 to 48.4375 Mbit. There are no packets in the > queue of the egress interface eth0 of the router at any stage. (every > packet is enqueued by pfifo_enqueue() to an empty queue. I have > confirmed this by adding adding a counter in sch_fifo.c that is > incremented every time there is a packet in the queue when a new packet > is enqueued.) The delay is at a maximum of 2ms. > > When I increase the policing rate and shaping rates to 48.4687. The > combined increase is 31.2 kbit which is very small. there are some > packets queued for a short period and some dropped which clears the > queue. The maximum number of packets dropped was 20 per second. But the > delay goes up to 30ms. > > check out the graphs at > http://frink.nuigalway.ie/~jlynch/queue/ > > > I cant seem to explain this. Even if the queue was full all the time and > each packet was of maximum size, the delay imposed by queueing should be > a maximum of 10 * 1500 * 8 /100,000,000 which equals 1ms. > > How can so much delay be added by such a small increase in the > throughput coming from net1 and net2 ? > > I would appreciate if someone could explain it to me. > > Btw im using a stratum 1 NTP server on the same LAN to ensure > measurement accuracy. > > > Jonathan > > _______________________________________________ > LARTC mailing list > LARTC@xxxxxxxxxxxxxxx > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc _______________________________________________ LARTC mailing list LARTC@xxxxxxxxxxxxxxx http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
