First, if you haven't done so, set the MTU on all your GigE interfaces to 9000. This should help a lot as there's a significant amount of per-packet overhead. Hopefully your switch supports this. Also, if you want to take advantage of checksum offloading (doing zero-copy), you'll have to use the sendfile(2) call. There's really no way around this with a sockets API. -John On Fri, 25 Apr 2003, Fred Gray wrote: > Dear linux-net, > > I have recently taken over the development of the data acquisition system > for a physics experiment. The first layer of this system needs to read data > from custom electronics across a VME backplane at about 15 MB/s and forward > it to a back-end machine for online analysis and storage. > > My predecessor invested lots of $$$ in a proprietary interconnection scheme > that has proven to be extremely unstable in actual use. I am trying to replace > it with gigabit Ethernet, but I am having trouble getting acceptable > performance. > > The front-end processors are Motorola MVME 2600 boards, which have 200 MHz > PowerPC 604e CPUs. I have equipped them with SBS Technologies PMC-Gigabit-ST3 > daughtercards, which use the Intel 82545EM chipset. The PMC (PCI Mezzanine > Connector--just PCI in a different form factor) slot is 66 MHz, 64-bit capable. > I am running a 2.4.21-pre6 kernel from the linuxppc_2_4_devel BK tree, > which includes version 4.4.19-k2 of the e1000 driver. The back-end server > is a 1.7 GHz Pentium 4 with an Intel PRO/1000 T Server Adapter, and they are > connected through a D-Link DGS-1008T gigabit switch. > > As measured with netperf and/or ttcp, I get between 80 and 90 Mbps from > the front-end to the back-end server, which is a 1.7 GHz Pentium 4, > even with the socket buffer size and TCP window size set to 256KB. > In other words, I might as well have used the built-in 100BaseT port. > The same benchmarks give around 350 Mbps between the server and another > gigabit-equipped PC connected through the same switch, so I don't think > my test is intrinsically flawed. > > While performing these benchmarks, the CPU utilization of netperf on > the front-end is between 90% and 100%, suggesting that the process of > sending the data is CPU-bound. I suppose that it might be occupied with > computing the TCP checksum. The documentation for the 82545EM chipset > claims that it supports transmit checksum offloading, but the Linux e1000 > driver does not seem to use this feature, only receive checksum offloading. > > Is a 200 MHz PowerPC processor simply not adequate to drive a gigabit link > faster than 80 Mbps, or do I have any options left to try? In our controlled > environment, I could possibly consider hacking the kernel on both ends to > disable checksum computation and verification, but I would certainly prefer > not to have to resort to such extreme measures, and I am in any event not > yet certain that the overhead comes from the checksum. I don't need full > gigabit speed: 150 Mbps would be adequate. > > Thanks in advance to the networking gurus for their advice, > > -- Fred > > -- Fred Gray / Visiting Postdoctoral Researcher -- > -- Department of Physics / University of California, Berkeley -- > -- fegray@socrates.berkeley.edu / phone 510-642-4057 / fax 510-642-9811 -- > - > : send the line "unsubscribe linux-net" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > - : send the line "unsubscribe linux-net" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
