Hi Daniel, On Wed, Mar 18, 2009 at 7:10 PM, Daniel Lezcano <dlezcano@xxxxxxxxxx> wrote: > Ryousei Takano wrote: >> >> Hi all, >> >> I am evaluating the networking performance of lxc on 10 Gigabit Ethernet >> by >> using netperf benchmark. > > Thanks for doing benchmarking. > I did two years ago similar tests and there is an analysis of the > performances at: > http://lxc.sourceforge.net/network/benchs.php > > It is not up to date, but that will give you some clues of what is happening > with this overhead. > I am using VServer because other virtualization mechanisms, including OpenVZ, Xen, and KVM cannot fully utilize the network bandwidth of 10 GbE. Here are the results of netperf bencmark: vanilla (2.6.27-9) 9525.94 Vserver (2.6.27.10) 9521.79 OpenVZ (2.6.27.10) 2049.89 Xen (2.6.26.1) 1011.47 KVM (2.6.27-9) 1022.42 Now I am interesting to use LXC instead of VServer. >> Using a macvlan device, the throughput was 9.6 Gbps. But, using a veth >> device, >> the throughput was only 2.7 Gbps. > > Yeah, definitively the macvlan interfaces is the best in terms of > performances but with the restriction of not being able to communicate > between containers on the same hosts. > This restriction is not so big issue for my purpose. > There are some discussions around that: > > http://marc.info/?l=linux-netdev&m=123643508124711&w=2 > > The veth is a virtual device hence it has not offloading. When the packet > are sent out, the network stack looks at the nic offloading capability which > is not present. So the kernel will compute the checksums instead of letting > the nic to do that either if the packet is transmitted through the physical > nic. This is a well known issue related to network virtualization and xen > has developed a specific network driver: > http://www.cse.psu.edu/~bhuvan/teaching/spring06/papers/xen-net-opt.pdf > >> I think this is because the overhead of bridging devices is high. > > Yes, bridging adds some overhead and AFAIR bridging + netfilter does some > skb copy. > Thank you for pointers. >> I also checked the host OS's performance when I used a veth device. >> I observed a strange phenomenon. >> >> Before issuing lxc-start command, the throughput was 9.6 Gbps. >> Here is the output of brctl show: >> $ brctl show >> bridge name bridge id STP enabled interfaces >> br0 8000.0060dd470d49 no eth1 >> >> After issuing lxc-start command, the throughput decreased to 3.2 Gbps. >> Here is the output of brctl show: >> $ sudo brctl show >> bridge name bridge id STP enabled interfaces >> br0 8000.0060dd470d49 no eth1 >> veth0_7573 >> >> I wonder why the performance is greatly influenced by adding a veth device >> to a bridge device. > > Hmm, good question :) > >> Here is my experimental setting: >> OS: Ubuntu server 8.10 amd64 >> Kernel: 2.6.27-rc8 (checkout from the lxc git repository) > > I would recommend to use the 2.6.29-rc8 vanilla because this kernel does no > longer need patches, a lot of fixes were done in the network namespace and > maybe the bridge has been improved in the meantime :) > I checked out the 2.6.29-rc8 vanilla kernel. The performance after issuing lxc-start improved to 8.7 Gbps! It's a big improvement, while some performance loss remains. Can not we avoid this loss? >> Userland tool: 0.6.0 >> NIC: Myricom Myri-10G >> >> Any comments and suggestions will be appreciate. >> If this list is not proper place to talk about this problem, can anyone >> tell me >> the proper one? > > The performances question is more related to the network virtualization > implementation and should be sent to netdev@ and containers@ (added in the > Cc' of this email), of course people at lxc-devel@ will be interested by > these aspects, so lxc-devel@ is the right mailing list too. > > Thanks for your testings > -- Daniel > Best regards, Ryousei Takano _______________________________________________ Containers mailing list Containers@xxxxxxxxxxxxxxxxxxxxxxxxxx https://lists.linux-foundation.org/mailman/listinfo/containers
