On Wed, Oct 06, 2010 at 11:13:31PM +0530, Krishna Kumar2 wrote: > "Michael S. Tsirkin" <mst@xxxxxxxxxx> wrote on 10/05/2010 11:53:23 PM: > > > > > Any idea where does this come from? > > > > Do you see more TX interrupts? RX interrupts? Exits? > > > > Do interrupts bounce more between guest CPUs? > > > > 4. Identify reasons for single netperf BW regression. > > > > > > After testing various combinations of #txqs, #vhosts, #netperf > > > sessions, I think the drop for 1 stream is due to TX and RX for > > > a flow being processed on different cpus. > > > > Right. Can we fix it? > > I am not sure how to. My initial patch had one thread but gave > small gains and ran into limitations once number of sessions > became large. Sure. We will need multiple RX queues, and have a single thread handle a TX and RX pair. Then we need to make sure packets from a given flow on TX land on the same thread on RX. As flows can be hashed differently, for this to work we'll have to expose this info in host/guest interface. But since multiqueue implies host/guest ABI changes anyway, this point is moot. BTW, an interesting approach could be using bonding and multiple virtio-net interfaces. What are the disadvantages of such a setup? One advantage is it can be made to work in existing guests. > > > I did two more tests: > > > 1. Pin vhosts to same CPU: > > > - BW drop is much lower for 1 stream case (- 5 to -8% range) > > > - But performance is not so high for more sessions. > > > 2. Changed vhost to be single threaded: > > > - No degradation for 1 session, and improvement for upto > > > 8, sometimes 16 streams (5-12%). > > > - BW degrades after that, all the way till 128 netperf > sessions. > > > - But overall CPU utilization improves. > > > Summary of the entire run (for 1-128 sessions): > > > txq=4: BW: (-2.3) CPU: (-16.5) RCPU: (-5.3) > > > txq=16: BW: (-1.9) CPU: (-24.9) RCPU: (-9.6) > > > > > > I don't see any reasons mentioned above. However, for higher > > > number of netperf sessions, I see a big increase in retransmissions: > > > > Hmm, ok, and do you see any errors? > > I haven't seen any in any statistics, messages, etc. Herbert, could you help out debugging this increase in retransmissions please? Older mail on netdev in this thread has some numbers that seem to imply that we start hitting retransmissions much more as # of flows goes up. > Also no > retranmissions for txq=1. While it's nice that we have this parameter, the need to choose between single stream and multi stream performance when you start the vm makes this patch much less interesting IMHO. -- MST -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
