On Wed, 19 May 2010 05:36:42 pm Avi Kivity wrote: > > Note that this is a exclusive->shared->exclusive bounce only, too. > > > > A bounce is a bounce. I tried to measure this to show that you were wrong, but I was only able to show that you're right. How annoying. Test code below. > Virtio is already way too bouncy due to the indirection between the > avail/used rings and the descriptor pool. I tried to do a more careful analysis below, and I think this is an overstatement. > A device with out of order > completion (like virtio-blk) will quickly randomize the unused > descriptor indexes, so every descriptor fetch will require a bounce. > > In contrast, if the rings hold the descriptors themselves instead of > pointers, we bounce (sizeof(descriptor)/cache_line_size) cache lines for > every descriptor, amortized. We already have indirect, this would be a logical next step. So let's think about it. The avail ring would contain 64 bit values, the used ring would contain indexes into the avail ring. So client writes descriptor page and adds to avail ring, then writes to index. Server reads index, avail ring, descriptor page (3). Writes used entry (1). Updates last_used (1). Client reads used (1), derefs avail (1), updates last_used (1), cleans descriptor page (1). That's 9 cacheline transfers, worst case. Best case of a half-full ring in steady state, assuming 128-byte cache lines, the avail ring costs are 1/16, the used entry is 1/64. This drops it to 6 and 9/64 transfers. (Note, the current scheme adds 2 more cacheline transfers, for the descriptor table, worst case. Assuming indirect, we get 2/8 xfer best case. Either way, it's not the main source of cacheline xfers). Can we do better? The obvious idea is to try to get rid of last_used and used, and use the ring itself. We would use an invalid entry to mark the head of the ring. Any other thoughts? Rusty. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
