On Thu, Dec 01, 2011 at 01:27:10AM +0200, Ohad Ben-Cohen wrote: > On Wed, Nov 30, 2011 at 6:24 PM, Ohad Ben-Cohen <ohad@xxxxxxxxxx> wrote: > > On Wed, Nov 30, 2011 at 6:15 PM, Michael S. Tsirkin <mst@xxxxxxxxxx> wrote: > >> How are the rings mapped? normal memory, right? > > > > No, device memory. > > Ok, I have more info. > > Originally remoteproc was mapping the rings using ioremap, and that > meant ARM Device memory. > > Recently, though, we moved to CMA (allocating memory for the rings via > dma_alloc_coherent), and that isn't Device memory anymore: it's > uncacheable Normal memory (on ARM v6+). And these accesses need to be ordered with DSB? Or DMB? > We still require mandatory barriers though: one very reproducible > problem I personally face is that the avail index doesn't get updated > before the kick. Aha! The *kick* really is MMIO. So I think we do need a mandatory barrier before the kick. Maybe we need it for virtio-pci as well (not on kvm, naturally :) Off-hand this seems to belong in the transport layer but need to think about it. > As a result, the remote processor misses a buffer > that was just added (the kick wakes it up only to find that the avail > index wasn't changed yet). In this case, it probably happens because > the mailbox, used to kick the remote processor, is mapped as Device > memory, and therefore the kick can be reordered before the updates to > the ring can be observed. > > I did get two additional reports about reordering issues, on different > setups than my own, and which I can't personally reproduce: the one > I've described earlier (avail index gets updated before the avail > array) and one in the receive path (reading a used buffer which we > already read). I couldn't personally verify those, but both issues > were reported to be gone when mandatory barriers were used. Hmm. So it's a hint that something is wrong with memory but not what's wrong exactly. > I expect those reports only to increase: the diversity of platforms > that are now looking into adopting virtio for this kind of > inter-process communication is quite huge, with several different > architectures and even more hardware implementations on the way (not > only ARM). > > Thanks, > Ohad. Right. We need to be very careful with memory, it's a tricky field. One known problem with virtio is its insistance on using native endian-ness for some fields. If power is used, we'll have to fix this. -- MST -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
