On Tue, Mar 19, 2024 at 06:08:27PM +1000, Gavin Shan wrote:
> On 3/19/24 17:09, Michael S. Tsirkin wrote:
> > On Tue, Mar 19, 2024 at 04:49:50PM +1000, Gavin Shan wrote:
> > >
> > > On 3/19/24 16:43, Michael S. Tsirkin wrote:
> > > > On Tue, Mar 19, 2024 at 04:38:49PM +1000, Gavin Shan wrote:
> > > > > On 3/19/24 16:09, Michael S. Tsirkin wrote:
> > > > >
> > > > > > > > > diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
> > > > > > > > > index 49299b1f9ec7..7d852811c912 100644
> > > > > > > > > --- a/drivers/virtio/virtio_ring.c
> > > > > > > > > +++ b/drivers/virtio/virtio_ring.c
> > > > > > > > > @@ -687,9 +687,15 @@ static inline int virtqueue_add_split(struct virtqueue *_vq,
> > > > > > > > > avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1);
> > > > > > > > > vq->split.vring.avail->ring[avail] = cpu_to_virtio16(_vq->vdev, head);
> > > > > > > > > - /* Descriptors and available array need to be set before we expose the
> > > > > > > > > - * new available array entries. */
> > > > > > > > > - virtio_wmb(vq->weak_barriers);
> > > > > > > > > + /*
> > > > > > > > > + * Descriptors and available array need to be set before we expose
> > > > > > > > > + * the new available array entries. virtio_wmb() should be enough
> > > > > > > > > + * to ensuere the order theoretically. However, a stronger barrier
> > > > > > > > > + * is needed by ARM64. Otherwise, the stale data can be observed
> > > > > > > > > + * by the host (vhost). A stronger barrier should work for other
> > > > > > > > > + * architectures, but performance loss is expected.
> > > > > > > > > + */
> > > > > > > > > + virtio_mb(false);
> > > > > > > > > vq->split.avail_idx_shadow++;
> > > > > > > > > vq->split.vring.avail->idx = cpu_to_virtio16(_vq->vdev,
> > > > > > > > > vq->split.avail_idx_shadow);
> > > > > > > >
> > > > > > > > Replacing a DMB with a DSB is _very_ unlikely to be the correct solution
> > > > > > > > here, especially when ordering accesses to coherent memory.
> > > > > > > >
> > > > > > > > In practice, either the larger timing different from the DSB or the fact
> > > > > > > > that you're going from a Store->Store barrier to a full barrier is what
> > > > > > > > makes things "work" for you. Have you tried, for example, a DMB SY
> > > > > > > > (e.g. via __smb_mb()).
> > > > > > > >
> > > > > > > > We definitely shouldn't take changes like this without a proper
> > > > > > > > explanation of what is going on.
> > > > > > > >
> > > > > > >
> > > > > > > Thanks for your comments, Will.
> > > > > > >
> > > > > > > Yes, DMB should work for us. However, it seems this instruction has issues on
> > > > > > > NVidia's grace-hopper. It's hard for me to understand how DMB and DSB works
> > > > > > > from hardware level. I agree it's not the solution to replace DMB with DSB
> > > > > > > before we fully understand the root cause.
> > > > > > >
> > > > > > > I tried the possible replacement like below. __smp_mb() can avoid the issue like
> > > > > > > __mb() does. __ndelay(10) can avoid the issue, but __ndelay(9) doesn't.
> > > > > > >
> > > > > > > static inline int virtqueue_add_split(struct virtqueue *_vq, ...)
> > > > > > > {
> > > > > > > :
> > > > > > > /* Put entry in available array (but don't update avail->idx until they
> > > > > > > * do sync). */
> > > > > > > avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1);
> > > > > > > vq->split.vring.avail->ring[avail] = cpu_to_virtio16(_vq->vdev, head);
> > > > > > >
> > > > > > > /* Descriptors and available array need to be set before we expose the
> > > > > > > * new available array entries. */
> > > > > > > // Broken: virtio_wmb(vq->weak_barriers);
> > > > > > > // Broken: __dma_mb();
> > > > > > > // Work: __mb();
> > > > > > > // Work: __smp_mb();
> > > > > > > // Work: __ndelay(100);
> > > > > > > // Work: __ndelay(10);
> > > > > > > // Broken: __ndelay(9);
> > > > > > >
> > > > > > > vq->split.avail_idx_shadow++;
> > > > > > > vq->split.vring.avail->idx = cpu_to_virtio16(_vq->vdev,
> > > > > > > vq->split.avail_idx_shadow);
> > > > > >
> > > > > > What if you stick __ndelay here?
> > > > > >
> > > > >
> > > > > /* Put entry in available array (but don't update avail->idx until they
> > > > > * do sync). */
> > > > > avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1);
> > > > > vq->split.vring.avail->ring[avail] = cpu_to_virtio16(_vq->vdev, head);
> > > > >
> > > > > /* Descriptors and available array need to be set before we expose the
> > > > > * new available array entries. */
> > > > > virtio_wmb(vq->weak_barriers);
> > > > > vq->split.avail_idx_shadow++;
> > > > > vq->split.vring.avail->idx = cpu_to_virtio16(_vq->vdev,
> > > > > vq->split.avail_idx_shadow);
> > > > > /* Try __ndelay(x) here as Michael suggested
> > > > > *
> > > > > * Work: __ndelay(200); possiblly make it hard to reproduce
> > > > > * Broken: __ndelay(100);
> > > > > * Broken: __ndelay(20);
> > > > > * Broken: __ndelay(10);
> > > > > */
> > > > > __ndelay(200);
> > > >
> > > > So we see that just changing the timing masks the race.
> > > > What are you using on the host side? vhost or qemu?
> > > >
> > >
> > > __ndelay(200) may make the issue harder to be reproduce as I understand.
> > > More delays here will give vhost relief, reducing the race.
> > >
> > > The issue is only reproducible when vhost is turned on. Otherwise, we
> > > aren't able to hit the issue.
> > >
> > > -netdev tap,id=vnet0,vhost=true,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
> > > -device virtio-net-pci,bus=pcie.8,netdev=vnet0,mac=52:54:00:f1:26:b0
> >
> >
> > Given it's vhost, it's also possible that the issue is host side.
> > I wonder what happens if we stick a delay or a stronger barrier
> > in vhost.c - either here:
> >
> > /* Make sure buffer is written before we update index. */
> > smp_wmb();
> >
> >
> > or here:
> >
> > /* Only get avail ring entries after they have been
> > * exposed by guest.
> > */
> > smp_rmb();
> >
> > ?
> >
>
> It's possible. However, I still can hit the issue after both of them are
> replaed with '__mb()'. So the issue seems on the guest side, where the
> written data isn't observed in time by the CPU on far end (for vhost worker).
>
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index 045f666b4f12..327b68d21b02 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -2529,7 +2529,8 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
> /* Only get avail ring entries after they have been
> * exposed by guest.
> */
> - smp_rmb();
> + // smp_rmb();
> + __mb();
> }
> /* Grab the next descriptor number they're advertising, and increment
> @@ -2703,7 +2704,9 @@ int vhost_add_used_n(struct vhost_virtqueue *vq, struct vring_used_elem *heads,
> r = __vhost_add_used_n(vq, heads, count);
> /* Make sure buffer is written before we update index. */
> - smp_wmb();
> + // smp_wmb();
> + __mb();
> +
>
> From the guest:
>
> [ 14.102608] virtio_net virtio0: output.0:id 80 is not a head!
You realize however that for that to happen, first vhost has to
use the wrong index?
Again the fact tweaking code helps is suggestive but does
not lead us to the root cause.
So you should be able to detect the bad index when vhost gets it.
Here's a suggestion: stick a valid flag in desc.
then vhost should notice if it gets a desc with that flag clear.
Like this maybe?
diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index 6f7e5010a673..2789e51b57e6 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -524,7 +524,7 @@ static inline unsigned int virtqueue_add_desc_split(struct virtqueue *vq,
struct vring_desc_extra *extra = vring->split.desc_extra;
u16 next;
- desc[i].flags = cpu_to_virtio16(vq->vdev, flags);
+ desc[i].flags = cpu_to_virtio16(vq->vdev, flags | (0x1 << 5)); /* Mark it valid */
desc[i].addr = cpu_to_virtio64(vq->vdev, addr);
desc[i].len = cpu_to_virtio32(vq->vdev, len);
@@ -721,6 +721,7 @@ static inline int virtqueue_add_split(struct virtqueue *_vq,
i = virtio16_to_cpu(_vq->vdev, desc[i].next);
} else
i = vring_unmap_one_split(vq, i);
+ desc[i].flags = 0x0;
}
free_indirect:
@@ -775,11 +776,13 @@ static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head,
i = head;
while (vq->split.vring.desc[i].flags & nextflag) {
+ vq->split.vring.desc[i].flags = 0x0;
vring_unmap_one_split(vq, i);
i = vq->split.desc_extra[i].next;
vq->vq.num_free++;
}
+ vq->split.vring.desc[i].flags = 0x0;
vring_unmap_one_split(vq, i);
vq->split.desc_extra[i].next = vq->free_head;
vq->free_head = head;
