On 2020/12/18 下午7:38, Stefano Garzarella wrote:
On Mon, Nov 16, 2020 at 11:37:48AM +0800, Jason Wang wrote:
On 2020/11/13 下午9:47, Stefano Garzarella wrote:
Thanks to Max that started this work!
I took his patches, and extended the block simulator a bit.
This series moves the network device simulator in a new module
(vdpa_sim_net) and leaves the generic functions in the vdpa_sim core
module, allowing the possibility to add new vDPA device simulators.
Then we added a new vdpa_sim_blk module to simulate a block device.
I'm not sure about patch 11 ("vringh: allow vringh_iov_xfer() to skip
bytes when ptr is NULL"), maybe we can add a new functions instead of
modify vringh_iov_xfer().
As Max reported, I'm also seeing errors with vdpa_sim_blk related to
iotlb and vringh when there is high load, these are some of the error
messages I can see randomly:
vringh: Failed to access avail idx at 00000000e8deb2cc
vringh: Failed to read head: idx 6289 address 00000000e1ad1d50
vringh: Failed to get flags at 000000006635d7a3
virtio_vdpa vdpa0: vringh_iov_push_iotlb() error: -14 offset:
0x2840000 len: 0x20000
virtio_vdpa vdpa0: vringh_iov_pull_iotlb() error: -14 offset:
0x58ee000 len: 0x3000
These errors should all be related to the fact that iotlb_translate()
fails with -EINVAL, so it seems that we miss some mapping.
Is this only reproducible when there's multiple co-current accessing
of IOTLB? If yes, it's probably a hint that some kind of
synchronization is still missed somewhere.
It might be useful to log the dma_map/unmp in both virtio_ring and
vringh to see who is missing the map.
Just an update about these issues with vdpa-sim-blk.
I've been focusing a little bit on these failures over the last few
days and have found two issues related to the IOTLB/IOMMU:
1. Some requests coming from the block layer fills the SG list with
multiple buffers that had the same physical address. This happens for
example while using 'mkfs', at some points multiple sectors are zeroed
so multiple SG elements point to the same physical page that is zeroed.
Since we are using vhost_iotlb_del_range() in the
vdpasim_unmap_page(), this removes all the overlapped ranges. I fixed
removing a single map in vdpasim_unmap_page(), but has an alternative
we can implement some kind of reference counts.
I think we need to do what hardware do. So using refcount is probably
not a good ida.
2. There was a race between dma_map/unmap and the worker thread, since
both are accessing the IOMMU. Taking the iommu_lock while using
vhost_iotlb_* API in the worker thread fixes the "vringh: Failed to *"
issues.
Whit these issues fixed the vdpa-blk simulator seems to work well.
I'll send the patches next week or after the break.
Good to know this.
Thanks
Thanks,
Stefano
_______________________________________________
Virtualization mailing list
Virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
