On 2/23/22 1:01 PM, Bob Pearson wrote:
On 2/22/22 22:43, Guoqing Jiang wrote:
On 2/23/22 12:58 AM, Pearson, Robert B wrote:
After investigation, seems the culprit is commit 647bf13ce944 ("RDMA/rxe:
Create duplicate mapping tables for FMRs"). The problem is
mr_check_range returns -EFAULT after find iova and length are not
valid, so connection between two VMs can't be established.
Revert the commit manually or apply below temporary change, rxe works
again with rnbd/rtrs though I don't think it is the right thing to do.
Could experts provide a proper solution? Thanks.
This patch fixed failures in blktests and srp which were discussed at length. See e.g.
https://lore.kernel.org/linux-rdma/20210907163939.GW1200268@xxxxxxxx/
Thanks for the link, which reminds me the always_invalidate parameter in rtrs_server.
and related messages. The conclusion was that two mappings were required. One owned by the
driver and one by the 'hardware', i.e. bottom half in the rxe case, allowing updating a new mapping
while the old one is still active and then switching them.
If this case has iova and length not valid as indicated is there a problem with the test case?
And after disable always_invalidate (which is enabled by default), rnbd/rtrs over roce
works either. So I suppose there might be potential issue for always_invalidate=Y in
rtrs server side since invalidate works for srp IIUC, @Jinpu.
Thanks,
Guoqing
It would help to understand what you are running. My main concern is that mr_check_range
shouldn't fail unless there is something very wrong.
Let me paste it again, I just try to map server block device to client
side, specifically the bold
line.
1. VM (server)
modprobe rdma_rxe
rdma link add rxe0 type rxe netdev ens3
modprobe rnbd-server
2. VM (client)
modprobe rdma_rxe
rdma link add rxe0 type rxe netdev ens3
modprobe rnbd-client
*echo "sessname=bla path=ip:$serverip
device_path=$block_device_in_server" >
/sys/devices/virtual/rnbd-client/ctl/map_device*
Thanks,
Guoqing
