在 2022/5/3 1:15, Pearson, Robert B 写道:
-----Original Message-----
From: Chengguang Xu <cgxu519@xxxxxxxxxxxx>
Sent: Monday, May 2, 2022 12:39 AM
To: zyjzyj2000@xxxxxxxxx; jgg@xxxxxxxx; leon@xxxxxxxxxx
Cc: linux-rdma@xxxxxxxxxxxxxxx; Chengguang Xu <cgxu519@xxxxxxxxxxxx>
Subject: [RFC PATCH] RDMA/rxe: skip adjusting remote addr for write in retry operation
For write request the remote addr will be sent only with first packet so we don't have to adjust wqe->iova in retry operation.
This is problematic for lossy networks. A very large read request, say 8MiB, sends 2048 packets in response without any acknowledgement
from the requester. If the packet loss rate was 1% the read request would never finish as the probability of sending 2048 packets without
loss is very small. The way the code works today is that the iova is adjusted, and if you are lucky the responder has already finished the
previous read operation and starts over with a new read reply starting with a first packet at iova. If you are less fortunate the previous
read reply has not finished and the responder will continue to work on it until it is finished before looking at the new read request wqe.
the completer will respond to each out of order packet by checking to see if it should start a retry but since it has already done so
it drops the packet. It's messy but one can make forward progress ~100 packets at a time. It would be faster if the responder saw that
the new request replaced the old one and stopped sending packets on the old read. I have no idea how CX NICs do this but just restarting
from scratch seems bad.
I agree that read request indeed needs to adjust iova during retry and
the adjustment(for read) has already done in below logic in req_retry().
if (mask & WR_READ_MASK) {
npsn = (wqe->dma.length - wqe->dma.resid) /
qp->mtu;
wqe->iova += npsn * qp->mtu;
}
For write request, retry will not send new iova because only first write
packet has RXE_RETH_MASK regardless iova adjustment.
Am I missing something?
Thanks,
Chengguang
