> On Jun 5, 2019, at 11:57 AM, Olga Kornievskaia <aglo@xxxxxxxxx> wrote:
>
> On Wed, Jun 5, 2019 at 8:15 AM Chuck Lever <chuck.lever@xxxxxxxxxx> wrote:
>>
>> The DRC is not working at all after an RPC/RDMA transport reconnect.
>> The problem is that the new connection uses a different source port,
>> which defeats DRC hash.
>>
>> An NFS/RDMA client's source port is meaningless for RDMA transports.
>> The transport layer typically sets the source port value on the
>> connection to a random ephemeral port. The server already ignores it
>> for the "secure port" check. See commit 16e4d93f6de7 ("NFSD: Ignore
>> client's source port on RDMA transports").
>>
>> I'm not sure why I never noticed this before.
>
> Hi Chuck,
>
> I have a question: is the reason for choosing this fix as oppose to
> fixing the client because it's server's responsibility to design a DRC
> differently for the NFSoRDMA?
The server DRC implementation is not specified in any standard.
The server implementation can use whatever works the best. The
current Linux DRC implementation is useless for NFS/RDMA, because
the clients have to disconnect before they send retransmissions.
If someone knows a way that a client side RDMA consumer can specify
the source port that is presented to a server, then we can make
that change too.
>> Signed-off-by: Chuck Lever <chuck.lever@xxxxxxxxxx>
>> Cc: stable@xxxxxxxxxxxxxxx
>> ---
>> net/sunrpc/xprtrdma/svc_rdma_transport.c | 7 ++++++-
>> 1 file changed, 6 insertions(+), 1 deletion(-)
>>
>> diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c
>> index 027a3b0..1b3700b 100644
>> --- a/net/sunrpc/xprtrdma/svc_rdma_transport.c
>> +++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c
>> @@ -211,9 +211,14 @@ static void handle_connect_req(struct rdma_cm_id *new_cma_id,
>> /* Save client advertised inbound read limit for use later in accept. */
>> newxprt->sc_ord = param->initiator_depth;
>>
>> - /* Set the local and remote addresses in the transport */
>> sa = (struct sockaddr *)&newxprt->sc_cm_id->route.addr.dst_addr;
>> svc_xprt_set_remote(&newxprt->sc_xprt, sa, svc_addr_len(sa));
>> + /* The remote port is arbitrary and not under the control of the
>> + * ULP. Set it to a fixed value so that the DRC continues to work
>> + * after a reconnect.
>> + */
>> + rpc_set_port((struct sockaddr *)&newxprt->sc_xprt.xpt_remote, 0);
>> +
>> sa = (struct sockaddr *)&newxprt->sc_cm_id->route.addr.src_addr;
>> svc_xprt_set_local(&newxprt->sc_xprt, sa, svc_addr_len(sa));
>>
>>
--
Chuck Lever
