> On Oct 15, 2019, at 5:16 PM, Trond Myklebust <trondmy@xxxxxxxxxxxxxxx> wrote:
>
> Hi Neil,
>
> On Tue, 2019-10-15 at 10:36 +1100, NeilBrown wrote:
>> The backchannel RPC requests - that are queued waiting
>> for the reply to be sent by the "NFSv4 callback" thread -
>> have a pointer to the xprt, but it is not reference counted.
>> It is possible for the xprt to be freed while there are
>> still queued requests.
>>
>> I think this has been a problem since
>> Commit fb7a0b9addbd ("nfs41: New backchannel helper routines")
>> when the code was introduced, but I suspect it became more of
>> a problem after
>> Commit 80b14d5e61ca ("SUNRPC: Add a structure to track multiple
>> transports")
>> (or there abouts).
>> Before this second patch, the nfs client would hold a reference to
>> the xprt to keep it alive. After multipath was introduced,
>> a client holds a reference to a swtich, and the switch can have
>> multiple
>> xprts which can be added and removed.
>>
>> I'm not sure of all the causal issues, but this patch has
>> fixed a customer problem were an NFSv4.1 client would run out
>> of memory with tens of thousands of backchannel rpc requests
>> queued for an xprt that had been freed. This was a 64K-page
>> machine so each rpc_rqst consumed more than 128K of memory.
>>
>> Fixes: 80b14d5e61ca ("SUNRPC: Add a structure to track multiple
>> transports")
>> cc: stable@xxxxxxxxxxxxxxx (v4.6)
>> Signed-off-by: NeilBrown <neilb@xxxxxxx>
>> ---
>> net/sunrpc/backchannel_rqst.c | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/net/sunrpc/backchannel_rqst.c
>> b/net/sunrpc/backchannel_rqst.c
>> index 339e8c077c2d..c95ca39688b6 100644
>> --- a/net/sunrpc/backchannel_rqst.c
>> +++ b/net/sunrpc/backchannel_rqst.c
>> @@ -61,6 +61,7 @@ static void xprt_free_allocation(struct rpc_rqst
>> *req)
>> free_page((unsigned long)xbufp->head[0].iov_base);
>> xbufp = &req->rq_snd_buf;
>> free_page((unsigned long)xbufp->head[0].iov_base);
>> + xprt_put(req->rq_xprt);
>> kfree(req);
>> }
>
> Would it perhaps make better sense to move the xprt_get() to
> xprt_lookup_bc_request() and to release it in xprt_free_bc_rqst()?
/me wonders if the same problem exists for the RPC/RDMA backchannel....
> Otherwise as far as I can tell, we will have freed slots on the xprt-
>> bc_pa_list that hold a reference to the transport itself, meaning that
> the latter never gets released.
>
>>
>> @@ -85,7 +86,7 @@ struct rpc_rqst *xprt_alloc_bc_req(struct rpc_xprt
>> *xprt, gfp_t gfp_flags)
>> if (req == NULL)
>> return NULL;
>>
>> - req->rq_xprt = xprt;
>> + req->rq_xprt = xprt_get(xprt);
>> INIT_LIST_HEAD(&req->rq_bc_list);
>>
>> /* Preallocate one XDR receive buffer */
> --
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> trond.myklebust@xxxxxxxxxxxxxxx
--
Chuck Lever
