On Mon, 2021-07-12 at 17:07 +0000, Chuck Lever III wrote: > Hi Trond- > > I'm seeing some interesting client hangs that arise from a well- > timed server crash or network partition. > > The easiest to see is gss_destroy() on an Kerberized NFSv4 mount. > > NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter I'll > refer to it as NORTO) when creating a new rpc_clnt. The initial > rpc_ping() for that rpc_clnt is done before the logic that sets > cl_noretranstimeo, thus that ping works as expected (SOFT | > SOFTCONN) and can time out properly if the server isn't > responsive. > > However, once that ping succeeds, cl_noretranstimeo is asserted, > and all subsequent RPC requests on that rpc_clnt are with NORTO > semantics. > > When it comes time to destroy the GSS context for that rpc_clnt, > the NULL procedure with the GSS decorations is sent with SOFT | > SOFTCONN | NORTO. If the server isn't responding at that point, > the client continues to retransmit the GSS context destruction > request forever, and the xprt and possibly the nfs_client are > pinned. > > The problem also arises for lease management operations such as > singleton SEQUENCE or RENEW requests. These are also done with > SOFT, as I recall they need to time out properly. But with > NORTO + SOFT, they will be retried until a connection loss that > might never come. > > I've thought of some ways to modify the cl_noretranstimeo logic > such that it can be disabled for particular RPC tasks, though > none is really striking me as exceptionally clever: > > - Add a field to struct rpc_procinfo that contains a mask of > RPC_TASK flags to clear for each procedure. > - Add logic to rpc_task_set_client() that clears NORTO in > some special cases. > - Reverse the meaning of NORTO (e.g., make it > RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller > for particular RPC tasks if the rpc_clnt-default behavior > is NORTO. > > Any thoughts? > Why would the connection not break when the server goes down? Aren't the TCP_USER_TIMEOUT or the TCP_KEEPALIVE kicking in as they should? Is this an RDMA problem? -- Trond Myklebust Linux NFS client maintainer, Hammerspace trond.myklebust@xxxxxxxxxxxxxxx
