On Mon, 2021-07-12 at 17:48 +0000, Chuck Lever III wrote: > > > > On Jul 12, 2021, at 1:36 PM, Trond Myklebust > > <trondmy@xxxxxxxxxxxxxxx> wrote: > > > > On Mon, 2021-07-12 at 17:07 +0000, Chuck Lever III wrote: > > > Hi Trond- > > > > > > I'm seeing some interesting client hangs that arise from a well- > > > timed server crash or network partition. > > > > > > The easiest to see is gss_destroy() on an Kerberized NFSv4 mount. > > > > > > NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter > > > I'll > > > refer to it as NORTO) when creating a new rpc_clnt. The initial > > > rpc_ping() for that rpc_clnt is done before the logic that sets > > > cl_noretranstimeo, thus that ping works as expected (SOFT | > > > SOFTCONN) and can time out properly if the server isn't > > > responsive. > > > > > > However, once that ping succeeds, cl_noretranstimeo is asserted, > > > and all subsequent RPC requests on that rpc_clnt are with NORTO > > > semantics. > > > > > > When it comes time to destroy the GSS context for that rpc_clnt, > > > the NULL procedure with the GSS decorations is sent with SOFT | > > > SOFTCONN | NORTO. If the server isn't responding at that point, > > > the client continues to retransmit the GSS context destruction > > > request forever, and the xprt and possibly the nfs_client are > > > pinned. > > > > > > The problem also arises for lease management operations such as > > > singleton SEQUENCE or RENEW requests. These are also done with > > > SOFT, as I recall they need to time out properly. But with > > > NORTO + SOFT, they will be retried until a connection loss that > > > might never come. > > > > > > I've thought of some ways to modify the cl_noretranstimeo logic > > > such that it can be disabled for particular RPC tasks, though > > > none is really striking me as exceptionally clever: > > > > > > - Add a field to struct rpc_procinfo that contains a mask of > > > RPC_TASK flags to clear for each procedure. > > > - Add logic to rpc_task_set_client() that clears NORTO in > > > some special cases. > > > - Reverse the meaning of NORTO (e.g., make it > > > RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller > > > for particular RPC tasks if the rpc_clnt-default behavior > > > is NORTO. > > > > > > Any thoughts? > > > > > > > Why would the connection not break when the server goes down? > > The server can't actively RST or FIN the connection if a network > partition occurs; and some servers might crash while their kernel > is still alive to respond to keep-alive. > > > > Aren't > > the TCP_USER_TIMEOUT or the TCP_KEEPALIVE kicking in as they > > should? > > I don't see them kicking in, but I let the test run only for about > 12 minutes. > TCP_USER_TIMEOUT should kick in any time when the server is failing to read the socket contents, and should close the connection. A more likely scenario is that the server is actually reading the socket, but is just dropping the requests on the floor. I agree that needs to be handled correctly. One way to do that could be to add a flag that says "don't apply any other default task flags" for special cases like this one? -- Trond Myklebust Linux NFS client maintainer, Hammerspace trond.myklebust@xxxxxxxxxxxxxxx
