> On Jul 12, 2021, at 1:36 PM, Trond Myklebust <trondmy@xxxxxxxxxxxxxxx> wrote: > > On Mon, 2021-07-12 at 17:07 +0000, Chuck Lever III wrote: >> Hi Trond- >> >> I'm seeing some interesting client hangs that arise from a well- >> timed server crash or network partition. >> >> The easiest to see is gss_destroy() on an Kerberized NFSv4 mount. >> >> NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter I'll >> refer to it as NORTO) when creating a new rpc_clnt. The initial >> rpc_ping() for that rpc_clnt is done before the logic that sets >> cl_noretranstimeo, thus that ping works as expected (SOFT | >> SOFTCONN) and can time out properly if the server isn't >> responsive. >> >> However, once that ping succeeds, cl_noretranstimeo is asserted, >> and all subsequent RPC requests on that rpc_clnt are with NORTO >> semantics. >> >> When it comes time to destroy the GSS context for that rpc_clnt, >> the NULL procedure with the GSS decorations is sent with SOFT | >> SOFTCONN | NORTO. If the server isn't responding at that point, >> the client continues to retransmit the GSS context destruction >> request forever, and the xprt and possibly the nfs_client are >> pinned. >> >> The problem also arises for lease management operations such as >> singleton SEQUENCE or RENEW requests. These are also done with >> SOFT, as I recall they need to time out properly. But with >> NORTO + SOFT, they will be retried until a connection loss that >> might never come. >> >> I've thought of some ways to modify the cl_noretranstimeo logic >> such that it can be disabled for particular RPC tasks, though >> none is really striking me as exceptionally clever: >> >> - Add a field to struct rpc_procinfo that contains a mask of >> RPC_TASK flags to clear for each procedure. >> - Add logic to rpc_task_set_client() that clears NORTO in >> some special cases. >> - Reverse the meaning of NORTO (e.g., make it >> RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller >> for particular RPC tasks if the rpc_clnt-default behavior >> is NORTO. >> >> Any thoughts? >> > > Why would the connection not break when the server goes down? The server can't actively RST or FIN the connection if a network partition occurs; and some servers might crash while their kernel is still alive to respond to keep-alive. > Aren't > the TCP_USER_TIMEOUT or the TCP_KEEPALIVE kicking in as they should? I don't see them kicking in, but I let the test run only for about 12 minutes. -- Chuck Lever
