> On Aug 11, 2021, at 2:38 PM, Olga Kornievskaia <aglo@xxxxxxxxx> wrote: > > On Wed, Aug 11, 2021 at 1:30 PM Chuck Lever III <chuck.lever@xxxxxxxxxx> wrote: >> >> >> >>> On Aug 11, 2021, at 12:20 PM, Timo Rothenpieler <timo@xxxxxxxxxxxxxxxx> wrote: >>> >>> resulting dmesg and trace logs of both client and server are attached. >>> >>> Test procedure: >>> >>> - start tracing on client and server >>> - mount NFS on client >>> - immediately run 'xfs_io -fc "copy_range testfile" testfile.copy' (which succeeds) >>> - wait 10~15 minutes for the backchannel to time out (still running 5.12.19 with the fix for that reverted) >>> - run xfs_io command again, getting stuck now >>> - let it sit there stuck for a minute, then cancel it >>> - run the command again >>> - while it's still stuck, finished recording the logs and traces >> >> The server tries to send CB_OFFLOAD when the offloaded copy >> completes, but finds the backchannel transport is not connected. >> >> The server can't report the problem until the client sends a >> SEQUENCE operation, but there's really no other traffic going >> on, so it just waits. >> >> The client eventually sends a singleton SEQUENCE to renew its >> lease. The server replies with the SEQ4_STATUS_BACKCHANNEL_FAULT >> flag set at that point. Client's recovery is to destroy that >> session and create a new one. That appears to be successful. >> >> But the server doesn't send another CB_OFFLOAD to let the client >> know the copy is complete, so the client hangs. >> >> This seems to be peculiar to COPY_OFFLOAD, but I wonder if the >> other CB operations suffer from the same "failed to retransmit >> after the CB path is restored" issue. It might not matter for >> some of them, but for others like CB_RECALL, that could be >> important. > > Thank you for the analysis Chuck (btw I haven't seen any attachments > with Timo's posts so I'm assuming some offline communication must have > happened). > ? > I'm looking at the code and wouldn't the mentioned flags be set on the > CB_SEQUENCE operation? CB_SEQUENCE is sent from server to client, and that can't work if the callback channel is down. So the server waits for the client to send a SEQUENCE and it sets the SEQ4_STATUS_BACKCHANNEL_FAULT in its reply. > nfsd4_cb_done() has code to mark the channel > and retry (or another way of saying this, this code should generically > handle retrying whatever operation it is be it CB_OFFLOAD or > CB_RECALL)? cb_done() marks the callback fault, but as far as I can tell the RPC is terminated at that point and there is no subsequent retry. The RPC_TASK flags on the CB_OFFLOAD operation cause that RPC to fail immediately if there's no connection. And in the BACKCHANNEL_FAULT case, the bc_xprt is destroyed as part of recovery. I think that would kill all pending RPC tasks. > Is that not working (not sure if this is a question or a > statement).... I would think that would be the place to handle this > problem. IMHO the OFFLOAD code needs to note that the CB_OFFLOAD RPC failed and then try the call again once the new backchannel is available. -- Chuck Lever
