On Wed, Mar 27, 2024 at 06:09:28PM -0700, Dai Ngo wrote:
>
> On 3/26/24 11:27 AM, Chuck Lever wrote:
> > On Tue, Mar 26, 2024 at 11:13:29AM -0700, Dai Ngo wrote:
> > > Currently when a nfs4_client is destroyed we wait for the cb_recall_any
> > > callback to complete before proceed. This adds unnecessary delay to the
> > > __destroy_client call if there is problem communicating with the client.
> > By "unnecessary delay" do you mean only the seven-second RPC
> > retransmit timeout, or is there something else?
>
> when the client network interface is down, the RPC task takes ~9s to
> send the callback, waits for the reply and gets ETIMEDOUT. This process
> repeats in a loop with the same RPC task before being stopped by
> rpc_shutdown_client after client lease expires.
I'll have to review this code again, but rpc_shutdown_client
should cause these RPCs to terminate immediately and safely. Can't
we use that?
> It takes a total of about 1m20s before the CB_RECALL is terminated.
> For CB_RECALL_ANY and CB_OFFLOAD, this process gets in to a infinite
> loop since there is no delegation conflict and the client is allowed
> to stay in courtesy state.
>
> The loop happens because in nfsd4_cb_sequence_done if cb_seq_status
> is 1 (an RPC Reply was never received) it calls nfsd4_mark_cb_fault
> to set the NFSD4_CB_FAULT bit. It then sets cb_need_restart to true.
> When nfsd4_cb_release is called, it checks cb_need_restart bit and
> re-queues the work again.
Something in the sequence_done path should check if the server is
tearing down this callback connection. If it doesn't, that is a bug
IMO.
Btw, have you checked NFSv4.0 behavior?
> > I can see that a server shutdown might want to cancel these, but why
> > is this a problem when destroying an nfs4_client?
>
> Destroying an nfs4_client is called when the export is unmounted.
Ah, agreed. Thanks for reminding me.
> Cancelling these calls just make the process a bit quicker when there
> is problem with the client connection, or preventing the unmount to
> hang if there is problem at the workqueue and a callback work is
> pending there.
>
> For CB_RECALL, even if we wait for the call to complete the client
> won't be able to return any delegations since the nfs4_client is
> already been destroyed. It just serves as a notice to the client that
> there is a delegation conflict so it can take appropriate actions.
>
> > > This patch addresses this issue by cancelling the CB_RECALL_ANY call from
> > > the workqueue when the nfs4_client is about to be destroyed.
> > Does CB_OFFLOAD need similar treatment?
>
> Probably. The copy is already done anyway, this is just a notification.
It would be a nicer design if all outstanding callback RPCs could
be handled with one mechanism instead of building a separate
shutdown method for each operation type.
> -Dai
>
> >
> >
> > > Signed-off-by: Dai Ngo <dai.ngo@xxxxxxxxxx>
> > > ---
> > > fs/nfsd/nfs4callback.c | 10 ++++++++++
> > > fs/nfsd/nfs4state.c | 10 +++++++++-
> > > fs/nfsd/state.h | 1 +
> > > 3 files changed, 20 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/fs/nfsd/nfs4callback.c b/fs/nfsd/nfs4callback.c
> > > index 87c9547989f6..e5b50c96be6a 100644
> > > --- a/fs/nfsd/nfs4callback.c
> > > +++ b/fs/nfsd/nfs4callback.c
> > > @@ -1568,3 +1568,13 @@ bool nfsd4_run_cb(struct nfsd4_callback *cb)
> > > nfsd41_cb_inflight_end(clp);
> > > return queued;
> > > }
> > > +
> > > +void nfsd41_cb_recall_any_cancel(struct nfs4_client *clp)
> > > +{
> > > + if (test_bit(NFSD4_CLIENT_CB_RECALL_ANY, &clp->cl_flags) &&
> > > + cancel_delayed_work(&clp->cl_ra->ra_cb.cb_work)) {
> > > + clear_bit(NFSD4_CLIENT_CB_RECALL_ANY, &clp->cl_flags);
> > > + atomic_add_unless(&clp->cl_rpc_users, -1, 0);
> > > + nfsd41_cb_inflight_end(clp);
> > > + }
> > > +}
> > > diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> > > index 1a93c7fcf76c..0e1db57c9a19 100644
> > > --- a/fs/nfsd/nfs4state.c
> > > +++ b/fs/nfsd/nfs4state.c
> > > @@ -2402,6 +2402,7 @@ __destroy_client(struct nfs4_client *clp)
> > > }
> > > nfsd4_return_all_client_layouts(clp);
> > > nfsd4_shutdown_copy(clp);
> > > + nfsd41_cb_recall_any_cancel(clp);
> > > nfsd4_shutdown_callback(clp);
> > > if (clp->cl_cb_conn.cb_xprt)
> > > svc_xprt_put(clp->cl_cb_conn.cb_xprt);
> > > @@ -2980,6 +2981,12 @@ static void force_expire_client(struct nfs4_client *clp)
> > > clp->cl_time = 0;
> > > spin_unlock(&nn->client_lock);
> > > + /*
> > > + * no need to send and wait for CB_RECALL_ANY
> > > + * when client is about to be destroyed
> > > + */
> > > + nfsd41_cb_recall_any_cancel(clp);
> > > +
> > > wait_event(expiry_wq, atomic_read(&clp->cl_rpc_users) == 0);
> > > spin_lock(&nn->client_lock);
> > > already_expired = list_empty(&clp->cl_lru);
> > > @@ -6617,7 +6624,8 @@ deleg_reaper(struct nfsd_net *nn)
> > > clp->cl_ra->ra_bmval[0] = BIT(RCA4_TYPE_MASK_RDATA_DLG) |
> > > BIT(RCA4_TYPE_MASK_WDATA_DLG);
> > > trace_nfsd_cb_recall_any(clp->cl_ra);
> > > - nfsd4_run_cb(&clp->cl_ra->ra_cb);
> > > + if (!nfsd4_run_cb(&clp->cl_ra->ra_cb))
> > > + clear_bit(NFSD4_CLIENT_CB_RECALL_ANY, &clp->cl_flags);
> > > }
> > > }
> > > diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
> > > index 01c6f3445646..259b4af7d226 100644
> > > --- a/fs/nfsd/state.h
> > > +++ b/fs/nfsd/state.h
> > > @@ -735,6 +735,7 @@ extern void nfsd4_change_callback(struct nfs4_client *clp, struct nfs4_cb_conn *
> > > extern void nfsd4_init_cb(struct nfsd4_callback *cb, struct nfs4_client *clp,
> > > const struct nfsd4_callback_ops *ops, enum nfsd4_cb_op op);
> > > extern bool nfsd4_run_cb(struct nfsd4_callback *cb);
> > > +extern void nfsd41_cb_recall_any_cancel(struct nfs4_client *clp);
> > > extern int nfsd4_create_callback_queue(void);
> > > extern void nfsd4_destroy_callback_queue(void);
> > > extern void nfsd4_shutdown_callback(struct nfs4_client *);
> > > --
> > > 2.39.3
> > >
--
Chuck Lever
