> On Mar 29, 2022, at 3:49 PM, Bruce Fields <bfields@xxxxxxxxxxxx> wrote:
>
> On Tue, Mar 29, 2022 at 07:32:57PM +0000, Chuck Lever III wrote:
>>
>>
>>> On Mar 29, 2022, at 2:39 PM, J. Bruce Fields <bfields@xxxxxxxxxxxx> wrote:
>>>
>>> On Tue, Mar 29, 2022 at 11:19:51AM -0700, dai.ngo@xxxxxxxxxx wrote:
>>>>
>>>> On 3/29/22 9:30 AM, J. Bruce Fields wrote:
>>>>> On Tue, Mar 29, 2022 at 09:20:02AM -0700, dai.ngo@xxxxxxxxxx wrote:
>>>>>> On 3/29/22 8:47 AM, J. Bruce Fields wrote:
>>>>>>> On Thu, Mar 24, 2022 at 09:34:42PM -0700, Dai Ngo wrote:
>>>>>>>> Update nfs4_client to add:
>>>>>>>> . cl_cs_client_state: courtesy client state
>>>>>>>> . cl_cs_lock: spinlock to synchronize access to cl_cs_client_state
>>>>>>>> . cl_cs_list: list used by laundromat to process courtesy clients
>>>>>>>>
>>>>>>>> Modify alloc_client to initialize these fields.
>>>>>>>>
>>>>>>>> Signed-off-by: Dai Ngo <dai.ngo@xxxxxxxxxx>
>>>>>>>> ---
>>>>>>>> fs/nfsd/nfs4state.c | 2 ++
>>>>>>>> fs/nfsd/nfsd.h | 1 +
>>>>>>>> fs/nfsd/state.h | 33 +++++++++++++++++++++++++++++++++
>>>>>>>> 3 files changed, 36 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
>>>>>>>> index 234e852fcdfa..a65d59510681 100644
>>>>>>>> --- a/fs/nfsd/nfs4state.c
>>>>>>>> +++ b/fs/nfsd/nfs4state.c
>>>>>>>> @@ -2009,12 +2009,14 @@ static struct nfs4_client *alloc_client(struct xdr_netobj name)
>>>>>>>> INIT_LIST_HEAD(&clp->cl_delegations);
>>>>>>>> INIT_LIST_HEAD(&clp->cl_lru);
>>>>>>>> INIT_LIST_HEAD(&clp->cl_revoked);
>>>>>>>> + INIT_LIST_HEAD(&clp->cl_cs_list);
>>>>>>>> #ifdef CONFIG_NFSD_PNFS
>>>>>>>> INIT_LIST_HEAD(&clp->cl_lo_states);
>>>>>>>> #endif
>>>>>>>> INIT_LIST_HEAD(&clp->async_copies);
>>>>>>>> spin_lock_init(&clp->async_lock);
>>>>>>>> spin_lock_init(&clp->cl_lock);
>>>>>>>> + spin_lock_init(&clp->cl_cs_lock);
>>>>>>>> rpc_init_wait_queue(&clp->cl_cb_waitq, "Backchannel slot table");
>>>>>>>> return clp;
>>>>>>>> err_no_hashtbl:
>>>>>>>> diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h
>>>>>>>> index 4fc1fd639527..23996c6ca75e 100644
>>>>>>>> --- a/fs/nfsd/nfsd.h
>>>>>>>> +++ b/fs/nfsd/nfsd.h
>>>>>>>> @@ -336,6 +336,7 @@ void nfsd_lockd_shutdown(void);
>>>>>>>> #define COMPOUND_ERR_SLACK_SPACE 16 /* OP_SETATTR */
>>>>>>>> #define NFSD_LAUNDROMAT_MINTIMEOUT 1 /* seconds */
>>>>>>>> +#define NFSD_COURTESY_CLIENT_TIMEOUT (24 * 60 * 60) /* seconds */
>>>>>>>> /*
>>>>>>>> * The following attributes are currently not supported by the NFSv4 server:
>>>>>>>> diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
>>>>>>>> index 95457cfd37fc..40e390abc842 100644
>>>>>>>> --- a/fs/nfsd/state.h
>>>>>>>> +++ b/fs/nfsd/state.h
>>>>>>>> @@ -283,6 +283,35 @@ struct nfsd4_sessionid {
>>>>>>>> #define HEXDIR_LEN 33 /* hex version of 16 byte md5 of cl_name plus '\0' */
>>>>>>>> /*
>>>>>>>> + * CLIENT_ CLIENT_ CLIENT_
>>>>>>>> + * COURTESY EXPIRED RECONNECTED Meaning Where set
>>>>>>>> + * -----------------------------------------------------------------------------
>>>>>>>> + * | false | false | false | Confirmed, active | Default |
>>>>>>>> + * |---------------------------------------------------------------------------|
>>>>>>>> + * | true | false | false | Courtesy state. | nfs4_get_client_reaplist |
>>>>>>>> + * | | | | Lease/lock/share | |
>>>>>>>> + * | | | | reservation conflict | |
>>>>>>>> + * | | | | can cause Courtesy | |
>>>>>>>> + * | | | | client to be expired | |
>>>>>>>> + * |---------------------------------------------------------------------------|
>>>>>>>> + * | false | true | false | Courtesy client to be| nfs4_laundromat |
>>>>>>>> + * | | | | expired by Laundromat| nfsd4_lm_lock_expired |
>>>>>>>> + * | | | | due to conflict | nfsd4_discard_courtesy_clnt |
>>>>>>>> + * | | | | | nfsd4_expire_courtesy_clnt |
>>>>>>>> + * |---------------------------------------------------------------------------|
>>>>>>>> + * | false | false | true | Courtesy client | nfsd4_courtesy_clnt_expired|
>>>>>>>> + * | | | | reconnected, | |
>>>>>>>> + * | | | | becoming active | |
>>>>>>>> + * -----------------------------------------------------------------------------
>>>>> By the way, where is a client returned to the normal (0) state? That
>>>>> has to happen at some point.
>>>>
>>>> For 4.1 courtesy client reconnects is detected in nfsd4_sequence,
>>>> nfsd4_bind_conn_to_session.
>>>
>>> Those are the places where NFSD54_CLIENT_RECONNECTED is set, which isn't
>>> the question I asked.
>>
>> "reconnected" simply means the client has gotten back in touch.
>
> Again, my question was: when is cl_cs_client_state set back to 0? As
> far as I can tell, the answer is never. That means, even long after the
> client has reconnected, it's left in a weird state where it can be
> suddenly expired for all sorts of reasons.
Got it. Agreed, cl_cs_client_state should be reinitialized if
a courtesy client is transitioned back to "active".
Dai, would you add
+enum courtesy_client_state {
>>> NFSD4_CLIENT_ACTIVE = 0,
+ NFSD4_CLIENT_COURTESY,
+ NFSD4_CLIENT_EXPIRED,
+ NFSD4_CLIENT_RECONNECTED,
+};
And set cl_cs_client_state to ACTIVE where the client is
allowed to transition back to being active?
>> The server then has to decide whether to allow the client to
>> become active again or it needs to purge it. That decision
>> is different for each operation and minor version. Look for
>> "if (cl_cs_client_state == NFSD4_CLIENT_RECONNECTED)" for how
>> those choices are made.
>>
>>
>>>>> Why are RECONNECTED clients discarded in so many cases? (E.g. whenever
>>>>> a bind_conn_to_session fails).
>>>>
>>>> find_in_sessionid_hashtbl: we discard the courtesy client when it
>>>> reconnects and there is error from nfsd4_get_session_locked. This
>>>> should be a rare condition so rather than reverting the client
>>>> state back to courtesy, it is simpler just to discard it.
>>>
>>> That may be a rare situation, but I don't believe the behavior of
>>> discarding the client in this case is correct.
>>
>> Can you explain this? It's a courtesy client... the server can
>> decide it's expired at that point, can't it? IOW what breaks?
>
> I'm not worried about courtesy clients, I'm worried about clients that
> were courtesy clients but have since succesfully renewed their state.
> Expiring them for a failed bind_conn_to_session isn't right.
--
Chuck Lever
