> On Apr 25, 2024, at 4:51 PM, Chris Packham <Chris.Packham@xxxxxxxxxxxxxxxxxxx> wrote:
>
>
> On 25/04/24 11:37, NeilBrown wrote:
>> On Thu, 25 Apr 2024, Chuck Lever III wrote:
>>>> On Apr 24, 2024, at 9:33 AM, Chuck Lever III <chuck.lever@xxxxxxxxxx> wrote:
>>>>
>>>>> On Apr 24, 2024, at 3:42 AM, Chris Packham <Chris.Packham@xxxxxxxxxxxxxxxxxxx> wrote:
>>>>>
>>>>> On 24/04/24 13:38, Chris Packham wrote:
>>>>>> On 24/04/24 12:54, Chris Packham wrote:
>>>>>>> Hi Jeff, Chuck, Greg,
>>>>>>>
>>>>>>> After updating one of our builds along the 5.15.y LTS branch our
>>>>>>> testing caught a new kernel bug. Output below.
>>>>>>>
>>>>>>> I haven't dug into it yet but wondered if it rang any bells.
>>>>>> A bit more info. This is happening at "reboot" for us. Our embedded
>>>>>> devices use a bit of a hacked up reboot process so that they come back
>>>>>> faster in the case of a failure.
>>>>>>
>>>>>> It doesn't happen with a proper `systemctl reboot` or with a SYSRQ+B
>>>>>>
>>>>>> I can trigger it with `killall -9 nfsd` which I'm not sure is a
>>>>>> completely legit thing to do to kernel threads but it's probably close
>>>>>> to what our customized reboot does.
>>>>> I've bisected between v5.15.153 and v5.15.155 and identified commit
>>>>> dec6b8bcac73 ("nfsd: Simplify code around svc_exit_thread() call in
>>>>> nfsd()") as the first bad commit. Based on the context that seems to
>>>>> line up with my reproduction. I'm wondering if perhaps something got
>>>>> missed out of the stable track? Unfortunately I'm not able to run a more
>>>>> recent kernel with all of the nfs related setup that is being used on
>>>>> the system in question.
>>>> Thanks for bisecting, that would have been my first suggestion.
>>>>
>>>> The backport included all of the NFSD patches up to v6.2, but
>>>> there might be a missing server-side SunRPC patch.
>>> So dec6b8bcac73 ("nfsd: Simplify code around svc_exit_thread()
>>> call in nfsd()") is from v6.6, so it was applied to v5.15.y
>>> only to get a subsequent NFSD fix to apply.
>>>
>>> The immediately previous upstream commit is missing:
>>>
>>> 390390240145 ("nfsd: don't allow nfsd threads to be signalled.")
>>>
>>> For testing, I've applied this to my nfsd-5.15.y branch here:
>>>
>>> https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git
>>>
>>> However even if that fixes the reported crash, this suggests
>>> that after v6.6, nfsd threads are not going to respond to
>>> "killall -9 nfsd".
>> I think this likely is the problem. The nfsd threads must be being
>> killed by a signal.
>> One only other cause for an nfsd thread to exit is if
>> svc_set_num_threads() is called, and all places that call that hold a
>> ref on the serv structure so the final put won't happen when the thread
>> exits.
>>
>> Before the patch that bisect found, the nfsd thread would exit with
>>
>> svc_get();
>> svc_exit_thread();
>> nfsd_put();
>>
>> This also holds a ref across the svc_exit_thread(), and ensures the
>> final 'put' happens from nfsD_put(), not svc_put() (in
>> svc_exit_thread()).
>>
>> Chris: what was the context when the crash happened? Could the nfsd
>> threads have been signalled? That hasn't been the standard way to stop
>> nfsd threads for a long time, so I'm a little surprised that it is
>> happening.
>
> We use a hacked up version of shutdown from util-linux and which does a
> `kill (-1, SIGTERM);` then `kill (-1, SIGKILL);` (I don't think that
> particular behaviour is the hackery). I'm not sure if -1 will pick up
> kernel threads but based on the symptoms it appears to be doing so (or
> maybe something else is in it's SIGTERM handler). I don't think we were
> ever really intending to send the signals to nfsd so whether it actually
> terminates or not I don't think is an issue for us. I can confirm that
> applying 390390240145 resolves the symptom we were seeing.
I'm 2/3 of the way through testing 5.15.156 with 390390240145
applied, so it would be just another day before I can send a
patch to stable@.
May I add Tested-by: Chris Packham <Chris.Packham@xxxxxxxxxxxxxxxxxxx> ?
--
Chuck Lever
