> On Apr 29, 2022, at 8:54 AM, Dennis Dalessandro <dennis.dalessandro@xxxxxxxxxxxxxxxxxxxx> wrote: > > On 4/28/22 3:56 PM, Trond Myklebust wrote: >> On Thu, 2022-04-28 at 15:47 -0400, Dennis Dalessandro wrote: >>> On 4/28/22 11:42 AM, Dennis Dalessandro wrote: >>>> On 4/28/22 10:57 AM, Chuck Lever III wrote: >>>>>> On Apr 28, 2022, at 9:05 AM, Dennis Dalessandro >>>>>> <dennis.dalessandro@xxxxxxxxxxxxxxxxxxxx> wrote: >>>>>> >>>>>> Hi NFS folks, >>>>>> >>>>>> I've noticed a pretty nasty regression in our NFS capability >>>>>> between 5.17 and >>>>>> 5.18-rc1. I've tried to bisect but not having any luck. The >>>>>> problem I'm seeing >>>>>> is it takes 3 minutes to copy a file from NFS to the local >>>>>> disk. When it should >>>>>> take less than half a second, which it did up through 5.17. >>>>>> >>>>>> It doesn't seem to be network related, but can't rule that out >>>>>> completely. >>>>>> >>>>>> I tried to bisect but the problem can be intermittent. Some >>>>>> runs I'll see a >>>>>> problem in 3 out of 100 cycles, sometimes 0 out of 100. >>>>>> Sometimes I'll see it >>>>>> 100 out of 100. >>>>> >>>>> It's not clear from your problem report whether the problem >>>>> appears >>>>> when it's the server running v5.18-rc or the client. >>>> >>>> That's because I don't know which it is. I'll do a quick test and >>>> find out. I >>>> was testing the same kernel across both nodes. >>> >>> Looks like it is the client. >>> >>> server client result >>> ------ ------ ------ >>> 5.17 5.17 Pass >>> 5.17 5.18 Fail >>> 5.18 5.18 Fail >>> 5.18 5.17 Pass >>> >>> Is there a patch for the client issue you mentioned that I could try? >>> >>> -Denny >> >> Try this one > > Thanks for the patch. Unfortunately it doesn't seem to solve the issue, still > see intermittent hangs. I applied it on top of -rc4: > > copy /mnt/nfs_test/run_nfs_test.junk to /dev/shm/run_nfs_test.tmp... > > real 2m6.072s > user 0m0.002s > sys 0m0.263s > Done > > While it was hung I checked the mem usage on the machine: > > # free -h > total used free shared buff/cache available > Mem: 62Gi 871Mi 61Gi 342Mi 889Mi 61Gi > Swap: 4.0Gi 0B 4.0Gi > > Doesn't appear to be under memory pressure. Hi, since you know now that it is the client, perhaps a bisect would be more successful? -- Chuck Lever
