On Fri, 3 Nov 2023 at 14:10, Bagas Sanjaya <bagasdotme@xxxxxxxxx> wrote: > > On Fri, Nov 03, 2023 at 09:40:44AM +0000, Daire Byrne wrote: > > Hi, > > > > We have large compute clusters that, amongst other things, spend their > > day mounting & unounting lots of Linux NFS servers via autofs. This > > has worked fine for many years and client kernel versions and was > > working without incident even with our current v6.3.x production > > kernels. > > > > During the v6.6-rc cycle while testing that kernel, I noticed that > > every now and then, the umount/mount would hang randomly and the > > compute host would get stuck and not complete it's work until a > > reboot. I thought I'd wait until v6.6 was released and check again - > > the issue persists. > > > > I have not had the opportunity to test the v6.4 & v6.5 kernels in > > between yet. The stack traces look something like this: > > Please do bisection to find the exact commit that introduces your > regression. See Documentation/admin-guide/bug-bisect.rst in the kernel > sources for more information. I tested the v6.5.10 kernel and I can't reproduce with that, so I guess I'll start the process of bisecting between v6.5.10 & v6.6. It takes around 24 hours and 72 client hosts until we see it happen on at least one, so it'll be slow going. Weirdly, I also have the v6.6 kernel running on a different set of hosts and they have not shown the problem at all. The only difference being AMD clients with higher process concurrency = bad, and Intel clients with less process concurrency = okay. > > > > [202752.264187] INFO: task umount.nfs:58118 blocked for more than 245 seconds. > > [202752.264237] Tainted: G E 6.6.0-1.dneg.x86_64 #1 > > Can you reproduce on untainted (vanilla) kernel? Yea, I'm not sure what is going on there. Some compile issue as it's the "deflate" module complaining about a missing key. > > And like I said, the mount/umount against the server hangs > > indefinitely on the client. It is somewhat interesting that autofs > > still tries to trigger a subsequent mount even though the umount has > > not completed. > > > > The NFS servers are running RHEL8.5 and we are using NFSv3. I also > > reproduced it with a fairly recent nfs-utils-2.6.2 on the client > > compute hosts. > > What distro on the client? It's EL79 on the client but I have updated the nfs-utils to something more modern. I have not updated the autofs package though. I'll report back in a week or so... Cheers, Daire
