On Tue, May 21, 2019 at 01:22:54PM -0600, Alan Post wrote: > On Tue, May 21, 2019 at 03:46:03PM +0000, Trond Myklebust wrote: > > > A representative sample of stack traces from hung user-submitted > > > processes (jobs). The first here is quite a lot more common than > > > the later two: > > > > > > $ sudo cat /proc/197520/stack > > > [<0>] io_schedule+0x12/0x40 > > > [<0>] nfs_lock_and_join_requests+0x309/0x4c0 [nfs] > > > [<0>] nfs_updatepage+0x2a2/0x8b0 [nfs] > > > [<0>] nfs_write_end+0x63/0x4c0 [nfs] > > > [<0>] generic_perform_write+0x138/0x1b0 > > > [<0>] nfs_file_write+0xdc/0x200 [nfs] > > > [<0>] new_sync_write+0xfb/0x160 > > > [<0>] vfs_write+0xa5/0x1a0 > > > [<0>] ksys_write+0x4f/0xb0 > > > [<0>] do_syscall_64+0x53/0x100 > > > [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9 > > > [<0>] 0xffffffffffffffff > > > > > > > Have you tried upgrading to 4.19.44? There is a fix that went in not > > too long ago that deals with a request leak that can cause stack traces > > like the above that wait forever. > > > > That I haven't tried. I gather you're talking about either or both > of: > > 63b0ee126f7e > be74fddc976e > > Which I do see went in after 4.19.24 (which I've tried) but didn't > get in 4.20.9 (which I've also tried). Let me see about trying the > 4.19.44 kernel. > > > By the way, the above stack trace with "nfs_lock_and_join_requests" > > usually means that you are using a very small rsize or wsize (less than > > 4k). Is that the case? If so, you might want to look into just > > increasing the I/O size. > > > > These exports have rsize and wsize set to 1048576. Are you getting that from the mount commandline? It could be negotiated down during mount. I think you can get the negotiated values form the rsize= and wsize= values on the opts: line in /proc/self/mountstats. See also /proc/fs/nfsd/max_block_size. --b. > That decision was > before my time, and I'll guess this value was picked to match > NFSSVC_MAXBLKSIZE. > > Thank you for your help, > > -A > -- > Alan Post | Xen VPS hosting for the technically adept > PO Box 61688 | Sunnyvale, CA 94088-1681 | https://prgmr.com/ > email: adp@xxxxxxxxx
