Hello,
I've put the broken server back into production (we need the capacity),
however, we are open to enabling whatever RPC/NFS debugging would help
to get to the source of this problem. Can anybody suggest which kernel
parameters would be most beneficial to determining the cause of the
problem? I am talking about one of these, or something else which I'm
not aware of:
sunrpc.rpc_debug
sunrpc.nfs_debug
sunrpc.nfsd_debug
sunrpc.nlm_debug
If somebody suggests any options to be enabled, can you also comment on
whether there would be any performance hit related to enabling the option?
Thanks!
Iordan Iordanov
On 02/13/13 15:45, Iordan Iordanov wrote:
Hello!
We've been suffering from NFS mounts and data transfers hanging on our
Ubuntu Precise 12.04 32-bit shared servers since last summer. The
problem has reoccurred over TCP and UDP.
Every month or so, some of our more heavily used shared servers would
see its NFS mounts hang, and a bunch of flush-(major:minor) processes
would sit at 100% in top. If the hang occurred while NFS over TCP was
being used, mounting over UDP would still work, but mounting over TCP
would hang (indefinitely). The reverse is also true. When we experienced
the hang while using UDP, mounts over TCP would work on the affected
system.
I've located a very similar discussion/bug-report for v3.1-rc4 which
ended seemingly without a resolution here:
http://lkml.indiana.edu/hypermail/linux/kernel/1109.1/00728.html
We're also seeing the:
[3121466.072728] RPC: 43506 failed to lock transport e030a000
errors when RPC debugging is enabled. In addition, we're also seeing the
socket in CLOSE_WAIT state symptom in netstat's output:
tcp 0 0 x.x.x.x:967 y.y.y.y:2049 CLOSE_WAIT
Running tcpdump on our file-server and specifying the hung host in
question results in NO NFS-related traffic unless a mount request is
executed on the nfs client. I've attached two tcpdumps representing a
successful mount over UDP and an unsuccessful mount over TCP in case
they are useful. The tcpdumps were captured on the fileserver.
The machine in question is currently in this hung state, and we would be
happy to provide any additional information you may need!
Here is the result of uname -a on the hung machine:
Linux gambo 3.2.0-35-generic-pae #55-Ubuntu SMP Wed Dec 5 18:04:39 UTC
2012 i686 athlon i386 GNU/Linux
Our NFS server is an up-to-date Debian Squeeze 6.0 box, and we would be
happy to provide information on that machine if you think it is relevant.
Any help in resolving this would be greatly appreciated, as we are
constantly suffering from this issue.
Many thanks in advance!
Iordan
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
