On 11/11/11 22:38, Trond Myklebust wrote: > On Fri, 2011-11-11 at 10:31 +0000, Andrew Cooper wrote: >> On 10/11/11 20:43, Trond Myklebust wrote: >>> On Thu, 2011-11-10 at 15:52 +0000, Andrew Cooper wrote: >>>> On 10/11/11 15:29, Chuck Lever wrote: >>>>> On Nov 10, 2011, at 6:15 AM, Andrew Cooper wrote: >>>>> >>>>>> On 09/11/11 22:36, Chuck Lever wrote: >>>>>>> On Nov 9, 2011, at 1:38 PM, Andrew Cooper wrote: >>>> Sorry. I am not sure I was clear. An EIO does not present itself with >>>> a hard mount, but a TCP FIN is still injected into the stream by the >>>> client, causing 15 seconds of deadlock, eventually fixed by sending a >>>> RST and restarting with a new TCP stream. At this point, softmounts >>>> throw an EIO while hardmounts restart and continue successfully. >>>> >>>> My problem is not the EIO on softmount or lack of EIO for hardmout, but >>>> the fact that the client sees fit to try and close the TCP stream while >>>> an apparently otherwise healthy NFS session is ongoing. >>> The client will attempt to close the TCP connection on any RPC level >>> error. That can happen, e.g., if the server sends a faulty RPC/TCP >>> record fragment header or some other garbage data. >>> >>> I'm assuming that you've checked that the TCP parameters are set to sane >>> values for a 10GigE connection (i.e. tcp_timestamps is on) so that there >>> is no corruption happening at that level? >>> >>> Cheers >>> Trond >> I have a TCPdump/wireshark analysis of the entire packet stream (4GiB). >> I cant see any RPC level errors (rpc.replystat != 0 yields no matches). >> What specifically would I be looking for? Wireshark seems not to have >> any problem decoding any of the RPC packets, so I hope that indicates no >> RPC level corruption. >> >> There is one case where the server sends a double write reply for the >> same write, with different length fields. However, this is a good 20 >> seconds before the FIN is sent, so I was hoping that it was unrelated. >> Might it not be? > Can you send us just that portion of the trace so that we can have a look? Attached is a small extract from the stream. It starts with the final NFS write, and continues through the FIN, RST and until the TCP stream gets reopened. Is this what you want? >> As for TCP timestamps; I have a Timestamp option in each TCP packet. >> Nothing appears corrupted. What would I be looking for with corrupted >> timestamps? > I just meant that you should check that you've enabled tcp window > scaling and timestamps in order to avoid problems with wrapped sequence > numbers (See http://tools.ietf.org/html/rfc1323 for details). > > On Linux this means that you need to check > > sysctl net.ipv4.tcp_timestamps > and > sysctl net.ipv4.tcp_window_scaling > > They should both be set to the value '1'. They both are. > Cheers > Trond > -- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 (0)1223 225 900, http://www.citrix.com
Attachment:
nfs-fin.capture.gz
Description: GNU Zip compressed data
