On Fri, Feb 22, 2019 at 10:50 AM Trond Myklebust
<trondmy@xxxxxxxxxxxxxxx> wrote:
>
> On Fri, 2019-02-22 at 10:11 -0500, Olga Kornievskaia wrote:
> > On Fri, Feb 22, 2019 at 10:06 AM Trond Myklebust
> > <trondmy@xxxxxxxxxxxxxxx> wrote:
> > > On Fri, 2019-02-22 at 09:46 -0500, Olga Kornievskaia wrote:
> > > > On Fri, Feb 22, 2019 at 8:45 AM Trond Myklebust <
> > > > trondmy@xxxxxxxxxxxxxxx> wrote:
> > > > > On Fri, 2019-02-22 at 07:12 -0500, Dave Wysochanski wrote:
> > > > > > Hi Olga,
> > > > > >
> > > > > > Do you have a reproducer for this? A number of months ago I
> > > > > > did
> > > > > > a
> > > > > > significant amount of testing with half-closed connections,
> > > > > > after
> > > > > > we
> > > > > > had reports of connections stuck in FIN_WAIT2 in some older
> > > > > > kernels.
> > > > > > What I found was with kernels that had the tcp keepalives
> > > > > > (commit
> > > > > > 7f260e8575bf53b93b77978c1e39f8e67612759c), I could only
> > > > > > reproduce
> > > > > > a
> > > > > > hang of a few minutes, after which time the tcp keepalive
> > > > > > code
> > > > > > would
> > > > > > reset the connection.
> > > > > >
> > > > > > That said it was a while ago and something subtle may have
> > > > > > changed.
> > > > > > Also I'm not not sure if your header implies an indefinite
> > > > > > hang
> > > > > > or
> > > > > > just
> > > > > > a few minutes.
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > >
> > > > > > On Wed, 2019-02-20 at 09:56 -0500, Olga Kornievskaia wrote:
> > > > > > > From: Olga Kornievskaia <kolga@xxxxxxxxxx>
> > > > > > >
> > > > > > > When server replies with an ACK to client's FIN/ACK, client
> > > > > > > ends
> > > > > > > up stuck in a TCP_FIN_WAIT2 state and client's mount hangs.
> > > > > > > Instead, make sure to close and reset client's socket and
> > > > > > > transport
> > > > > > > when transitioned into that state.
> > > >
> > > > Hi Trond,
> > > >
> > > > > So, please do note that we do not want to ignore the FIN_WAIT2
> > > > > state
> > > >
> > > > But we do ignore the FIN_WAIT2 state.
> > >
> > > We do not. We wait for the server to send a FIN, which is precisely
> > > the
> > > reason for which FIN_WAIT2 exists.
> > >
> > > > > because it implies that the server has not closed the socket on
> > > > > its
> > > > > side.
> > > >
> > > > That's correct.
> > > >
> > > > > That again means that we cannot re-establish a connection using
> > > > > the same source IP+port to the server, which is problematic for
> > > > > protocols such as NFSv3 which rely on standard duplicate reply
> > > > > cache
> > > > > for correct replay semantics.
> > > >
> > > > that's exactly what's happening that a client is unable to
> > > > establish
> > > > a
> > > > new connection to the server. With the patch, the client does an
> > > > RST
> > > > and it re-uses the port and all is well for NFSv3.
> > >
> > > RST is not guaranteed to be delivered to the recipient. That's why
> > > the
> > > TCP protocol defines FIN: it is a guaranteed to be delivered
> > > because it
> > > is ACKed.
> > >
> > > > > This is why we don't just set the TCP_LINGER2 socket option and
> > > > > call
> > > > > sock_release(). The choice to try to wait it out is deliberate
> > > > > because
> > > > > the alternative is that we end up with busy-waiting re-
> > > > > connection
> > > > > attempts.
> > > >
> > > > Why would it busy-wait? In my testing, RST happens and new
> > > > connection
> > > > is established?
> > >
> > > Only if the server has dropped the connection without notifying the
> > > client.
> >
> > Yes the server dropped the connection without notifying the client
> > (or
> > perhaps something in the middle did it as an attack). Again, I raise
> > this concern for the sake of dealing with this as an attack. I have
> > no
> > intentions of catering to broken servers. If this is not a possible
> > attack, then we don't have to deal with it.
>
> A man in the middle might be able to intercept the FIN from the server
> and ACK it, causing the connection to be closed on that server.
> However, as Dave pointed out, why wouldn't the keepalive mechanism then
> eventually kick in and close the socket on the client as well?
The mechanism is already kicked in and got stuck in FIN_WAIT2. NFS
connection was idle, so TCP layer was sending keep-alives. Then it
sent a FIN/ACK to which the server replied with just an ACK. Kernel
notified NFS that we are in FIN_WAIT2 and I believe it is NFS
responsibility to act accordingly. Kernel then keeps sending
"keep-alives" forever. Because of this code:
case TCP_FIN_WAIT1:
case TCP_FIN_WAIT2:
/* RFC 793 says to queue data in these states,
* RFC 1122 says we MUST send a reset.
* BSD 4.4 also does reset.
*/
if (sk->sk_shutdown & RCV_SHUTDOWN) {
if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
after(TCP_SKB_CB(skb)->end_seq - th->fin,
tp->rcv_nxt)) {
NET_INC_STATS(sock_net(sk),
LINUX_MIB_TCPABORTONDATA);
tcp_reset(sk); << this is never triggered
return 1;
}
}
In our case TCP_SKB_CB(skb)->end_seq always equals
TCP_SKB_CB(skb)->seq. (No i don't know the meaning of end_seq and seq
:-/).
> If the FIN is not ACKed, then the server is supposed to keep
> retransmitting it. Until that ACK is received, it cannot close the
> socket without violating the TCP protocol.
Something in the middle can keep intercepting the the FIN/ACK from the
server and keep sending an ACK back?
>
> --
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> trond.myklebust@xxxxxxxxxxxxxxx
>
>
