I'm seeing some very strange problems which I believe are at least half
netfilter issues - my apologies if not! I hope you can help me.
I have a Fedora Core 4 system (2.6.12-ish, iptables 1.3.0-ish) acting as a
masquerading gateway between an ethernet network (MTU 1500) and a PPPoATM
connection (MTU 1492).
I've simplified things to a single iptables rule,
-t nat -A POSTROUTING -o ppp0 -j MASQUERADE
with net.ipv4.ip_forward=1, and can reproduce the problem.
Machines on the internal network are unable to access certain websites (the
homepage of www.nationalrail.co.uk, for example). It appears that large
packets are being dropped, but also, whilst snooping to debug, I saw
unmasqueraded packets being leaked out.
Snooping on the LAN address of the gateway I see (eg.):
No. Time Source Destination Protocol Info
1 0.000000 192.168.31.191 213.174.202.41 TCP 1851 > http [SYN] Seq=0 Ack=0 Win=65535 Len=0 MSS=1460
2 0.018139 213.174.202.41 192.168.31.191 TCP http > 1851 [SYN, ACK] Seq=0 Ack=1 Win=32767 Len=0 MSS=16856
3 0.018416 192.168.31.191 213.174.202.41 TCP 1851 > http [ACK] Seq=1 Ack=1 Win=65535 Len=0
4 0.020234 192.168.31.191 213.174.202.41 HTTP GET /live/styles/main/home.css HTTP/1.1
5 0.051078 213.174.202.41 192.168.31.191 TCP http > 1851 [ACK] Seq=1 Ack=408 Win=32767 Len=0
6 0.054043 213.174.202.41 192.168.31.191 HTTP Continuation or non-HTTP traffic
7 0.054246 192.168.31.191 213.174.202.41 TCP [TCP Dup ACK 3#1] 1851 > http [ACK] Seq=408 Ack=1 Win=65535 Len=0 SLE=1461 SRE=1674
10 16.051440 192.168.31.191 213.174.202.41 TCP 1845 > http [FIN, ACK] Seq=0 Ack=0 Win=65535 Len=0
11 54.377013 192.168.31.191 213.174.202.41 TCP [TCP Retransmission] 1845 > http [FIN, ACK] Seq=0 Ack=0 Win=65535 Len=0
(packet 6 was seq 1461, ack 408, len 213). As you can see, a 1460-byte
packet went missing between packets 5 and 6.
Snooping on ppp0, I see nearly the same thing, with the internal IP address
replaced by the external IP (correctly, by the masquerading rule). Once
again, the 1460-byte packet is missing. However, and worryingly, the
retransmitted packet, packet 11, and subsequent retransmissions, _appear_
(according to tcpdump and ethereal) to still have the original (RFC1918)
source address. Is this a config error, a reporting error or a bug? :)
Snooping at the endpoint (213.174.202.41 in this example) showed:
21:34:43.491939 85.210.143.231.1851 > 213.174.202.41.http: S 2346403573:2346403573(0) win 65535 <mss 1460,nop,nop,sackOK> (DF)
21:34:43.492005 213.174.202.41.http > 213.174.202.41.1851: S 4283534505:4283534505(0) ack 2346403574 win 32767 <mss 16856,nop,nop,sackOK> (DF)
21:34:43.509632 85.210.143.231.1851 > 213.174.202.41.http: . ack 1 win 65535 (DF)
21:34:43.524462 85.210.143.231.1851 > 213.174.202.41.http: P 1:408(407) ack 1 win 65535 (DF)
21:34:43.524484 213.174.202.41.http > 85.210.143.231.1851: . ack 408 win 32767 (DF)
21:34:43.525913 213.174.202.41.http > 85.210.143.231.1851: . 1:1461(1460) ack 408 win 32767 (DF)
21:34:43.525935 213.174.202.41.http > 85.210.143.231.1851: P 1461:1674(213) ack 408 win 32767 (DF)
21:34:43.544692 85.210.143.231.1851 > 213.174.202.41.http: . ack 1 win 65535 <nop,nop,sack sack 1 {1461:1674} > (DF)
21:34:46.528362 213.174.202.41.http > 85.210.143.231.1851: . 1:1461(1460) ack 408 win 32767 (DF)
21:34:52.527773 213.174.202.41.http > 85.210.143.231.1851: . 1:1461(1460) ack 408 win 32767 (DF)
21:35:04.526655 213.174.202.41.http > 85.210.143.231.1851: . 1:1461(1460) ack 408 win 32767 (DF)
21:35:28.524181 213.174.202.41.http > 85.210.143.231.1851: . 1:1461(1460) ack 408 win 32767 (DF)
Everything matches up, and you can see the 1460-byte packets going out,
which never make it. But the reason I think this part of the problem is
potentially netfilter related, rather than a connection configuration
issue, is that the gateway box itself can access all of these affected site
with no problem at all!
I'd be grateful for any suggestions, I've currently run out of ideas of
things to try...
The first thing that comes to mind is (I know that I have said this about 3 or 4 times in the last week to various people on and off this list) that your Path MTU value is killing things. If you notice in the 2nd traffic dump (tcpdump output) the Don't Fragment (DF) flag is set on the packet. I would be willing to bet that the returning traffic is being dropped b/c the packets are too large to fit in the frame of the PPP over ATM frame.
21:34:43.491939 85.210.143.231.1851 > 213.174.202.41.http: S 2346403573:2346403573(0) win 65535 <mss 1460,nop,nop,sackOK> (DF)
21:34:43.492005 213.174.202.41.http > 213.174.202.41.1851: S 4283534505:4283534505(0) ack 2346403574 win 32767 <mss 16856,nop,nop,sackOK> (DF)
21:34:43.509632 85.210.143.231.1851 > 213.174.202.41.http: . ack 1 win 65535 (DF)
21:34:43.524462 85.210.143.231.1851 > 213.174.202.41.http: P 1:408(407) ack 1 win 65535 (DF)
21:34:43.524484 213.174.202.41.http > 85.210.143.231.1851: . ack 408 win 32767 (DF)
21:34:43.525913 213.174.202.41.http > 85.210.143.231.1851: . 1:1461(1460) ack 408 win 32767 (DF)
21:34:43.525935 213.174.202.41.http > 85.210.143.231.1851: P 1461:1674(213) ack 408 win 32767 (DF)
21:34:43.544692 85.210.143.231.1851 > 213.174.202.41.http: . ack 1 win 65535 <nop,nop,sack sack 1 {1461:1674} > (DF)
If you notice the next 4 packets are duplications of the 3rd packet from the server to the client.
21:34:46.528362 213.174.202.41.http > 85.210.143.231.1851: . 1:1461(1460) ack 408 win 32767 (DF)
--------------- This packet is approximately 3 seconds after the 1st transmission.
21:34:52.527773 213.174.202.41.http > 85.210.143.231.1851: . 1:1461(1460) ack 408 win 32767 (DF)
--------------- This packet is approximately 6 seconds after the 2nd transmission.
21:35:04.526655 213.174.202.41.http > 85.210.143.231.1851: . 1:1461(1460) ack 408 win 32767 (DF)
--------------- This packet is approximately 12 seconds after the 3rd transmission.
21:35:28.524181 213.174.202.41.http > 85.210.143.231.1851: . 1:1461(1460) ack 408 win 32767 (DF)
--------------- This packet is approximately 24 seconds after the 4th transmission.
If you notice in your tcpdump output from the server 5 packets are identical to each other. These packets also have the DF flag set. If you will notice the timestamps between the last 4 packets from the server is doubling in delay between each packet. This is as if the server is going through it's TCP/IP backout timer assuming that things are congested on the way back to the client.
The ""missing packet you are referring to is a fairly small packet and could easily be a datagram that exceeded the TCP/IP window size of the packet and thus was forced to wrap it the remaining data in to a subsequent packet. Seeing as how the DF flag is set I could see how things would get lost on their way back to the client.
I would like to see a TCPDump output from the client system verses the dump that you did provide as comparing apples to apples is much easier.
As far as a solution to you try adding a TCPMSS target to your FORWARD chain as such:
iptables -t filter -A FORWARD -j TCPMSS --clamp-mss-to-pmtu
Grant. . . .
