Will Storey <will@xxxxxxxxxxxxx> wrote: > I've been experiencing sporadic timeouts when connecting to daemons on > 127.0.0.1. I narrowed the cause down to an iptables INPUT rule that blocks > INVALID state packets: > > 603K 24M DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID > > I can work around this by allowing everything on lo before this rule, but > I'm wondering if this is expected or not. Its not. > All involved systems are running Ubuntu Bionic with kernel > 4.15.0-52-generic. > > On systems with the problem, there are half open TCP connections: > > tcp 0 0 127.0.0.1:2348 127.0.0.1:47268 ESTABLISHED > > When a client connects with source port 47268, it gets stuck in SYN_SENT > and eventually times out: > > 22:09:17.601482 IP (tos 0x0, ttl 64, id 53505, offset 0, flags [DF], proto TCP (6), length 60) > 127.0.0.1.47268 > 127.0.0.1.2348: Flags [S], cksum 0xfe30 (incorrect -> 0x02e6), seq 3436316390, win 43690, options [mss 65495,sackOK,TS val 712761924 ecr 0,nop,wscale 7], length 0 > 22:09:17.601487 IP (tos 0x0, ttl 64, id 42105, offset 0, flags [DF], proto TCP (6), length 52) > 127.0.0.1.2348 > 127.0.0.1.47268: Flags [.], cksum 0xfe28 (incorrect -> 0x08f5), seq 1489307482, ack 3500129728, win 2309, options [nop,nop,TS val 712761924 ecr 696680490], length 0 Looks like ACK is marked as invalid. What is the conntrack state before the syn? (conntrack -L)? > My understanding is that I should see a RST from the client and the > handshake beginning from scratch Unless connection is still established, yes. > From what I can gather, either the ACK from the server or the RST from the > client (which doesn't show in the tcpdump if it is occurring) is getting > blocked by the INVALID state rule. If I allow everything on lo, I see the > RST and the connection succeeds. > > I've tried setting nf_conntrack_log_invalid to 255, but I don't see any > logs about what's invalid. That switch was broken from 4.15 to 5.1. You could ask the distro maintainers to pick up commit d48668052b2603b6262459625c86108c493588dd (netfilter: fix nf_l4proto_log_invalid to log invalid packets). > I've attached packet captures. One shows a case where the timeout happens > (synack_loop_timeout). The other is a case where I created a half open > connection and the timeout didn't occur (expected_rst). > > What do you think? It would be interesting to see the conntrack state right before the syn packet gets picked up.
