On Mon 2019-07-08 15:49:44 +0000, Florian Westphal wrote: > > When a client connects with source port 47268, it gets stuck in SYN_SENT > > and eventually times out: > > > > 22:09:17.601482 IP (tos 0x0, ttl 64, id 53505, offset 0, flags [DF], proto TCP (6), length 60) > > 127.0.0.1.47268 > 127.0.0.1.2348: Flags [S], cksum 0xfe30 (incorrect -> 0x02e6), seq 3436316390, win 43690, options [mss 65495,sackOK,TS val 712761924 ecr 0,nop,wscale 7], length 0 > > 22:09:17.601487 IP (tos 0x0, ttl 64, id 42105, offset 0, flags [DF], proto TCP (6), length 52) > > 127.0.0.1.2348 > 127.0.0.1.47268: Flags [.], cksum 0xfe28 (incorrect -> 0x08f5), seq 1489307482, ack 3500129728, win 2309, options [nop,nop,TS val 712761924 ecr 696680490], length 0 > > Looks like ACK is marked as invalid. > What is the conntrack state before the syn? > (conntrack -L)? # conntrack -L | grep 42190 tcp 6 339840 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=42190 dport=2346 src=127.0.0.1 dst=127.0.0.1 sport=2346 dport=42190 [ASSURED] mark=0 use=1 We found if we remove that entry, we see the RST. Likewise if we tune down net.netfilter.nf_conntrack_tcp_timeout_established from 5 days. > > From what I can gather, either the ACK from the server or the RST from the > > client (which doesn't show in the tcpdump if it is occurring) is getting > > blocked by the INVALID state rule. If I allow everything on lo, I see the > > RST and the connection succeeds. > > > > I've tried setting nf_conntrack_log_invalid to 255, but I don't see any > > logs about what's invalid. > > That switch was broken from 4.15 to 5.1. > You could ask the distro maintainers to pick up > commit d48668052b2603b6262459625c86108c493588dd > (netfilter: fix nf_l4proto_log_invalid to log invalid packets). Good to know! Another thing I should mention: We started seeing this behaviour after updating from Ubuntu Trusty. I'm having trouble finding the kernel version Trusty was on right now though, but I thought it worth mentioning.
