Hello, I am facing a strange problem with recent kernels. On "bad" kernel, nf_conntrack_tcp_timeout_established default value is not honored, and conntrack -L return different results on the same machine in different ssh root sessions. Ubuntu vendor kernel 4.15 (64bits) : correct behaviour Ubuntu 5.3.0 vendor kernel (64bits): BAD (with iptable 1.6.1 -> iptable rules) Debian kernel 5.4.19 (32bits): BAD (with iptable-nft -> nft rules) Clean boot, no conntrack module loaded: # cat /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established cat: /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established: No such file or directory # modprobe nf_conntrack # cat /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established 432000 Add an ip table rule to start connection tracking: # iptable -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT show tcp session tracking : # conntrack -L |grep ^tcp tcp 6 299 ESTABLISHED src=10.222.219.164 dst=10.222.219.8 sport=54470 dport=22 src=10.222.219.8 dst=10.222.219.164 sport=22 dport=54470 [ASSURED] mark=0 use=1 timeout is not 432000s but 300s. On a moderated loaded smtp server, all sessions are at 300s do # echo 432000 >/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established sometimes sessions start to pick 432000 as new timeout sometimes not. Force things to happen: # conntrack -F # conntrack -L |grep ^tcp |grep ESTABLISHED |grep ASSURED now on the loaded server, most tcp sessions pick the 432000 timeout value, but time to time some still pick 300s. On the debian test machine tree ssh sessions are opened in tree window (I dont have console on this machine) First ssh session: # conntrack -L |grep ^tcp conntrack v1.4.5 (conntrack-tools): 24 flow entries have been shown. tcp 6 431144 ESTABLISHED src=10.222.219.164 dst=10.222.219.8 sport=55243 dport=22 src=10.222.219.8 dst=10.222.219.164 sport=22 dport=55243 [ASSURED] mark=0 use=1 tcp 6 431120 ESTABLISHED src=10.222.219.8 dst=10.222.219.164 sport=22 dport=55339 src=10.222.219.164 dst=10.222.219.8 sport=55339 dport=22 [ASSURED] mark=0 use=1 tcp 6 299 ESTABLISHED src=10.222.219.164 dst=10.222.219.8 sport=54470 dport=22 src=10.222.219.8 dst=10.222.219.164 sport=22 dport=54470 [ASSURED] mark=0 use=1 second one: ~# conntrack -L |grep ^tcp conntrack v1.4.5 (conntrack-tools): 27 flow entries have been shown. tcp 6 431099 ESTABLISHED src=10.222.219.164 dst=10.222.219.8 sport=55243 dport=22 src=10.222.219.8 dst=10.222.219.164 sport=22 dport=55243 [ASSURED] mark=0 use=1 tcp 6 431999 ESTABLISHED src=10.222.219.8 dst=10.222.219.164 sport=22 dport=55339 src=10.222.219.164 dst=10.222.219.8 sport=55339 dport=22 [ASSURED] mark=0 use=1 tcp 6 431963 ESTABLISHED src=10.222.219.164 dst=10.222.219.8 sport=54470 dport=22 src=10.222.219.8 dst=10.222.219.164 sport=22 dport=54470 [ASSURED] mark=0 use=1 last one: # conntrack -L |grep ^tcp conntrack v1.4.5 (conntrack-tools): 22 flow entries have been shown. tcp 6 431999 ESTABLISHED src=10.222.219.164 dst=10.222.219.8 sport=55243 dport=22 src=10.222.219.8 dst=10.222.219.164 sport=22 dport=55243 [ASSURED] mark=0 use=1 tcp 6 431979 ESTABLISHED src=10.222.219.8 dst=10.222.219.164 sport=22 dport=55339 src=10.222.219.164 dst=10.222.219.8 sport=55339 dport=22 [ASSURED] mark=0 use=1 tcp 6 431942 ESTABLISHED src=10.222.219.164 dst=10.222.219.8 sport=54470 dport=22 src=10.222.219.8 dst=10.222.219.164 sport=22 dport=54470 [ASSURED] mark=0 use=1 crazy no ?!?!..... Ok these are all "vendor" kernels, but the Debian one is pretty genuine. It seems that some upstream bugs are lurking around. Debian kernel 5.2.9 (32bits) seems not affected, but Ubuntu 5.0 is partially affected: 10% of connections (due to some backports ?) On the most affected production machine (Ubuntu with 5.3 kernel), the same conntrack -L invocation sometimes return 300 sometimes 432000 for the same long-running tcp connection. I don't know if it is a netlink problem or a real conntrack timer change on activity on the tcp session. But as my ssh sessions never survive more than 10~15 min I think there is a real problem on the conntrack timers. Any thoughts ? Emmanuel.
