I’m running iptables on a fedora 20 vm, kernel 3.18.5-101.fc20.i686+PAE. I’ve also had the same issue with a 3.17 kernel, but don’t have the logged data.
The setup is quite vanilla, but I seem to be losing conntrack connections on a regular basis (maybe a few per hour). If I do a regular dump of the conntrack data (conntrack -L -o ktimestamp), every 100s, I can occasionally catch this sort of output, which I think is occurring immediately before the lost connection:
tcp 6 431996 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=20 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 431987 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=120 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 431997 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=220 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 431936 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=320 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 431991 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=420 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 431996 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=520 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 431980 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=620 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 431880 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=720 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 431780 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=820 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 431680 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=920 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 431580 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=1020 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 431480 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=1120 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 431380 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=1221 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 431280 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=1321 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 431179 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=1421 [start=Wed Feb 11 09:07:14 2015] use=1
tcp 6 295 ESTABLISHED src=172.17.1.111 dst=50.22.240.167 sport=50119 dport=5222 src=50.22.240.167 dst=192.168.31.60 sport=5222 dport=50119 [ASSURED] mark=0 delta-time=1521 [start=Wed Feb 11 09:07:14 2015] use=1
The vm has an IP address of 192.168.31.60, and an address on the 172.17.0.0/16 subnet.
The lines are separated by 100s. If I understand that output correctly, the tcp connection between 172.17.1.111 and 50.22.240.167 is initially being used, and then the timeout starts to tick down from ~431980 seconds to 431179 as one would expect. And then 100s later, it drops to 295. The connection is missing 100s after that line.
I’ve not fully nailed it down, but I suspect that the connection then gets timed out and is therfore no longer present when one end tries to use it.
I cannot find a known bug for this behaviour, and I could be misinterpreting how conntrack should behave (eg I may have missed a timeout), but I don’t think so.
Can anyone confirm that there is an issue here, or where else I should look/what I can look at?
iptables reports these rules, which show the dropped packets. The uptime at this point was just under two days:
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
2027K 2061M ACCEPT all -- any any anywhere anywhere state RELATED,ESTABLISHED
19 1400 ACCEPT icmp -- any any anywhere anywhere
0 0 ACCEPT all -- lo any anywhere anywhere
10 704 ACCEPT tcp -- any any anywhere anywhere state NEW tcp dpt:ssh
0 0 ACCEPT udp -- any any anywhere anywhere udp dpt:bootpc
151 53741 ACCEPT udp -- any any anywhere anywhere udp dpt:bootps
20611 2020K ACCEPT all -- any any 172.17.0.0/16 anywhere
113 21844 LOG all -- any any anywhere anywhere limit: avg 2/min burst 5 LOG level warning prefix "dropping: "
133 27243 REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT 39689 packets, 3595K bytes)
pkts bytes target prot opt in out source destination
632K 398M ACCEPT all -- any any anywhere anywhere state RELATED,ESTABLISHED
0 0 ACCEPT icmp -- any any anywhere anywhere
0 0 ACCEPT all -- lo any anywhere anywhere
Chain OUTPUT (policy ACCEPT 2041K packets, 1307M bytes)
pkts bytes target prot opt in out source destination
tia
Tim--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
