Hi, I have a strange problem I’ve been troubleshooting for days to no avail. I have traffic from a host (192.168.2.2) that appears to skip the NAT table altogether for TCP traffic but not for ICMP traffic. My goal is to SNAT all traffic from the host but the SNAT occurs for ICMP traffic but not for TCP traffic?! I have a Linux host that has KVM VMs. I've configured the host as a router doing SNAT (well actually masquerading in this example, but I tried SNAT earlier): iptables SNAT rule: Chain PREROUTING (policy ACCEPT 59 packets, 4855 bytes) pkts bytes target prot opt in out source destination 69 6603 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 7 prefix "PRE-RTG-nat:" Chain INPUT (policy ACCEPT 43 packets, 2697 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 88 packets, 5292 bytes) pkts bytes target prot opt in out source destination 88 5292 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 7 prefix "OUT-nat:" Chain POSTROUTING (policy ACCEPT 61 packets, 3686 bytes) pkts bytes target prot opt in out source destination 120 6965 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 7 prefix "POST-RTG-nat:" 431 21166 MASQUERADE all -- * ens160 0.0.0.0/0 0.0.0.0/0 The IP address of the ens160 interface is 10.8.131.129. Example tcpdump output captured on the KVM host when pinging a random host 5.5.5.5 from VM 192.168.2.2 tcpdump -ni ens160 port 80 or icmp 20:09:43.547144 IP 10.8.131.129 > 5.5.5.5: ICMP echo request, id 52336, seq 1, length 64 20:09:44.550362 IP 10.8.131.129 > 5.5.5.5: ICMP echo request, id 52336, seq 2, length 64 20:09:45.574299 IP 10.8.131.129 > 5.5.5.5: ICMP echo request, id 52336, seq 3, length 64 20:09:46.597927 IP 10.8.131.129 > 5.5.5.5: ICMP echo request, id 52336, seq 4, length 64 Example tcpdump output from the KVM host when curl 5.5.5.5 from VM 192.168.2.2: tcpdump -ni ens160 port 80 or icmp listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes 20:11:48.421960 IP 192.168.2.2.25867 > 5.5.5.5.80: Flags [S], seq 2120886070, win 64240, options [mss 1460,sackOK,TS val 2611268392 ecr 0,nop,wscale 7], length 0 20:11:49.446042 IP 192.168.2.2.25867 > 5.5.5.5.80: Flags [S], seq 2120886070, win 64240, options [mss 1460,sackOK,TS val 2611269421 ecr 0,nop,wscale 7], length 0 20:11:51.461925 IP 192.168.2.2.25867 > 5.5.5.5.80: Flags [S], seq 2120886070, win 64240, options [mss 1460,sackOK,TS val 2611271437 ecr 0,nop,wscale 7], length 0 Notice how for the TCP traffic the source address is not NATed, 192.168.2.2, but for ICMP it NATs the traffic with the source reported as 10.8.131.129. It seems the SNAT rule is not applying! I though next to add the LOG statements, as shown earlier in the iptables rules above (there is a similar LOG rule in the FORWARD chain of the filter table that outputs FWD-filter log statements not shown in the rules above), here is the output in the kernel logs when pinging: Dec 1 20:14:18 XXXXX kernel: [16654.813561] PRE-RTG-nat:IN=veth1 OUT= MAC=4a:69:c2:22:60:28:b6:15:50:68:fc:76:08:00 SRC=192.168.2.2 DST=5.5.5.5 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=29316 DF PROTO=ICMP TYPE=8 CODE=0 ID=29910 SEQ=1 Dec 1 20:14:18 XXXXX kernel: [16654.813582] FWD-filter:IN=veth1 OUT=ens160 MAC=4a:69:c2:22:60:28:b6:15:50:68:fc:76:08:00 SRC=192.168.2.2 DST=5.5.5.5 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=29316 DF PROTO=ICMP TYPE=8 CODE=0 ID=29910 SEQ=1 Dec 1 20:14:18 XXXXX kernel: [16654.813588] POST-RTG-nat:IN= OUT=ens160 SRC=192.168.2.2 DST=5.5.5.5 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=29316 DF PROTO=ICMP TYPE=8 CODE=0 ID=29910 SEQ=1 Dec 1 20:14:18 XXXXX kernel: [16654.813561] PRE-RTG-nat:IN=veth1 OUT= MAC=4a:69:c2:22:60:28:b6:15:50:68:fc:76:08:00 SRC=192.168.2.2 DST=5.5.5.5 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=29316 DF PROTO=ICMP TYPE=8 CODE=0 ID=29910 SEQ=1 Dec 1 20:14:18 XXXXX kernel: [16654.813582] FWD-filter:IN=veth1 OUT=ens160 MAC=4a:69:c2:22:60:28:b6:15:50:68:fc:76:08:00 SRC=192.168.2.2 DST=5.5.5.5 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=29316 DF PROTO=ICMP TYPE=8 CODE=0 ID=29910 SEQ=1 Dec 1 20:14:18 XXXXX kernel: [16654.813588] POST-RTG-nat:IN= OUT=ens160 SRC=192.168.2.2 DST=5.5.5.5 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=29316 DF PROTO=ICMP TYPE=8 CODE=0 ID=29910 SEQ=1 Here are the kernel logs when curl/TCP: ec 1 20:18:33 XXXXXX kernel: [16910.155962] FWD-filter:IN=veth1 OUT=ens160 MAC=4a:69:c2:22:60:28:b6:15:50:68:fc:76:08:00 SRC=192.168.2.2 DST=5.5.5.5 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=8112 DF PROTO=TCP SPT=33380 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 Dec 1 20:18:33 XXXXXX kernel: [16910.155962] FWD-filter:IN=veth1 OUT=ens160 MAC=4a:69:c2:22:60:28:b6:15:50:68:fc:76:08:00 SRC=192.168.2.2 DST=5.5.5.5 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=8112 DF PROTO=TCP SPT=33380 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 Dec 1 20:18:34 XXXXXX kernel: [16911.169639] FWD-filter:IN=veth1 OUT=ens160 MAC=4a:69:c2:22:60:28:b6:15:50:68:fc:76:08:00 SRC=192.168.2.2 DST=5.5.5.5 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=8113 DF PROTO=TCP SPT=33380 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 Dec 1 20:18:34 XXXXXX kernel: [16911.169639] FWD-filter:IN=veth1 OUT=ens160 MAC=4a:69:c2:22:60:28:b6:15:50:68:fc:76:08:00 SRC=192.168.2.2 DST=5.5.5.5 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=8113 DF PROTO=TCP SPT=33380 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 I ran a follow-up test to determine if the traffic is in the POSTROUTING mangle table and it is there for both the ICMP and TCP traffic: ICMP Dec 1 20:31:29 XXXXXX kernel: [17685.625732] FWD-filter:IN=veth1 OUT=ens160 MAC=4a:69:c2:22:60:28:b6:15:50:68:fc:76:08:00 SRC=192.168.2.2 DST=5.5.5.5 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=4680 DF PROTO=ICMP TYPE=8 CODE=0 ID=25900 SEQ=1 Dec 1 20:31:29 XXXXXX kernel: [17685.625738] RULE-PSTRTG-mangle:IN= OUT=ens160 SRC=192.168.2.2 DST=5.5.5.5 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=4680 DF PROTO=ICMP TYPE=8 CODE=0 ID=25900 SEQ=1 Dec 1 20:31:29 XXXXXX kernel: [17685.625743] POST-RTG-nat:IN= OUT=ens160 SRC=192.168.2.2 DST=5.5.5.5 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=4680 DF PROTO=ICMP TYPE=8 CODE=0 ID=25900 SEQ=1 TCP Dec 1 20:31:37 XXXXXX kernel: [17694.202915] FWD-filter:IN=veth1 OUT=ens160 MAC=4a:69:c2:22:60:28:b6:15:50:68:fc:76:08:00 SRC=192.168.2.2 DST=5.5.5.5 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=8604 DF PROTO=TCP SPT=62653 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 Dec 1 20:31:37 XXXXXX kernel: [17694.202926] RULE-PSTRTG-mangle:IN= OUT=ens160 SRC=192.168.2.2 DST=5.5.5.5 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=8604 DF PROTO=TCP SPT=62653 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 The traffic seems to skip the NAT table altogether when it's TCP traffic. Any idea what I'm doing wrong? Thanks, Nick
