I hate replying to my own email, but after discussions with a
knowledgeable person in the Linux advanced routing and traffic control
IRC channel, I simplified things down to just five iptable rules and
simplified the routing tables, unfortunately the result is still exactly
the same.
Here is hopefully all the necessary information including tcpdump output
with one route being successful and another failing:
root@server:/etc# ip route show table main | sort
10.8.0.0/24 dev tun0 proto kernel scope link src 10.8.0.2 metric 10
174.4.4.0/22 dev eth0 proto kernel scope link src 174.4.4.74 metric 10
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.1
metric 10
50.92.224.0/19 dev eth2 proto kernel scope link src 50.92.247.211
metric 10
root@server:/etc# ip route show table ISP1 | sort
default via 174.4.4.1 dev eth0
root@server:/etc# ip route show table ISP2 | sort
default via 50.92.224.1 dev eth2
root@server:/etc# ip route show table VPN1 | sort
default via 10.8.0.2 dev tun0
root@server:/etc# iptables -L -n -v
Chain INPUT (policy ACCEPT 1327 packets, 110K bytes)
pkts bytes target prot opt in out source
destination
Chain FORWARD (policy ACCEPT 4688 packets, 1071K bytes)
pkts bytes target prot opt in out source
destination
Chain OUTPUT (policy ACCEPT 766 packets, 82125 bytes)
pkts bytes target prot opt in out source
destination
root@server:/etc# iptables -L -n -v -t nat
Chain PREROUTING (policy ACCEPT 241 packets, 20425 bytes)
pkts bytes target prot opt in out source
destination
Chain INPUT (policy ACCEPT 37 packets, 6564 bytes)
pkts bytes target prot opt in out source
destination
Chain OUTPUT (policy ACCEPT 6 packets, 456 bytes)
pkts bytes target prot opt in out source
destination
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
144 9466 SNAT all -- * eth0 0.0.0.0/0
0.0.0.0/0 to:174.4.4.74
66 4851 SNAT all -- * eth2 0.0.0.0/0
0.0.0.0/0 to:50.92.247.211
0 0 SNAT all -- * tun0 0.0.0.0/0
0.0.0.0/0 to:10.8.0.2
root@server:/etc# iptables -L -n -v -t mangle
Chain PREROUTING (policy ACCEPT 6143 packets, 1200K bytes)
pkts bytes target prot opt in out source
destination
111 8218 MARK all -- eth1 * 192.168.1.236
0.0.0.0/0 MARK set 0x2
2 168 MARK all -- eth1 * 192.168.1.246
0.0.0.0/0 MARK set 0x1
Chain INPUT (policy ACCEPT 1355 packets, 113K bytes)
pkts bytes target prot opt in out source
destination
Chain FORWARD (policy ACCEPT 4770 packets, 1086K bytes)
pkts bytes target prot opt in out source
destination
Chain OUTPUT (policy ACCEPT 787 packets, 84473 bytes)
pkts bytes target prot opt in out source
destination
Chain POSTROUTING (policy ACCEPT 5557 packets, 1170K bytes)
pkts bytes target prot opt in out source
destination
root@server:/etc# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
root@server:/etc# iptables -S -t nat
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -o eth0 -j SNAT --to-source 174.4.4.74
-A POSTROUTING -o eth2 -j SNAT --to-source 50.92.247.211
-A POSTROUTING -o tun0 -j SNAT --to-source 10.8.0.2
root@server:/etc# iptables -S -t mangle
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A PREROUTING -s 192.168.1.236/32 -i eth1 -j MARK --set-xmark 0x2/0xffffffff
-A PREROUTING -s 192.168.1.246/32 -i eth1 -j MARK --set-xmark 0x1/0xffffffff
root@server:/etc# iptables -S -t raw
-P PREROUTING ACCEPT
-P OUTPUT ACCEPT
#SUCCEEDS: TCPDUMP of ping from 192.168.1.246 (packets marked 0x01)
root@server:~# tcpdump -n -p -i eth1 icmp
20:47:46.026827 IP 192.168.1.246 > 8.8.8.8: ICMP echo request, id 14644,
seq 0, length 64
20:47:46.057753 IP 8.8.8.8 > 192.168.1.246: ICMP echo reply, id 14644,
seq 0, length 64
root@server:~# tcpdump -n -p -i eth0 icmp
20:47:46.026854 IP 174.4.4.74 > 8.8.8.8: ICMP echo request, id 14644,
seq 0, length 64
20:47:46.057734 IP 8.8.8.8 > 174.4.4.74: ICMP echo reply, id 14644, seq
0, length 64
#FAILS: TCPDUMP of ping from 192.168.1.236 (packets marked 0x02)
root@server:~# tcpdump -n -p -i eth1 icmp
20:37:41.852604 IP 192.168.1.236 > 8.8.8.8: ICMP echo request, id 3636,
seq 0, length 64
root@server:~# tcpdump -n -p -i eth2 icmp
20:37:41.852642 IP 50.92.247.211 > 8.8.8.8: ICMP echo request, id 3636,
seq 0, length 64
20:37:41.906329 IP 8.8.8.8 > 50.92.247.211: ICMP echo reply, id 3636,
seq 0, length 64
Thanks.
On 11-09-11 04:06 PM, Mike wrote:
I'm in the process of upgrading an older Linux router from Mandriva
running kernel v2.6.16 to Ubuntu running v2.6.38 kernel, however my
moderately complex firewall/routing script doesn't quite work the same
way on the newer system. The basic idea is that I have three routes to
three different ISPs, and one to the internal network. I then mark
packets to go out a specific ISP depending on the type of traffic.
This all works fine if the packets are initiated from the router
itself or from a computer on the intenral network with packets
destined out the default ISP, but it fails completely if the packets
are initiated from a computer on the internal network destined out an
non-default route.
What I don't understand is I diff'd the routing tables and all
iptables commands they are virtually identical between the two
servers, yet the newer server doesn't work as expected.
Linux server 2.6.38-10-server #44-Ubuntu SMP Thu Jun 2 21:49:30 UTC
2011 x86_64 x86_64 x86_64 GNU/Linux
eth0 = ISP1
eth1 = Local network
eth2 = ISP2
tun0 = VPN to ISP3
root@server:/etc# ip route show | sort
10.8.0.0/24 dev tun0 proto kernel scope link src 10.8.0.2
174.4.4.74 dev eth0 scope link src 174.4.4.74
174.4.4.0/22 dev eth0 proto kernel scope link src 174.4.4.74
metric 10
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.1
metric 10
50.92.224.0/19 dev eth2 proto kernel scope link src 50.92.247.211
metric 10
50.92.247.211 dev eth2 scope link src 50.92.247.211
63.211.239.14 via 50.92.224.1 dev eth2
8.3.252.23 via 50.92.224.1 dev eth2
default via 174.4.4.1 dev eth0
root@server:~# ip rule show
0: from all lookup local
32760: from all fwmark 0x3 lookup VPN1
32761: from all fwmark 0x2 lookup ISP2
32762: from all fwmark 0x1 lookup ISP1
32763: from 10.8.0.2 lookup VPN1
32764: from 50.92.247.211 lookup ISP2
32765: from 174.4.4.74 lookup ISP1
32766: from all lookup main
32767: from all lookup default
root@server:~# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -s 207.150.193.134/32 -p tcp -m tcp --dport 5060:5061 -j ACCEPT
-A INPUT -s 64.34.96.201/32 -p tcp -m tcp --dport 5060:5061 -j ACCEPT
-A INPUT -s 64.34.96.202/32 -p tcp -m tcp --dport 5060:5061 -j ACCEPT
-A INPUT -s 8.3.252.23/32 -p tcp -m tcp --dport 5060:5061 -j ACCEPT
-A INPUT -s 63.211.239.14/32 -p tcp -m tcp --dport 5060:5061 -j ACCEPT
-A INPUT -s 207.150.193.134/32 -p udp -m udp --dport 5060:5061 -j ACCEPT
-A INPUT -s 64.34.96.201/32 -p udp -m udp --dport 5060:5061 -j ACCEPT
-A INPUT -s 64.34.96.202/32 -p udp -m udp --dport 5060:5061 -j ACCEPT
-A INPUT -s 8.3.252.23/32 -p udp -m udp --dport 5060:5061 -j ACCEPT
-A INPUT -s 63.211.239.14/32 -p udp -m udp --dport 5060:5061 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 5060:5061 -j DROP
-A INPUT -i eth2 -p tcp -m tcp --dport 5060:5061 -j DROP
-A INPUT -i eth0 -p udp -m udp --dport 5060:5061 -j DROP
-A INPUT -i eth2 -p udp -m udp --dport 5060:5061 -j DROP
-A INPUT -s 68.75.86.8/32 -j DROP
-A INPUT -s 174.133.3.178/32 -j DROP
root@server:~# iptables -S -t nat
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A PREROUTING -i eth0 -p tcp -m tcp --dport 88 -j DNAT
--to-destination 192.168.1.19
-A PREROUTING -i eth0 -p tcp -m tcp --dport 3074 -j DNAT
--to-destination 192.168.1.19
-A PREROUTING -i eth2 -p tcp -m tcp --dport 88 -j DNAT
--to-destination 192.168.1.19
-A PREROUTING -i eth2 -p tcp -m tcp --dport 3074 -j DNAT
--to-destination 192.168.1.19
-A PREROUTING -i eth0 -p tcp -m tcp --dport 8080 -j DNAT
--to-destination 192.168.1.9:80
-A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -j DNAT
--to-destination 192.168.1.9:443
-A PREROUTING -i eth0 -p tcp -m tcp --dport 4343 -j DNAT
--to-destination 192.168.1.9:443
-A PREROUTING -i eth0 -p tcp -m tcp --dport 69 -j DNAT
--to-destination 192.168.1.9:69
-A PREROUTING -i eth0 -p udp -m udp --dport 69 -j DNAT
--to-destination 192.168.1.9:69
-A PREROUTING -i eth0 -p tcp -m tcp --dport 22 -j DNAT
--to-destination 192.168.1.9:22
-A PREROUTING -i eth0 -p tcp -m tcp --dport 2323 -j DNAT
--to-destination 192.168.1.201:23
-A PREROUTING -i eth0 -p tcp -m tcp --dport 2380 -j DNAT
--to-destination 192.168.1.201:80
-A PREROUTING -i eth0 -p tcp -m tcp --dport 5501 -j DNAT
--to-destination 192.168.1.98:5501
-A PREROUTING -i eth0 -p tcp -m tcp --dport 5800 -j DNAT
--to-destination 192.168.1.98:5800
-A PREROUTING -i eth0 -p tcp -m tcp --dport 5900 -j DNAT
--to-destination 192.168.1.98:5900
-A PREROUTING -i eth0 -p tcp -m tcp --dport 5901 -j DNAT
--to-destination 192.168.1.98:5901
-A PREROUTING -i eth0 -p tcp -m tcp --dport 5902 -j DNAT
--to-destination 192.168.1.98:5902
-A PREROUTING -i eth0 -p tcp -m tcp --dport 5903 -j DNAT
--to-destination 192.168.1.98:5903
-A PREROUTING -i eth0 -p tcp -m tcp --dport 5904 -j DNAT
--to-destination 192.168.1.98:5904
-A PREROUTING -i eth0 -p tcp -m tcp --dport 5910 -j DNAT
--to-destination 192.168.1.9:5900
-A PREROUTING -i eth0 -p tcp -m tcp --dport 40696 -j DNAT
--to-destination 192.168.1.99:40696
-A PREROUTING -i eth0 -p tcp -m tcp --dport 50263 -j DNAT
--to-destination 192.168.1.9:50263
-A PREROUTING -i eth0 -p udp -m udp --dport 4444 -j DNAT
--to-destination 192.168.1.9:4444
-A PREROUTING -i eth0 -p udp -m udp --dport 6881 -j DNAT
--to-destination 192.168.1.9:6881
-A PREROUTING -i eth0 -p tcp -m tcp --dport 6881 -j DNAT
--to-destination 192.168.1.9:6881
-A PREROUTING -i eth0 -p udp -m udp --dport 1200 -j DNAT
--to-destination 192.168.1.98:1200
-A PREROUTING -i eth0 -p udp -m udp --dport 27000:27015 -j DNAT
--to-destination 192.168.1.98
-A PREROUTING -i eth0 -p tcp -m tcp --dport 27030:27039 -j DNAT
--to-destination 192.168.1.98
-A POSTROUTING -o tun0 -j SNAT --to-source 10.8.0.2
-A POSTROUTING -o eth2 -j SNAT --to-source 50.92.247.211
-A POSTROUTING -o eth0 -j SNAT --to-source 174.4.4.74
root@server:~# iptables -S -t mangle
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A PREROUTING -i eth1 -p udp -m udp --dport 4569 -j MARK --set-xmark
0x2/0xffffffff
-A PREROUTING -p udp -m udp --dport 5060:5061 -j MARK --set-xmark
0x2/0xffffffff
-A PREROUTING -p udp -m udp --dport 10000:20000 -j MARK --set-xmark
0x2/0xffffffff
-A PREROUTING -s 192.168.1.19/32 -i eth1 -j MARK --set-xmark
0x3/0xffffffff
-A PREROUTING -d 69.53.236.17/32 -i eth1 -p tcp -m tcp --dport 80 -j
MARK --set-xmark 0x3/0xffffffff
-A PREROUTING -d 69.53.236.17/32 -i eth1 -p tcp -m tcp --dport 443 -j
MARK --set-xmark 0x3/0xffffffff
-A PREROUTING -d 24.244.52.99/32 -i eth1 -p tcp -m tcp --dport 80 -j
MARK --set-xmark 0x3/0xffffffff
-A PREROUTING -d 24.244.52.81/32 -i eth1 -p tcp -m tcp --dport 80 -j
MARK --set-xmark 0x3/0xffffffff
-A PREROUTING -d 24.244.52.104/32 -i eth1 -p tcp -m tcp --dport 80 -j
MARK --set-xmark 0x3/0xffffffff
-A PREROUTING -d 24.244.52.83/32 -i eth1 -p tcp -m tcp --dport 80 -j
MARK --set-xmark 0x3/0xffffffff
-A PREROUTING -d 24.244.52.104/32 -i eth1 -p tcp -m tcp --dport 443 -j
MARK --set-xmark 0x3/0xffffffff
-A PREROUTING -d 24.244.52.83/32 -i eth1 -p tcp -m tcp --dport 443 -j
MARK --set-xmark 0x3/0xffffffff
-A PREROUTING -d 64.59.168.13/32 -i eth1 -j MARK --set-xmark
0x1/0xffffffff
-A PREROUTING -d 64.59.168.15/32 -i eth1 -j MARK --set-xmark
0x1/0xffffffff
-A PREROUTING -d 154.11.128.187/32 -i eth1 -j MARK --set-xmark
0x2/0xffffffff
-A PREROUTING -d 154.11.128.59/32 -i eth1 -j MARK --set-xmark
0x2/0xffffffff
-A PREROUTING -p tcp -m tcp --sport 80 -j TOS --set-tos 0x10/0x3f
-A PREROUTING -p tcp -m tcp --sport 22 -j TOS --set-tos 0x10/0x3f
-A PREROUTING -p tcp -m tcp --sport 4569 -j TOS --set-tos 0x10/0x3f
-A PREROUTING -p tcp -m tcp --sport 21 -j TOS --set-tos 0x10/0x3f
-A PREROUTING -p tcp -m tcp --sport 20 -j TOS --set-tos 0x08/0x3f
-A PREROUTING -p icmp -m icmp --icmp-type 8 -j TOS --set-tos 0x10/0x3f
-A PREROUTING -p udp -j TOS --set-tos 0x10/0x3f
-A OUTPUT -p tcp -m tcp --dport 80 -j TOS --set-tos 0x10/0x3f
-A OUTPUT -p tcp -m tcp --dport 22 -j TOS --set-tos 0x10/0x3f
-A OUTPUT -p tcp -m tcp --dport 4569 -j TOS --set-tos 0x10/0x3f
-A OUTPUT -p tcp -m tcp --dport 21 -j TOS --set-tos 0x10/0x3f
-A OUTPUT -p tcp -m tcp --dport 20 -j TOS --set-tos 0x08/0x3f
-A OUTPUT -p icmp -m icmp --icmp-type 8 -j TOS --set-tos 0x10/0x3f
-A OUTPUT -p udp -j TOS --set-tos 0x10/0x3f
root@server:~# iptables -S -t raw
-P PREROUTING ACCEPT
-P OUTPUT ACCEPT
I enabled logging on each table/chain to try and diagnose where the
issue is, below is the output of several single packet ping tests. The
lines starting with "a" are logging at the first rule in each
table/chain and lines starting with "z" are the last rule.
SUCCESS: Ping from router (with marking enabled)
Sep 10 15:36:47 server kernel: [11147.602519] aMANGLE:OUTPUT: IN=
OUT=tun0 SRC=10.8.0.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64
ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=22025 SEQ=1 UID=0 GID=0 MARK=0x3
Sep 10 15:36:47 server kernel: [11147.602530] zMANGLE:OUTPUT: IN=
OUT=tun0 SRC=10.8.0.2 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00 TTL=64
ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=22025 SEQ=1 UID=0 GID=0 MARK=0x3
Sep 10 15:36:47 server kernel: [11147.602543] aNAT:OUTPUT: IN=
OUT=tun0 SRC=10.8.0.2 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00 TTL=64
ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=22025 SEQ=1 UID=0 GID=0 MARK=0x3
Sep 10 15:36:47 server kernel: [11147.602550] zNAT:OUTPUT: IN=
OUT=tun0 SRC=10.8.0.2 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00 TTL=64
ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=22025 SEQ=1 UID=0 GID=0 MARK=0x3
Sep 10 15:36:47 server kernel: [11147.602560] aFILTER:OUTPUT: IN=
OUT=tun0 SRC=10.8.0.2 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00 TTL=64
ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=22025 SEQ=1 UID=0 GID=0 MARK=0x3
Sep 10 15:36:47 server kernel: [11147.602567] zFILTER:OUTPUT: IN=
OUT=tun0 SRC=10.8.0.2 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00 TTL=64
ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=22025 SEQ=1 UID=0 GID=0 MARK=0x3
Sep 10 15:36:47 server kernel: [11147.602575] aMANGLE:POSTROUTING: IN=
OUT=tun0 SRC=10.8.0.2 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00 TTL=64
ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=22025 SEQ=1 UID=0 GID=0 MARK=0x3
Sep 10 15:36:47 server kernel: [11147.602583] zMANGLE:POSTROUTING: IN=
OUT=tun0 SRC=10.8.0.2 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00 TTL=64
ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=22025 SEQ=1 UID=0 GID=0 MARK=0x3
Sep 10 15:36:47 server kernel: [11147.734585] aMANGLE:PREROUTING:
IN=tun0 OUT= MAC= SRC=8.8.8.8 DST=10.8.0.2 LEN=84 TOS=0x00 PREC=0x00
TTL=52 ID=64821 PROTO=ICMP TYPE=0 CODE=0 ID=22025 SEQ=1
Sep 10 15:36:47 server kernel: [11147.734594] zMANGLE:PREROUTING:
IN=tun0 OUT= MAC= SRC=8.8.8.8 DST=10.8.0.2 LEN=84 TOS=0x00 PREC=0x00
TTL=52 ID=64821 PROTO=ICMP TYPE=0 CODE=0 ID=22025 SEQ=1
Sep 10 15:36:47 server kernel: [11147.734602] aMANGLE:INPUT: IN=tun0
OUT= MAC= SRC=8.8.8.8 DST=10.8.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=52
ID=64821 PROTO=ICMP TYPE=0 CODE=0 ID=22025 SEQ=1
Sep 10 15:36:47 server kernel: [11147.734608] zMANGLE:INPUT: IN=tun0
OUT= MAC= SRC=8.8.8.8 DST=10.8.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=52
ID=64821 PROTO=ICMP TYPE=0 CODE=0 ID=22025 SEQ=1
Sep 10 15:36:47 server kernel: [11147.734614] aFILTER:INPUT: IN=tun0
OUT= MAC= SRC=8.8.8.8 DST=10.8.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=52
ID=64821 PROTO=ICMP TYPE=0 CODE=0 ID=22025 SEQ=1
Sep 10 15:36:47 server kernel: [11147.734621] zFILTER:INPUT: IN=tun0
OUT= MAC= SRC=8.8.8.8 DST=10.8.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=52
ID=64821 PROTO=ICMP TYPE=0 CODE=0 ID=22025 SEQ=1
FAILS: Ping from Client (with marking enabled 0x3)
Sep 10 15:37:21 server kernel: [11181.508668] aMANGLE:PREROUTING:
IN=eth1 OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00
SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64
ID=56590 PROTO=ICMP TYPE=8 CODE=0 ID=62237 SEQ=0
Sep 10 15:37:21 server kernel: [11181.508682] zMANGLE:PREROUTING:
IN=eth1 OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00
SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00 TTL=64
ID=56590 PROTO=ICMP TYPE=8 CODE=0 ID=62237 SEQ=0 MARK=0x3
Sep 10 15:37:21 server kernel: [11181.508694] aNAT:PREROUTING: IN=eth1
OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00 SRC=192.168.1.236
DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00 TTL=64 ID=56590 PROTO=ICMP
TYPE=8 CODE=0 ID=62237 SEQ=0 MARK=0x3
Sep 10 15:37:21 server kernel: [11181.508704] zNAT:PREROUTING: IN=eth1
OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00 SRC=192.168.1.236
DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00 TTL=64 ID=56590 PROTO=ICMP
TYPE=8 CODE=0 ID=62237 SEQ=0 MARK=0x3
Sep 10 15:37:21 server kernel: [11181.508758] aMANGLE:FORWARD: IN=eth1
OUT=tun0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00
TTL=63 ID=56590 PROTO=ICMP TYPE=8 CODE=0 ID=62237 SEQ=0 MARK=0x3
Sep 10 15:37:21 server kernel: [11181.508765] zMANGLE:FORWARD: IN=eth1
OUT=tun0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00
TTL=63 ID=56590 PROTO=ICMP TYPE=8 CODE=0 ID=62237 SEQ=0 MARK=0x3
Sep 10 15:37:21 server kernel: [11181.508773] aFILTER:FORWARD: IN=eth1
OUT=tun0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00
TTL=63 ID=56590 PROTO=ICMP TYPE=8 CODE=0 ID=62237 SEQ=0 MARK=0x3
Sep 10 15:37:21 server kernel: [11181.508779] zFILTER:FORWARD: IN=eth1
OUT=tun0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00
TTL=63 ID=56590 PROTO=ICMP TYPE=8 CODE=0 ID=62237 SEQ=0 MARK=0x3
Sep 10 15:37:21 server kernel: [11181.508787] aMANGLE:POSTROUTING: IN=
OUT=tun0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00
TTL=63 ID=56590 PROTO=ICMP TYPE=8 CODE=0 ID=62237 SEQ=0 MARK=0x3
Sep 10 15:37:21 server kernel: [11181.508793] zMANGLE:POSTROUTING: IN=
OUT=tun0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00
TTL=63 ID=56590 PROTO=ICMP TYPE=8 CODE=0 ID=62237 SEQ=0 MARK=0x3
Sep 10 15:37:21 server kernel: [11181.642875] aMANGLE:PREROUTING:
IN=tun0 OUT= MAC= SRC=8.8.8.8 DST=10.8.0.2 LEN=84 TOS=0x00 PREC=0x00
TTL=52 ID=64822 PROTO=ICMP TYPE=0 CODE=0 ID=62237 SEQ=0
Sep 10 15:37:21 server kernel: [11181.642885] zMANGLE:PREROUTING:
IN=tun0 OUT= MAC= SRC=8.8.8.8 DST=10.8.0.2 LEN=84 TOS=0x00 PREC=0x00
TTL=52 ID=64822 PROTO=ICMP TYPE=0 CODE=0 ID=62237 SEQ=0
FAILS: Ping from Client (with marking enabled: 0x2)
Sep 11 10:09:19 server kernel: [77836.447776] aMANGLE:PREROUTING:
IN=eth1 OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00
SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64
ID=13133 PROTO=ICMP TYPE=8 CODE=0 ID=812 SEQ=0
Sep 11 10:09:19 server kernel: [77836.447789] zMANGLE:PREROUTING:
IN=eth1 OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00
SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64
ID=13133 PROTO=ICMP TYPE=8 CODE=0 ID=812 SEQ=0 MARK=0x2
Sep 11 10:09:19 server kernel: [77836.447801] aNAT:PREROUTING: IN=eth1
OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00 SRC=192.168.1.236
DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=13133 PROTO=ICMP
TYPE=8 CODE=0 ID=812 SEQ=0 MARK=0x2
Sep 11 10:09:19 server kernel: [77836.447811] zNAT:PREROUTING: IN=eth1
OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00 SRC=192.168.1.236
DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=13133 PROTO=ICMP
TYPE=8 CODE=0 ID=812 SEQ=0 MARK=0x2
Sep 11 10:09:19 server kernel: [77836.447843] aMANGLE:FORWARD: IN=eth1
OUT=eth2 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00
TTL=63 ID=13133 PROTO=ICMP TYPE=8 CODE=0 ID=812 SEQ=0 MARK=0x2
Sep 11 10:09:19 server kernel: [77836.447850] zMANGLE:FORWARD: IN=eth1
OUT=eth2 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00
TTL=63 ID=13133 PROTO=ICMP TYPE=8 CODE=0 ID=812 SEQ=0 MARK=0x2
Sep 11 10:09:19 server kernel: [77836.447857] aFILTER:FORWARD: IN=eth1
OUT=eth2 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00
TTL=63 ID=13133 PROTO=ICMP TYPE=8 CODE=0 ID=812 SEQ=0 MARK=0x2
Sep 11 10:09:19 server kernel: [77836.447863] zFILTER:FORWARD: IN=eth1
OUT=eth2 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00
TTL=63 ID=13133 PROTO=ICMP TYPE=8 CODE=0 ID=812 SEQ=0 MARK=0x2
Sep 11 10:09:19 server kernel: [77836.447870] aMANGLE:POSTROUTING: IN=
OUT=eth2 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00
TTL=63 ID=13133 PROTO=ICMP TYPE=8 CODE=0 ID=812 SEQ=0 MARK=0x2
Sep 11 10:09:19 server kernel: [77836.447877] zMANGLE:POSTROUTING: IN=
OUT=eth2 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00
TTL=63 ID=13133 PROTO=ICMP TYPE=8 CODE=0 ID=812 SEQ=0 MARK=0x2
Sep 11 10:09:19 server kernel: [77836.501409] aMANGLE:PREROUTING:
IN=eth2 OUT= MAC=00:1b:21:8c:07:34:00:90:1a:a0:7c:04:08:00 SRC=8.8.8.8
DST=50.92.247.211 LEN=84 TOS=0x00 PREC=0x00 TTL=54 ID=5396 PROTO=ICMP
TYPE=0 CODE=0 ID=812 SEQ=0
Sep 11 10:09:19 server kernel: [77836.501421] zMANGLE:PREROUTING:
IN=eth2 OUT= MAC=00:1b:21:8c:07:34:00:90:1a:a0:7c:04:08:00 SRC=8.8.8.8
DST=50.92.247.211 LEN=84 TOS=0x00 PREC=0x00 TTL=54 ID=5396 PROTO=ICMP
TYPE=0 CODE=0 ID=812 SEQ=0
SUCCESS: Ping from client (with marking enabled: 0x1 [default ISP1])
Sep 11 15:50:24 server kernel: [19600.171454] aMANGLE:PREROUTING:
IN=eth1 OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00
SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64
ID=17642 PROTO=ICMP TYPE=8 CODE=0 ID=16944 SEQ=0
Sep 11 15:50:24 server kernel: [19600.171467] zMANGLE:PREROUTING:
IN=eth1 OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00
SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64
ID=17642 PROTO=ICMP TYPE=8 CODE=0 ID=16944 SEQ=0 MARK=0x1
Sep 11 15:50:24 server kernel: [19600.171479] aNAT:PREROUTING: IN=eth1
OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00 SRC=192.168.1.236
DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=17642 PROTO=ICMP
TYPE=8 CODE=0 ID=16944 SEQ=0 MARK=0x1
Sep 11 15:50:24 server kernel: [19600.171489] zNAT:PREROUTING: IN=eth1
OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00 SRC=192.168.1.236
DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=17642 PROTO=ICMP
TYPE=8 CODE=0 ID=16944 SEQ=0 MARK=0x1
Sep 11 15:50:24 server kernel: [19600.171500] aMANGLE:FORWARD: IN=eth1
OUT=eth0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00
TTL=63 ID=17642 PROTO=ICMP TYPE=8 CODE=0 ID=16944 SEQ=0 MARK=0x1
Sep 11 15:50:24 server kernel: [19600.171506] zMANGLE:FORWARD: IN=eth1
OUT=eth0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00
TTL=63 ID=17642 PROTO=ICMP TYPE=8 CODE=0 ID=16944 SEQ=0 MARK=0x1
Sep 11 15:50:24 server kernel: [19600.171513] aFILTER:FORWARD: IN=eth1
OUT=eth0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00
TTL=63 ID=17642 PROTO=ICMP TYPE=8 CODE=0 ID=16944 SEQ=0 MARK=0x1
Sep 11 15:50:24 server kernel: [19600.171520] zFILTER:FORWARD: IN=eth1
OUT=eth0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00
TTL=63 ID=17642 PROTO=ICMP TYPE=8 CODE=0 ID=16944 SEQ=0 MARK=0x1
Sep 11 15:50:24 server kernel: [19600.171527] aMANGLE:POSTROUTING: IN=
OUT=eth0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00
TTL=63 ID=17642 PROTO=ICMP TYPE=8 CODE=0 ID=16944 SEQ=0 MARK=0x1
Sep 11 15:50:24 server kernel: [19600.171534] zMANGLE:POSTROUTING: IN=
OUT=eth0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00
TTL=63 ID=17642 PROTO=ICMP TYPE=8 CODE=0 ID=16944 SEQ=0 MARK=0x1
Sep 11 15:50:24 server kernel: [19600.214666] aMANGLE:PREROUTING:
IN=eth0 OUT= MAC=f4:6d:04:9a:07:bd:00:30:b8:c9:5c:90:08:00 SRC=8.8.8.8
DST=174.4.4.74 LEN=84 TOS=0x00 PREC=0x00 TTL=54 ID=59275 PROTO=ICMP
TYPE=0 CODE=0 ID=16944 SEQ=0
Sep 11 15:50:24 server kernel: [19600.214678] zMANGLE:PREROUTING:
IN=eth0 OUT= MAC=f4:6d:04:9a:07:bd:00:30:b8:c9:5c:90:08:00 SRC=8.8.8.8
DST=174.4.4.74 LEN=84 TOS=0x00 PREC=0x00 TTL=54 ID=59275 PROTO=ICMP
TYPE=0 CODE=0 ID=16944 SEQ=0
Sep 11 15:50:24 server kernel: [19600.214690] aMANGLE:FORWARD: IN=eth0
OUT=eth1 SRC=8.8.8.8 DST=192.168.1.236 LEN=84 TOS=0x00 PREC=0x00
TTL=53 ID=59275 PROTO=ICMP TYPE=0 CODE=0 ID=16944 SEQ=0
Sep 11 15:50:24 server kernel: [19600.214696] zMANGLE:FORWARD: IN=eth0
OUT=eth1 SRC=8.8.8.8 DST=192.168.1.236 LEN=84 TOS=0x00 PREC=0x00
TTL=53 ID=59275 PROTO=ICMP TYPE=0 CODE=0 ID=16944 SEQ=0
Sep 11 15:50:24 server kernel: [19600.214702] aFILTER:FORWARD: IN=eth0
OUT=eth1 SRC=8.8.8.8 DST=192.168.1.236 LEN=84 TOS=0x00 PREC=0x00
TTL=53 ID=59275 PROTO=ICMP TYPE=0 CODE=0 ID=16944 SEQ=0
Sep 11 15:50:24 server kernel: [19600.214709] zFILTER:FORWARD: IN=eth0
OUT=eth1 SRC=8.8.8.8 DST=192.168.1.236 LEN=84 TOS=0x00 PREC=0x00
TTL=53 ID=59275 PROTO=ICMP TYPE=0 CODE=0 ID=16944 SEQ=0
Sep 11 15:50:24 server kernel: [19600.214715] aMANGLE:POSTROUTING: IN=
OUT=eth1 SRC=8.8.8.8 DST=192.168.1.236 LEN=84 TOS=0x00 PREC=0x00
TTL=53 ID=59275 PROTO=ICMP TYPE=0 CODE=0 ID=16944 SEQ=0
Sep 11 15:50:24 server kernel: [19600.214721] zMANGLE:POSTROUTING: IN=
OUT=eth1 SRC=8.8.8.8 DST=192.168.1.236 LEN=84 TOS=0x00 PREC=0x00
TTL=53 ID=59275 PROTO=ICMP TYPE=0 CODE=0 ID=16944 SEQ=0
SUCCESS: Ping from client (WITHOUT marking)
Sep 10 15:44:21 server kernel: [11601.127159] aMANGLE:PREROUTING:
IN=eth1 OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00
SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64
ID=31275 PROTO=ICMP TYPE=8 CODE=0 ID=3358 SEQ=0
Sep 10 15:44:21 server kernel: [11601.127173] zMANGLE:PREROUTING:
IN=eth1 OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00
SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00 TTL=64
ID=31275 PROTO=ICMP TYPE=8 CODE=0 ID=3358 SEQ=0
Sep 10 15:44:21 server kernel: [11601.127185] aNAT:PREROUTING: IN=eth1
OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00 SRC=192.168.1.236
DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00 TTL=64 ID=31275 PROTO=ICMP
TYPE=8 CODE=0 ID=3358 SEQ=0
Sep 10 15:44:21 server kernel: [11601.127194] zNAT:PREROUTING: IN=eth1
OUT= MAC=f4:6d:04:9a:15:2d:00:1f:f3:d5:2d:78:08:00 SRC=192.168.1.236
DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00 TTL=64 ID=31275 PROTO=ICMP
TYPE=8 CODE=0 ID=3358 SEQ=0
Sep 10 15:44:21 server kernel: [11601.127207] aMANGLE:FORWARD: IN=eth1
OUT=eth0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00
TTL=63 ID=31275 PROTO=ICMP TYPE=8 CODE=0 ID=3358 SEQ=0
Sep 10 15:44:21 server kernel: [11601.127213] zMANGLE:FORWARD: IN=eth1
OUT=eth0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00
TTL=63 ID=31275 PROTO=ICMP TYPE=8 CODE=0 ID=3358 SEQ=0
Sep 10 15:44:21 server kernel: [11601.127220] aFILTER:FORWARD: IN=eth1
OUT=eth0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00
TTL=63 ID=31275 PROTO=ICMP TYPE=8 CODE=0 ID=3358 SEQ=0
Sep 10 15:44:21 server kernel: [11601.127226] zFILTER:FORWARD: IN=eth1
OUT=eth0 SRC=192.168.1.236 DST=8.8.8.8 LEN=84 TOS=0x10 PREC=0x00
TTL=63 ID=31275 PROTO=ICMP TYPE=8 CODE=0 ID=3358 SEQ=0
Sep 10 15:44:21 server kernel: [11601.169794] aMANGLE:FORWARD: IN=eth0
OUT=eth1 SRC=8.8.8.8 DST=192.168.1.236 LEN=84 TOS=0x00 PREC=0x00
TTL=53 ID=37282 PROTO=ICMP TYPE=0 CODE=0 ID=3358 SEQ=0
Sep 10 15:44:21 server kernel: [11601.169804] zMANGLE:FORWARD: IN=eth0
OUT=eth1 SRC=8.8.8.8 DST=192.168.1.236 LEN=84 TOS=0x00 PREC=0x00
TTL=53 ID=37282 PROTO=ICMP TYPE=0 CODE=0 ID=3358 SEQ=0
Sep 10 15:44:21 server kernel: [11601.169811] aFILTER:FORWARD: IN=eth0
OUT=eth1 SRC=8.8.8.8 DST=192.168.1.236 LEN=84 TOS=0x00 PREC=0x00
TTL=53 ID=37282 PROTO=ICMP TYPE=0 CODE=0 ID=3358 SEQ=0
Sep 10 15:44:21 server kernel: [11601.169818] zFILTER:FORWARD: IN=eth0
OUT=eth1 SRC=8.8.8.8 DST=192.168.1.236 LEN=84 TOS=0x00 PREC=0x00
TTL=53 ID=37282 PROTO=ICMP TYPE=0 CODE=0 ID=3358 SEQ=0
Whenever marking is enabled for anything other than the default ISP
and the packet originates from a internal computer, the packets get
out and upon return they seem to get dropped at the router after the
mangle prerouting chain.
I know it has to be something simple, but I'm all out of ideas at this
point, especially since the comparison between the two servers is a
match as far as I can tell. You can see the same output from the old
server and new server for a comparison here:
http://pastebin.com/EvmzfCe1
http://pastebin.com/xNSt60D9
Any help would be greatly appreciated.
--
Mike
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
