Hi, I have a suspicious problem with multiple uplinks configuration.
First of all my configuration:
1) kernel 2.6.20.3
2) iptables 1.3.7
3) last iproute (for masked marks)
All wan interfaces are bridged (stp disabled) in only one interface
(wan0), all lan interfaces are bridged (stp enabled) in only one interface
(zlan0).
The wan0 bridge is to allow UPnP works.
To allow related incoming traffic from one fisical interface I mark
connections, and the same to allow outgoing related.
The routing rules are the same than lartc documentation plus a rule by
interface to allow the routing using marks (masked).
The comands I use are:
==BEGIN==
/sbin/ip rule del prio 50 table main
/sbin/ip rule del prio 100 fwmark 0x8000/0xf000 table 150
/sbin/ip rule del prio 150 from 217.125.139.204/26 table 150
/sbin/ip rule del prio 101 fwmark 0x4000/0xf000 table 151
/sbin/ip rule del prio 151 from 80.32.61.58/24 table 151
/sbin/ip rule del prio 200 table 200
/sbin/ip route flush table 150
/sbin/ip route flush table 151
/sbin/ip route flush table 200
/sbin/iptables -t mangle -D PREROUTING -j MARCAR_IFACE
/sbin/iptables -t mangle -F MARCAR_IFACE
/sbin/iptables -t mangle -X MARCAR_IFACE
/sbin/iptables -t mangle -F MARCAR_IFACE_TRAFICO
/sbin/iptables -t mangle -X MARCAR_IFACE_TRAFICO
/sbin/iptables -t mangle -D POSTROUTING -j MARCAR_IFACE_OUT
/sbin/iptables -t mangle -F MARCAR_IFACE_OUT
/sbin/iptables -t mangle -X MARCAR_IFACE_OUT
/sbin/iptables -t mangle -N MARCAR_IFACE
/sbin/iptables -t mangle -N MARCAR_IFACE_TRAFICO
/sbin/iptables -t mangle -A MARCAR_IFACE -j CONNMARK --restore-mark
/sbin/iptables -t mangle -A MARCAR_IFACE -m mark ! --mark 0x0000/0xf000 -j
RETURN
/sbin/iptables -t mangle -A MARCAR_IFACE -m mark --mark 0x0000/0xf000 -j
MARCAR_IFACE_TRAFICO
/sbin/iptables -t mangle -N MARCAR_IFACE_OUT
/sbin/iptables -t mangle -A MARCAR_IFACE_OUT -j CONNMARK --restore-mark
/sbin/iptables -t mangle -A MARCAR_IFACE_OUT -m mark ! --mark
0x0000/0xf000 -j RETURN
/sbin/iptables -t mangle -A MARCAR_IFACE -m mark --mark 0x0000/0xf000 -i
wan0 -m physdev --physdev-in eth1 -m state --state NEW -j MARK --or-mark
0x8000
/sbin/iptables -t mangle -A MARCAR_IFACE_OUT -m mark --mark 0x0000/0xf000
-o wan0 -m conntrack --ctreplsrc 217.125.139.204 -j MARK --or-mark 0x8000
/sbin/iptables -t mangle -A MARCAR_IFACE_OUT -m mark --mark 0x0000/0xf000
-o wan0 -m conntrack --ctrepldst 217.125.139.204 -j MARK --or-mark 0x8000
/sbin/iptables -t mangle -A MARCAR_IFACE_OUT -m mark --mark 0x0000/0xf000
-o wan0 -s 217.125.139.204 -j MARK --or-mark 0x8000
/sbin/iptables -t mangle -A MARCAR_IFACE_OUT -m mark --mark 0x0000/0xf000
-o wan0 -m conntrack --ctorigsrc 217.125.139.204 -j MARK --or-mark 0x8000
/sbin/iptables -t mangle -A MARCAR_IFACE_OUT -m mark --mark 0x0000/0xf000
-o wan0 -m conntrack --ctorigdst 217.125.139.204 -j MARK --or-mark 0x8000
/sbin/iptables -t mangle -A MARCAR_IFACE -m mark --mark 0x0000/0xf000 -i
wan0 -m physdev --physdev-in eth3 -m state --state NEW -j MARK --or-mark
0x4000
/sbin/iptables -t mangle -A MARCAR_IFACE_OUT -m mark --mark 0x0000/0xf000
-o wan0 -m conntrack --ctreplsrc 80.32.61.58 -j MARK --or-mark 0x4000
/sbin/iptables -t mangle -A MARCAR_IFACE_OUT -m mark --mark 0x0000/0xf000
-o wan0 -m conntrack --ctrepldst 80.32.61.58 -j MARK --or-mark 0x4000
/sbin/iptables -t mangle -A MARCAR_IFACE_OUT -m mark --mark 0x0000/0xf000
-o wan0 -s 80.32.61.58 -j MARK --or-mark 0x4000
/sbin/iptables -t mangle -A MARCAR_IFACE_OUT -m mark --mark 0x0000/0xf000
-o wan0 -m conntrack --ctorigsrc 80.32.61.58 -j MARK --or-mark 0x4000
/sbin/iptables -t mangle -A MARCAR_IFACE_OUT -m mark --mark 0x0000/0xf000
-o wan0 -m conntrack --ctorigdst 80.32.61.58 -j MARK --or-mark 0x4000
/sbin/iptables -t mangle -A MARCAR_IFACE -j CONNMARK --save-mark
/sbin/iptables -t mangle -A MARCAR_IFACE -j RETURN
/sbin/iptables -t mangle -I PREROUTING -j MARCAR_IFACE
/sbin/iptables -t mangle -A MARCAR_IFACE_OUT -j CONNMARK --save-mark
/sbin/iptables -t mangle -A MARCAR_IFACE_OUT -j RETURN
/sbin/iptables -t mangle -I POSTROUTING -j MARCAR_IFACE_OUT
/sbin/ip rule add prio 50 table main
/sbin/ip rule add prio 100 fwmark 0x8000/0xf000 table 150
/sbin/ip rule add prio 150 from 217.125.139.204/26 table 150
/sbin/ip route add default via 217.125.139.193 dev wan0 src
217.125.139.204 proto static table 150
/sbin/ip route append prohibit default table 150 metric 1 proto static
/sbin/ip rule add prio 101 fwmark 0x4000/0xf000 table 151
/sbin/ip rule add prio 151 from 80.32.61.58/24 table 151
/sbin/ip route add default via 80.32.61.1 dev wan0 src 80.32.61.58 proto
static table 151
/sbin/ip route append prohibit default table 151 metric 1 proto static
/sbin/ip rule add prio 200 table 200
/sbin/ip route add default table 200 proto static nexthop via
217.125.139.193 dev wan0 weight 1 nexthop via 80.32.61.1 dev wan0 weight 1
/sbin/ip route flush cache
==END==
I have this "output" for all chains and routes:
==BEGIN==
=== REGLAS IPTABLES PARA EL ENRUTADO ===
Chain PREROUTING (policy ACCEPT 8664K packets, 5097M bytes)
num pkts bytes target prot opt in out source
destination
1 3348K 1832M MARCAR_IFACE 0 -- * * 0.0.0.0/0
0.0.0.0/0
Chain MARCAR_IFACE (1 references)
num pkts bytes target prot opt in out source
destination
1 3348K 1832M CONNMARK 0 -- * * 0.0.0.0/0
0.0.0.0/0 CONNMARK restore
2 2841K 1653M RETURN 0 -- * * 0.0.0.0/0
0.0.0.0/0 MARK match !0x0/0xf000
3 507K 179M MARCAR_IFACE_TRAFICO 0 -- * * 0.0.0.0/0
0.0.0.0/0 MARK match 0x0/0xf000
4 40690 2721K MARK 0 -- wan0 * 0.0.0.0/0
0.0.0.0/0 MARK match 0x0/0xf000 PHYSDEV match --physdev-in eth1
state NEW MARK or 0x8000
5 48680 3062K MARK 0 -- wan0 * 0.0.0.0/0
0.0.0.0/0 MARK match 0x0/0xf000 PHYSDEV match --physdev-in eth3
state NEW MARK or 0x4000
6 507K 179M CONNMARK 0 -- * * 0.0.0.0/0
0.0.0.0/0 CONNMARK save
7 507K 179M RETURN 0 -- * * 0.0.0.0/0
0.0.0.0/0
Chain MARCAR_IFACE_TRAFICO (1 references)
num pkts bytes target prot opt in out source
destination
Chain POSTROUTING (policy ACCEPT 16M packets, 8665M bytes)
num pkts bytes target prot opt in out source
destination
1 6483K 3397M MARCAR_IFACE_OUT 0 -- * * 0.0.0.0/0
0.0.0.0/0
Chain MARCAR_IFACE_OUT (1 references)
num pkts bytes target prot opt in out source
destination
1 6483K 3397M CONNMARK 0 -- * * 0.0.0.0/0
0.0.0.0/0 CONNMARK restore
2 5781K 2966M RETURN 0 -- * * 0.0.0.0/0
0.0.0.0/0 MARK match !0x0/0xf000
3 0 0 MARK 0 -- * wan0 0.0.0.0/0
0.0.0.0/0 MARK match 0x0/0xf000 ctreplsrc 217.125.139.204 MARK
or 0x8000
4 104K 7470K MARK 0 -- * wan0 0.0.0.0/0
0.0.0.0/0 MARK match 0x0/0xf000 ctrepldst 217.125.139.204 MARK
or 0x8000
5 135 7091 MARK 0 -- * wan0 217.125.139.204
0.0.0.0/0 MARK match 0x0/0xf000 MARK or 0x8000
6 0 0 MARK 0 -- * wan0 0.0.0.0/0
0.0.0.0/0 MARK match 0x0/0xf000 ctorigsrc 217.125.139.204 MARK
or 0x8000
7 0 0 MARK 0 -- * wan0 0.0.0.0/0
0.0.0.0/0 MARK match 0x0/0xf000 ctorigdst 217.125.139.204 MARK
or 0x8000
8 0 0 MARK 0 -- * wan0 0.0.0.0/0
0.0.0.0/0 MARK match 0x0/0xf000 ctreplsrc 80.32.61.58 MARK or
0x4000
9 101K 7298K MARK 0 -- * wan0 0.0.0.0/0
0.0.0.0/0 MARK match 0x0/0xf000 ctrepldst 80.32.61.58 MARK or
0x4000
10 175 7578 MARK 0 -- * wan0 80.32.61.58
0.0.0.0/0 MARK match 0x0/0xf000 MARK or 0x4000
11 0 0 MARK 0 -- * wan0 0.0.0.0/0
0.0.0.0/0 MARK match 0x0/0xf000 ctorigsrc 80.32.61.58 MARK or
0x4000
12 1 48 MARK 0 -- * wan0 0.0.0.0/0
0.0.0.0/0 MARK match 0x0/0xf000 ctorigdst 80.32.61.58 MARK or
0x4000
13 702K 431M CONNMARK 0 -- * * 0.0.0.0/0
0.0.0.0/0 CONNMARK save
14 702K 431M RETURN 0 -- * * 0.0.0.0/0
0.0.0.0/0
=== REGLAS DE ENRUTAMIENTO ===
0: from all lookup local
50: from all lookup main
100: from all fwmark 0x8000/0xf000 lookup uno
101: from all fwmark 0x4000/0xf000 lookup dos
150: from 217.125.139.204/26 lookup uno
151: from 80.32.61.58/24 lookup dos
200: from all lookup defecto
32766: from all lookup main
32767: from all lookup default
=== TABLAS DE RUTAS ===
=== MAIN ===
217.125.139.192/26 dev wan0 proto kernel scope link src 217.125.139.204
80.32.61.0/24 dev wan0 proto kernel scope link src 80.32.61.58
192.168.3.0/24 dev zlan0 proto kernel scope link src 192.168.3.247
192.168.2.0/24 dev zlan0 proto kernel scope link src 192.168.2.247
192.168.1.0/24 dev zlan0 proto kernel scope link src 192.168.1.247
10.1.1.0/24 dev zlan0 proto kernel scope link src 10.1.1.6
169.254.0.0/16 dev zlan0 scope link
239.0.0.0/8 dev zlan0 scope link
=== wan0 TABLA 150 ===
default via 217.125.139.193 dev wan0 proto static src 217.125.139.204
prohibit default proto static metric 1
=== wan0 TABLA 151 ===
default via 80.32.61.1 dev wan0 proto static src 80.32.61.58
prohibit default proto static metric 1
=== TABLA 200 (defecto) ===
default proto static
nexthop via 217.125.139.193 dev wan0 weight 1
nexthop via 80.32.61.1 dev wan0 weight 1
==END==
The -t nat POSTROUTING rules:
==BEGIN==Chain POSTROUTING (policy ACCEPT 58524 packets, 42M bytes)
pkts bytes target prot opt in out source
destination
0 0 SNAT 0 -- * eth3 10.1.1.0/24
0.0.0.0/0 to:80.32.61.58
0 0 SNAT 0 -- * eth1 10.1.1.0/24
0.0.0.0/0 to:217.125.139.204
0 0 SNAT 0 -- * wan0 10.1.1.0/24
0.0.0.0/0 PHYSDEV match --physdev-out eth3 to:80.32.61.58
0 0 SNAT 0 -- * wan0 10.1.1.0/24
0.0.0.0/0 PHYSDEV match --physdev-out eth1
to:217.125.139.204
0 0 SNAT 0 -- * eth3 10.1.1.0/24
0.0.0.0/0 to:80.32.61.58
0 0 SNAT 0 -- * eth1 10.1.1.0/24
0.0.0.0/0 to:217.125.139.204
578K 39M MASQUERADE 0 -- * wan0 10.1.1.0/24
0.0.0.0/0
0 0 MASQUERADE 0 -- * wan0:1 10.1.1.0/24
0.0.0.0/0
0 0 SNAT 0 -- * wan0 10.1.1.0/24
0.0.0.0/0 to:80.32.61.58
0 0 SNAT 0 -- * wan0 10.1.1.0/24
0.0.0.0/0 to:217.125.139.204
==END==
The problems I have are:
1) If I make ssh conections from internet to the router (not to any pc
into the lan zone), sometimes the ssh sesions disconnect.
2) If I run tcpdump as these:
tcpdump -n -i eth3 not host 80.32.61.58
tcpdump -n -i eth1 not host 217.125.139.204
I can see :
a) IP frames not nated, where the source address is from lan zone.
b) Source IPs are not the correct.
With tcpdump command I expect don't see anything, instead I can see
frames as described below.
Because the wan interface is only 1 (with 2 ip's), I only can use "-j
MASQUERADE" for the nating, I can't use -m physdev --physdev-out,
netfilter layer appears don't know what is the real outgoing interface in
the bridge wan0 and "wan0:1" is not handled by netfilter layer.
The questions:
1) Does anyone know if this is a known issue (the tcpdump output and
physdev issue)?
2) Does anyone know how to use SNAT in this case (I cant use -j SNAT)?
3) With 2.6.19.7 I were using "-m physdev --physdev-out" into the chain
"MARCAR_IFACE_OUT", but with the 2.6.20.3 updated kernel, the -m
physdev appears to be broken and I then must use -m conntrack. Is this
a good solution?
Please, I need any help, with this configuration I discovered these
problems but I don't know how to solve them:
1) wan0 bridge don't appears to be working 100% of time (appears that
packets from one IP in the bridge are sent to the other interface).
2) NAT appears to be a bit confused and don't nat all packets,
MASQUERADE don't want to be working all time.
3) -m physdev --physdev-out don't know what is the read physical
interface where the packets a sent. (Whith 2.6.19.7 kernel, this
extension were working, or, at least, there were counters in the rules.
4) Conections from internet to the router machine are lost randomly.
I have no problem to use POSTROUTING chain in nat table to DROP o REJECT
incorrect packets, but ... really need I to do that?
Thanks!! All help are apretiated!!
Regards.
P.D.: Sorry, my english is a bit poor.
