Hello, LARTC mailing readers, I hope u can help with this mysterious issue i'm having with my linux box acting as a router.
Scenario:
Linux running 2.6.8.1 /w julians patches /w support for multipath routing Latest iproute (iproute2-ss040702) 4 NICS
----------------- | x eth0 (63.43.x.x) network mask (255.255.240.0) | | x eth1 (63.43.x.x) network mask (255.255.240.0) | | Linux x eth2 (63.43.x.x) network mask (255.255.240.0) | | x eth3 (172.31.0.6) LAN of routers | (172.31.0.1, 172.31.0.2, 172.31.0.3) | network mask (255.255.255.248) x eth4 (172.31.0.254) LAN of users | network mask (255.255.255.0) | -----------------
FIRST OF ALL:
I know it has not a sane routing table in relation with eth3 and eth4, but it work because in the main table first appears eth3 entry, and the range of the lan is (172.31.0.8 - 172.31.0.254).
As you can deduce I have the same provider for eth0, eth1, eth2, it's a cable operator.
Im using iptables / ip for doing port based routing. So I can separate well known services in my enviroment.
This is basically my running config:
xerver:~# ip ru ls 0: from all lookup local 32758: from all fwmark 0x8 lookup balanceo2 32759: from all fwmark 0x7 lookup cablemodem3 32760: from all fwmark 0x5 lookup cablemodem 32761: from all fwmark 0x6 lookup cablemodem2 32762: from all fwmark 0x4 lookup balanceo 32763: from all fwmark 0x3 lookup router3 32764: from all fwmark 0x2 lookup router2 32765: from all fwmark 0x1 lookup router1 32766: from all lookup main 32767: from all lookup default
xerver:~# cat /etc/iproute2/rt_tables 255 local 254 main 253 default 0 unspec 200 balanceo2 201 balanceo 202 cablemodem 204 router1 205 router2 206 router3 207 cablemodem2 208 cablemodem3
Routes declarations:
ip route add default via 172.31.0.1 dev eth3 src 172.31.0.6 table router1
ip route add default via 172.31.0.2 dev eth3 src 172.31.0.6 table router2
ip route add default via 172.31.0.3 dev eth3 src 172.31.0.6 table router3
ip route add default proto static via 63.43.1.1 dev eth0 src $IP1 table cablemodem
ip route add default proto static via 63.43.1.1 dev eth1 src $IP2 table cablemodem2
ìp route add default proto static via 63.43.1.1 dev eth2 src $IP3 table cablemodem3
ip route add table balanceo2 default proto static \
nexthop via $GW dev eth0 weight 1 \
nexthop via $GW2 dev eth1 weight 1 \
nexthop via $GW3 dev eth2 weight 1(also tested with eql modifier, but no success) I've read eql only load-balances per packet basis, on my network, that's counterproducent. ?¿ (I need per connection basis load-balance)
xerver:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
172.31.0.0 0.0.0.0 255.255.255.248 U 0 0 0 eth3
172.31.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth4
63.43.x.x 0.0.0.0 255.255.224.0 U 0 0 0 eth0
63.43.x.x 0.0.0.0 255.255.224.0 U 0 0 0 eth1
63.43.x.x 0.0.0.0 255.255.224.0 U 0 0 0 eth2
0.0.0.0 63.43.1.1 0.0.0.0 UG 0 0 0 eth0
Ok, every cablemodem has the same gateway, they all are in the same subnet. So eth0 has gw 63.43.1.1, eth1 has gw 63.43.1.1 and eth2 has gw 63.43.1.1.
Those three lines are 1Mbit Down 512 Up
Ok, when I mark my packets to follow a specific route through a specific gw, it works great. I have to say I had some trouble with arp flux problem, but, I suppose if I've been able to make the three links download at the same time it's correct to suppose that the arp flux is corrected.
Tkweaks to /proc
echo "0" > /proc/sys/net/ipv4/conf/all/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth0/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth1/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth2/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth3/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth4/rp_filter
echo "1" > /proc/sys/net/ipv4/conf/eth0/hidden echo "1" > /proc/sys/net/ipv4/conf/eth1/hidden echo "1" > /proc/sys/net/ipv4/conf/eth2/hidden echo "1" > /proc/sys/net/ipv4/conf/eth3/hidden
echo "1" > /proc/sys/net/ipv4/conf/eth0/arp_filter echo "1" > /proc/sys/net/ipv4/conf/eth1/arp_filter echo "1" > /proc/sys/net/ipv4/conf/eth2/arp_filter echo "1" > /proc/sys/net/ipv4/conf/eth3/arp_filter
My troubles comes with load balance outgoing connections, I use the table balanceo2 for this purpose, and looks like this:
xerver:~#ip ro sh table balanceo2
default proto static
nexthop via 62.43.0.1 dev eth0 weight 1
nexthop via 62.43.0.1 dev eth1 weight 1
nexthop via 62.43.0.1 dev eth2 weight 1But when I mark www traffic like this:
iptables -A PREROUTING -t mangle -i eth4 -p tcp --dport 80 -j MARK --set-mark 8
Everything breaks down, no www traffic is possible for the clients.
I've tried the same with ICMP echo requests and it seemed to work.
I've tkweaked a bit /proc for route cache times expiration but no success.
I don't know what I'm missing.
NOTE: Some time ago, my ISP had another network structure, and sometimes, I got different gw in each cablemodem, and load balance worked fine.
I think the problem comes in the routing cache, or expiration times, really I don't know.
I would appreciate any help.
Thanks in advance.
Xisco Fernandez. _______________________________________________ LARTC mailing list / LARTC@xxxxxxxxxxxxxxx http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
