Hy Fatih thanks for helping me Just to remember that I m using debian with kernel 3.16.0-4 I clean all my iptables rules and route, set all parameters at your script as you adviced. Reboot and call you script. Unfortunately I still see packages from LAN leaving WAN1 and WAN2, but GOOD NEWS!!! I haven't seen packages from ipwan1 leaving wan2 and vice versa....80) Humm could be something related with sysctl.conf???? best regards 2017-11-08 19:12 GMT-02:00 Fatih USTA <fatihusta86@xxxxxxxxx>: > Hi Paulo > I tried my scripts today. It's work. > Please run My script. I think very easy configure. > > Don't use weighted multiple nexthop for default gw. > Please use dummy interface for default gateway. > > Note1: use 1-1 weight on statistics module during test. > > Note2: My script have failover feature. (link and gw detection.) > > > On Nov 8, 2017 20:58, "paulo bruck" <paulobruck1@xxxxxxxxx> wrote: >> >> Hi Mark >> >> Yes but using CONNMARK and MARK I still see packages from lan leaving >> WAN and packages that should leave WAN1 with address of WAN2.... >> >> Certanly I am still doing something wrong,...80( >> >> I tried to mark ALL packages at prerouting mangle into enp2so ( lan) >> with mark=1 but even this not worked.... >> >> Comparing with scripts that Fatih developed I see now minor >> differences ( one of then is route default ) Maybe the problem is >> there? >> >> root@zeus:~# ip r l >> default >> nexthop via 192.168.1.1 dev enp7s0 weight 99 >> nexthop via 201.6.110.1 dev enp6s1 weight 1 >> 192.168.1.0/24 dev enp7s0 proto kernel scope link src 192.168.1.2 >> 192.168.10.0/24 dev enp2s0 proto kernel scope link src 192.168.10.1 >> 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 >> 201.6.110.0/24 dev enp6s1 proto kernel scope link src 201.6.110.xxx >> >> root@zeus:~# ip r l table WAN1 >> default via 192.168.1.1 dev enp7s0 >> 127.0.0.0/8 dev lo scope link >> 192.168.1.0/24 dev enp7s0 proto kernel scope link src 192.168.1.2 >> 192.168.10.0/24 dev enp2s0 proto kernel scope link src 192.168.10.1 >> 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 >> 201.6.110.0/24 dev enp6s1 proto kernel scope link src 201.6.110.xxx >> >> root@zeus:~# ip r l table WAN2 >> default via 201.6.110.1 dev enp6s1 >> 127.0.0.0/8 dev lo scope link >> 192.168.1.0/24 dev enp7s0 proto kernel scope link src 192.168.1.2 >> 192.168.10.0/24 dev enp2s0 proto kernel scope link src 192.168.10.1 >> 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 >> 201.6.110.0/24 dev enp6s1 proto kernel scope link src 201.6.110.xxx >> >> root@zeus:~# ip rule list >> 0: from all lookup local >> 32762: from 201.6.110.0/24 lookup WAN2 >> 32763: from all fwmark 0x2 lookup WAN2 >> 32764: from 192.168.1.0/24 lookup WAN1 >> 32765: from all fwmark 0x1 lookup WAN1 >> 32766: from all lookup main >> 32767: from all lookup default >> >> and iptables mangle >> Chain PREROUTING (policy ACCEPT 2911 packets, 570078 bytes) >> pkts bytes target prot opt in out source >> destination >> 99 4574 CONNMARK all -- virbr0 * 0.0.0.0/0 >> 0.0.0.0/0 ctstate RELATED,ESTABLISHED CONNMARK restore >> 0 0 MARK tcp -- virbr0 * 192.168.122.2 >> 0.0.0.0/0 multiport dports 25 /* mercurio#5 */ ctstate >> NEW MARK set 0x2 >> 99 4574 CONNMARK all -- virbr0 * 0.0.0.0/0 >> 0.0.0.0/0 CONNMARK save >> 373 113087 CONNMARK all -- enp2s0 * 0.0.0.0/0 >> 0.0.0.0/0 ctstate RELATED,ESTABLISHED CONNMARK restore >> 2 183 MARK udp -- enp2s0 * 0.0.0.0/0 >> 0.0.0.0/0 multiport dports 15000:16000 /* >> saida_do_80_e_443_preferencial_pelo_SPeedy */ ctstate NEW MARK set 0x1 >> 4 240 MARK tcp -- enp2s0 * 0.0.0.0/0 >> 0.0.0.0/0 multiport dports 80,443,8080,873,15000:16000 >> /* saida_do_80_e_443_preferencial_pelo_SPeedy */ ctstate NEW MARK set >> 0x1 >> 469 119098 CONNMARK all -- enp2s0 * 0.0.0.0/0 >> 0.0.0.0/0 CONNMARK save >> 373 113087 CONNMARK all -- enp2s0 * 0.0.0.0/0 >> 0.0.0.0/0 ctstate RELATED,ESTABLISHED CONNMARK restore >> 96 6011 CONNMARK1 all -- enp2s0 * 0.0.0.0/0 >> 0.0.0.0/0 ctstate NEW statistic mode random probability >> 0.99000000022 >> 1 60 CONNMARK2 all -- enp2s0 * 0.0.0.0/0 >> 0.0.0.0/0 ctstate NEW statistic mode random probability >> 0.00999999978 >> 99 4574 CONNMARK all -- virbr0 * 0.0.0.0/0 >> 0.0.0.0/0 ctstate RELATED,ESTABLISHED CONNMARK restore >> 0 0 CONNMARK1 all -- virbr0 * 0.0.0.0/0 >> 0.0.0.0/0 ctstate NEW statistic mode random probability >> 0.99000000022 >> 0 0 CONNMARK2 all -- virbr0 * 0.0.0.0/0 >> 0.0.0.0/0 ctstate NEW statistic mode random probability >> 0.00999999978 >> 675 145108 CONNMARK all -- enp7s0 * 0.0.0.0/0 >> 0.0.0.0/0 ctstate RELATED,ESTABLISHED CONNMARK restore >> 141 9097 MARK all -- enp7s0 * 0.0.0.0/0 >> 0.0.0.0/0 ctstate NEW MARK set 0x1 >> 816 154205 CONNMARK all -- enp7s0 * 0.0.0.0/0 >> 0.0.0.0/0 CONNMARK save >> 2 112 CONNMARK all -- enp6s1 * 0.0.0.0/0 >> 0.0.0.0/0 ctstate RELATED,ESTABLISHED CONNMARK restore >> 8 400 MARK all -- enp6s1 * 0.0.0.0/0 >> 0.0.0.0/0 ctstate NEW MARK set 0x2 >> 10 512 CONNMARK all -- enp6s1 * 0.0.0.0/0 >> 0.0.0.0/0 CONNMARK save >> ---cut---------- >> Chain CONNMARK1 (2 references) >> pkts bytes target prot opt in out source >> destination >> 5118 1549128 MARK all -- * * 0.0.0.0/0 >> 0.0.0.0/0 MARK set 0x1 >> 5118 1549128 CONNMARK all -- * * 0.0.0.0/0 >> 0.0.0.0/0 CONNMARK save >> >> Chain CONNMARK2 (2 references) >> pkts bytes target prot opt in out source >> destination >> 4 242 MARK all -- * * 0.0.0.0/0 >> 0.0.0.0/0 MARK set 0x2 >> 4 242 CONNMARK all -- * * 0.0.0.0/0 >> 0.0.0.0/0 CONNMARK save >> >> >> NAT >> Chain POSTROUTING (policy ACCEPT 2452 packets, 118448 bytes) >> pkts bytes target prot opt in out source >> destination >> 2597 163863 SNAT all -- * enp7s0 0.0.0.0/0 >> 0.0.0.0/0 /* saida padrão do POSTROUTING da 192.168.1.2 >> */ to:192.168.1.2 >> 43 3138 SNAT all -- * enp6s1 0.0.0.0/0 >> 0.0.0.0/0 /* saida padrão do POSTROUTING da >> 201.6.110.xxx */ to:201.6.110.xxx >> >> >> >> root@zeus:~# tcpdump -nlptqi enp7s0 host ! 192.168.1.2 >> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode >> listening on enp7s0, link-type EN10MB (Ethernet), capture size 262144 >> bytes >> IP 192.168.10.21.52207 > 51.15.4.13.80: tcp 0 >> IP 201.6.110.xxx.57096 > 124.150.9.21.51568: tcp 0 >> >> root@zeus:~# conntrack -L --dst=51.15.4.13 >> tcp 6 6 CLOSE_WAIT src=192.168.10.21 dst=51.15.4.13 sport=42080 >> dport=1337 src=51.15.4.13 dst=192.168.1.2 sport=1337 dport=42080 >> [ASSURED] mark=1 use=1 >> tcp 6 10 CLOSE_WAIT src=192.168.10.21 dst=51.15.4.13 sport=52404 >> dport=80 src=51.15.4.13 dst=192.168.1.2 sport=80 dport=52404 [ASSURED] >> mark=1 use=1 >> Strange isnt it? packages are marked but leaving wihout NAT >> >> root@zeus:~# conntrack -L --dst=124.150.9.21 >> udp 17 125 src=192.168.10.21 dst=124.150.9.21 sport=51413 >> dport=51568 src=124.150.9.21 dst=192.168.1.2 sport=51568 dport=51413 >> [ASSURED] mark=1 use=1 >> This is real strange package with ource IP from WAN2 leaning WAN1 and >> marked as WAN1..... >> >> Any ideas would be very appreciated... 80) >> >> 2017-11-08 9:01 GMT-02:00 Mark Coetser <mark@xxxxxxxxxxxx>: >> > >> > On 08/11/2017 01:58, paulo bruck wrote: >> >> >> >> Hi Fatih >> >> >> >> Thanks for quickly response. >> >> >> >> Unfortunately using your scripts is the same. I see packages leaving >> >> wan interface w/ lan address and packages leaving WAN1 w/ WAN2 ip... >> >> 80(( >> >> >> >> I have read about routing cache not exist after kernel 3....8( but >> >> I not found any replacement for it untill now. >> >> >> >> Tested in Debian stretch using kernel 3.16 and 4.9.51-1 >> >> >> >> I think that with your ideia I have reach 1/2 of path..... >> >> >> >> Any other idea ? >> >> >> >> Thanks >> > >> > >> > You are correct, the route cache is no longer available. you need to use >> > iptables with conntrack and statistic to balance the connections across >> > each >> > link. >> > >> > >> > -- >> > Thank you, >> > >> > Mark Adrian Coetser >> >> >> >> -- >> Paulo Ricardo Bruck consultor >> tel 011 3596-4881/4882 011 98140-9184 (TIM) >> http://www.contatogs.com.br >> http://www.protejasuarede.com.br >> gpg AAA59989 at wwwkeys.us.pgp.net >> -- >> To unsubscribe from this list: send the line "unsubscribe netfilter" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- Paulo Ricardo Bruck consultor tel 011 3596-4881/4882 011 98140-9184 (TIM) http://www.contatogs.com.br http://www.protejasuarede.com.br gpg AAA59989 at wwwkeys.us.pgp.net -- To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
