[LARTC] Qeues seem to stop working after an hour, and it collapses, no idea why!

Linux Advanced Routing and Traffic Control

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hello, this is my first message to the board, so please excuse me if i am omiting any prestablished rule ;)

Let me explain you my problem..

I have 300 users, studens, wich automatically means Edonkey,KaZaa,and most of this peer to peer sharing programs.

I actually have 2 x 2Mbit ADSL's. My solution includes load balancing for those 2 inet access with a linux firewall separing both routers and my LAN.

Right know i have a stateful firewall doing connection tracking (required for load balancing) DROPING all connections but web/smtp/pop3/https/imap etc.... That works fine, but people want to use p2p programs. After doing my research, i found out about QoS and ingress/egress queues.

My idea was to priorize critical traffic like http/smtp/pop3/ssh/etc.., granting bandwith and leaving the rest to downloads (having these the posibility of ceiling bw as well).

I have two egress and two ingress queues attached to the ethernets connected to each of the ethernet connected to the routers.

Ingress is done with IMQ, and egress with HTB.

At first it seems to work really well, but after half an hour having it working, INTERACTIVE TRAFFIC COLLAPSES, and looks as if there is no queues, hence interactive traffic is no interactive anymore.. :)

I red somewhere that to decrease latency i had to set the txqueuelen from 100 to 30, but still collapses...

I have noticed using iptraf, that the number of packets arriving to the LAN-eth (eth3 in my little sketch) is about 30% bigger than the sum of eth0 and eth2, is that normal?

I attach the configuration files of both the firewall, and the queues...

And a little graphical description of my topology:
Obviously doing SNAT and LB :)
                 ______
<hub1> <-------> |Switch|            ______
                |      |           |      |  eth0
<hub2> <-------> |      |           |Linux | <----------> Router ADSL1
 .              |      |    eth3   |      |192.168.3.5   192.168.3.6
 .              |      |  <----->  | Box  |
 .              |      |192.169.1.1|      |  eth2
 .              |      |           |      | <----------> Router ADSL2
<hub24> <------> |______|           |______|192.168.4.2   192.168.4.1

I am really desperate, because it has been a month right now, and i am begging to suspect i am a bit of a fool ;) Thank you very much!!


_________________________________________________________________
Charla con tus amigos en línea mediante MSN Messenger: http://messenger.yupimsn.com/ 
#killall adslrunning
#/usr/local/bin/adslrunning &
#/usr/local/bin/adsl/shaper/1
#/usr/local/bin/adsl/shaper/2

ip rule del from 192.168.2.0/24

ip rule del prio 50
ip rule del from 192.168.3.0/24 prio 201
ip rule del from 192.168.4.0/24 prio 202
ip rule del prio 222
ip rule del from 192.168.2.252
ip rule del from 192.168.2.20
ip rule del from 192.168.0.0/24

ip route del default table main

ip rule add prio 50 table main
ip rule add prio 201 from 192.168.3.0/24 table 201
ip route add default via 192.168.3.3 dev eth2 src 192.168.3.5 proto static
table 201
ip route append prohibit default table 201 metric 1 proto static


ip rule add prio 202 from 192.168.4.0/24 table 202
ip route add default via 192.168.4.1 dev eth0 src 192.168.4.2 proto static
table 202
ip route append prohibit default table 202 metric 1 proto static

ip rule add prio 222 table 222
ip route add default table 222 proto static nexthop via 192.168.3.3 dev eth2
nexthop via 192.168.4.1 dev eth0

##################################################
## Firewall para CMU CHAMINADE
##
##

   VLAN0=192.168.0.0/255.255.255.0
   VLAN1=192.168.1.0/255.255.255.0
   VLAN2=192.168.2.0/255.255.255.0
   VLAN3=192.168.3.0/255.255.255.0
   VLAN4=192.168.4.0/255.255.255.0

##################################################
## Primero limpiamos todas las reglas
##
##
   /sbin/iptables -t filter -F
   /sbin/iptables -t nat -F
   /sbin/iptables -t mangle -F
   echo Limpiando reglas...

##################################################
## Aceptamos todas las conexiones## excepto las de nuestra red
##

/sbin/iptables -t filter -P FORWARD ACCEPT

##################################################
## SNAT
##


  /sbin/iptables -t nat -A POSTROUTING -o eth2 -s 192.168.1.0/255.255.0.0
-j SNAT --to 192.168.3.5
  /sbin/iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.0/255.255.0.0
-j SNAT --to 192.168.4.2


###################################################
## Forzamos a que los usuarios usen el proxy
##
##

#  /sbin/iptables -t nat -A PREROUTING -s $VLAN2 -i eth3 -p tcp --dport 80
-j REDIRECT --to-ports 8080
#  /sbin/iptables -t nat -A PREROUTING -s $VLAN0 -i eth3 -p tcp --dport 80
-j REDIRECT --to-ports 8080
#  /sbin/iptables -t nat -A PREROUTING -s $VLAN1 -i eth3 -p tcp --dport 80
-j REDIRECT --to-ports 8080

###################################################
## No permitimos que utilicen otro proxy
##
##

#   /sbin/iptables -A FORWARD -s $VLAN0 -d ! 192.168.1.1 -p tcp --dport 3128
-j DROP
#   /sbin/iptables -A FORWARD -s $VLAN0 -d ! 192.168.1.1 -p tcp --dport 8080
-j DROP

#   /sbin/iptables -A FORWARD -s $VLAN1 -d ! 192.168.1.1 -p tcp --dport 3128
-j DROP
#   /sbin/iptables -A FORWARD -s $VLAN1 -d ! 192.168.1.1 -p tcp --dport 8080
-j DROP

#   /sbin/iptables -A FORWARD -s $VLAN2 -d ! 192.168.1.1 -p tcp --dport 3128
-j DROP
#   /sbin/iptables -A FORWARD -s $VLAN2 -d ! 192.168.1.1 -p tcp --dport 8080
-j DROP



####################################################
##  Stateful Firewall
##
##
##

	iptables -t filter -N keep_state
	iptables -t filter -A keep_state -m state --state RELATED,ESTABLISHED -j
ACCEPT
	iptables -t filter -A keep_state -j RETURN

	iptables -t nat -N keep_state
	iptables -t nat -A keep_state -m state --state RELATED,ESTABLISHED -j
ACCEPT
	iptables -t nat -A keep_state -j RETURN

	iptables -t nat -A PREROUTING -j keep_state
	iptables -t nat -A POSTROUTING -j keep_state
	iptables -t nat -A OUTPUT -j keep_state

	iptables -t filter -A INPUT -j keep_state
	iptables -t filter -A OUTPUT -j keep_state
	iptables -t filter -A FORWARD -j keep_state


DEV=eth2
IDEV=imq1
IQDEV=1

tc qdisc add dev ${IDEV} handle 1: root htb default 1
tc class add dev ${IDEV} parent 1: classid 1:1 htb rate 1900kbit

tc qdisc add dev ${IDEV} parent 1:1 handle 10: htb default 2
tc class add dev ${IDEV} parent 10: classid 10:1 htb rate 1500kbit burst 6k
prio 1
tc class add dev ${IDEV} parent 10: classid 10:2 htb rate 512kbit ceil
1900kbit burst 6k prio 2


tc qdisc add dev ${IDEV} parent 10:1 handle 21:0 pfifo
tc qdisc add dev ${IDEV} parent 10:2 handle 22:0 sfq

tc filter add dev ${IDEV} protocol ip pref 1 parent 10: handle 1 fw classid
10:1
tc filter add dev ${IDEV} protocol ip pref 2 parent 10: handle 2 fw classid
10:2

#iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS
--clamp-mss-to-pmtu
#iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 128

iptables -t mangle -N IMQ${IQDEV}
iptables -t mangle -A IMQ${IQDEV} -p icmp -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 23 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 25 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 53 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 80 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 110 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 443 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 21 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 22 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 6667 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 20 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -j RETURN


iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ${IQDEV}
iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ --todev ${IQDEV}

ip link set ${IDEV} up


DEV=eth0
IDEV=imq0
IQDEV=0

tc qdisc add dev ${IDEV} handle 1: root htb default 1
tc class add dev ${IDEV} parent 1: classid 1:1 htb rate 1900kbit

tc qdisc add dev ${IDEV} parent 1:1 handle 10: htb default 2
tc class add dev ${IDEV} parent 10: classid 10:1 htb rate 1500kbit burst 6k
prio 1
tc class add dev ${IDEV} parent 10: classid 10:2 htb rate 512kbit ceil
1900kbit burst 6k prio 2


tc qdisc add dev ${IDEV} parent 10:1 handle 21:0 pfifo
tc qdisc add dev ${IDEV} parent 10:2 handle 22:0 sfq

tc filter add dev ${IDEV} protocol ip pref 1 parent 10: handle 1 fw classid
10:1
tc filter add dev ${IDEV} protocol ip pref 2 parent 10: handle 2 fw classid
10:2

#iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS
--clamp-mss-to-pmtu
#iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 128

iptables -t mangle -N IMQ${IQDEV}
iptables -t mangle -A IMQ${IQDEV} -p icmp -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 23 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 25 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 53 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 80 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 110 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 443 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 21 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 22 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 6667 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 20 -j MARK --set-mark 1
iptables -t mangle -A IMQ${IQDEV} -j RETURN


iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ${IQDEV}
iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ --todev ${IQDEV}

ip link set ${IDEV} up


DOWNLINK=1800
UPLINK=250
DEV=eth2

#tc qdisc del dev $DEV root
#tc qdisc del dev $DEV ingress

#####################
## UPLINK
##

tc qdisc add dev $DEV root handle 1: htb default 20

tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit ceil
${UPLINK}kbit

tc class add dev $DEV parent 1:1 classid 1:10 htb rate 100kbit ceil 150kbit
prio 0
tc class add dev $DEV parent 1:1 classid 1:20 htb rate 100kbit ceil
${UPLINK}kbit prio 1

tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10

#####################
## DOWNLINK
##

#tc qdisc add dev $DEV handle ffff: ingress
#tc filter add dev $DEV parent fffff: protocol ip u32 match ip dst 0.0.0.0/0
police rate ${DOWNLINK}kbit burst 30k drop flowid :1

############################
## Filtros de prioridades
##

tc filter add dev $DEV protocol ip parent 1:0 prio 0 handle 1 fw classid
1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip protocol 1
0xff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 80
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 25
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 110
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 22
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 8080
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 53
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 6667
0xffff flowid 1:10


DOWNLINK=1800
UPLINK=250
DEV=eth0

#tc qdisc del dev $DEV root
#tc qdisc del dev $DEV ingress

#####################
## UPLINK
##

tc qdisc add dev $DEV root handle 1: htb default 20

tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit ceil
${UPLINK}kbit

tc class add dev $DEV parent 1:1 classid 1:10 htb rate 200kbit ceil 150kbit
prio 0
tc class add dev $DEV parent 1:1 classid 1:20 htb rate 100kbit ceil
${UPLINK}kbit prio 1

tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10

#####################
## DOWNLINK
##

#tc qdisc add dev $DEV handle ffff: ingress
#tc filter add dev $DEV parent fffff: protocol ip u32 match ip dst 0.0.0.0/0
police rate ${DOWNLINK}kbit burst 30k drop flowid :1

############################
## Filtros de prioridades
##

tc filter add dev $DEV protocol ip parent 1:0 prio 0 handle 1 fw classid
1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip protocol 1
0xff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 80
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 25
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 110
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 22
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 8080
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 53
0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 6667
0xffff flowid 1:10


[Index of Archives]     [LARTC Home Page]     [Netfilter]     [Netfilter Development]     [Network Development]     [Bugtraq]     [GCC Help]     [Yosemite News]     [Linux Kernel]     [Fedora Users]
  Powered by Linux