All,
I can really use some advice here. I help to maintain a server providing internet service to troops in Iraq. Bandwidth is slim, so it is a challenge. A second line is being installed and I need to modify the scripts to make it work.
The server performs NAT and the qos script runs on IMQ devices both up and down. eth1 (LAN facing) is currently SNAT'ed to eth0 (internet facing). eth2 (internet facing) will be the second line coming in. It is simple enough attaching two lines to an IMQ, but here are my concerns:
The qos script adjusts bandwidth every minute based upon quota and time remaining, so both internet facing NICs need to get an equal share. Does IMQ do load balancing on its own or do I need more? With the two internet facing NICs having different IPs, is it even possible to have a high bandwidth requirement pass through both IPs simultaneously?
How do I deal with SNAT? Can I SNAT to both eth0 and eth2 from the LAN side? Also, coming back in, eth0 is DNAT'ed to eth1. This is the same basic question--do I DNAT from eth0 and eth2 to eth1?
Attached is the script for reference. The firewall script is seperate and handles the NAT'ing, but it's pretty simple. Thanks so much in advance.
Jason
#/bin/bash
################################################################################
# This script requires two network cards, one internet facing and one
# LAN facing on a NAT machine. Read the script for required modules
# (there are a lot). Note that IMQ, if built as a module, must
# be manually loaded either by adding to this script or loading
# elsewhere (it will not autoload).
#
# First set up variables used throughout the script. This particular
# ISP uses quotas that reset every 8 hours. Upon consuming X number
# of bytes, speed restrictions occur. As the server MUST be the bottle
# neck in the link in order to maintain qos, the adjust_bandwidth()
# function takes care of this. Ensure the num_users variable below
# matches the number of IP groups in user_ip below.
################################################################################
export dev_lan=eth1 dev_net=eth0 imq_up=imq0 imq_dwn=imq1 imq_up_all=imq2
export rate_up=10880 rate_dwn=80000 max_up=57600 max_dwn=140800
export quota_dwn=2306867200 quota_up=311427072
export logfile=/var/log/bandwidth.log num_users=34 lan_sub=192.168.1
################################################################################
# Users are grouped by numbers of machines so that everyone, not every
# machine, gets equal bandwidth. Make sure IPs are assigned in a manner
# consistent with the below IP sets. A space represents one user. For
# example, the user with 5 machines gets the same bandwidth as the user
# with one. Don't forget to include both the wireless and wired cards
# for each user that has both.
################################################################################
user_ip="8 9 10:11 12 13 14:15 16 17 18 19 20 21 22 23 \
24 25 26 27:56 28 29:57 30 31:55 32:33 34 35:36 37:38:39 40:41 42:43 44:45 \
46:47 48:49 50:53 51:52 54"
################################################################################
# Clear tables and reset everything. This won't affect regular firewall
# rules, as we are using seperate chains for qos.
################################################################################
iptables -t mangle -D PREROUTING -i $dev_net -j bytes_in &> /dev/null
iptables -t mangle -D POSTROUTING -o $dev_net -j bytes_out &> /dev/null
iptables -t mangle -D PREROUTING -i $dev_lan -j qos_dwn &> /dev/null
iptables -t mangle -D POSTROUTING -o $dev_lan -j qos_up &> /dev/null
for dev in $imq_up $imq_dwn; do
tc qdisc del dev $dev root &> /dev/null
ip link set $dev down &> /dev/null
done
iptables -t mangle -D PREROUTING -i $dev_lan -j IMQ --todev 0 &> /dev/null
iptables -t mangle -D POSTROUTING -o $dev_lan -j IMQ --todev 1 &> /dev/null
for chain in bytes_in bytes_out qos_up qos_dwn; do
iptables -t mangle -F $chain &> /dev/null
iptables -t mangle -X $chain &> /dev/null
iptables -t mangle -N $chain &> /dev/null
done
################################################################################
# The follwing adjusts bandwidth according to a number of parameters.
# First, it has been determined that the max consistent speed attainable with
# this ISP is 1100/450 megabits down/up. Second, this ISP sets a quota
# cap of 2200/297 MB down/up per 8 hour period. We also hit a speed
# restriction of 512/60 kilobits/second after 700/180 MB down/up are consumed.#
#
# First, we have to determine what time it is UTC (Greenwich). The ISP
# resets the quotas at noon, 2000, and 0400 daily. We use the Linux
# epoch to accomplish this. The date command below gives the number
# of seconds since midnight, Jan 1st 1970. Dividing this by one day
# and taking the remainder gives us the number of seconds since midnight
# today. From here we can divide by seconds, minutes, and hours to
# determine what time it is and how much time remains in a quota period.
# Once we have time remaining, we divide by the amount of qouta remaining
# to determine desired speed. This is checked against our speed caps and
# max speed attainable to ensure that we are not above what is workable.
# Once we have our speed calculated, this speed is passed to the
# user_class() and parent_class functions to change the bandwidth
# allotted to the queues.
################################################################################
adjust_bandwidth() {
sectime=$(($(date +%s)%86400))
bytes_in=$(iptables -t mangle -L bytes_in -n -x -v | \
grep $dev_net | awk '{ print $2 }')
bytes_out=$(iptables -t mangle -L bytes_out -n -x -v | \
grep $dev_net | awk '{ print $2 }')
if [ $sectime -ge 0 -a $sectime -lt 14400 ]; then
secsremain=$((14400-$sectime))
elif [ $sectime -ge 14400 -a $sectime -lt 43200 ]; then
secsremain=$((43200-$sectime))
elif [ $sectime -ge 43200 -a $sectime -lt 72000 ]; then
secsremain=$((72000-$sectime))
elif [ $sectime -ge 72000 -a $sectime -lt 86400 ]; then
secsremain=$(($((86400-$sectime))+14400))
fi
if [ $bytes_in -eq 0 ]; then
speed_dwn=$(($quota_dwn/$secsremain))
elif [ $bytes_in -ge 734003200 ]; then
speed_dwn=65536
else
speed_dwn=$((($quota_dwn-$bytes_in)/$secsremain))
fi
if [ $bytes_out -eq 0 ]; then
speed_up=$(($quota_up/$secsremain))
elif [ $bytes_out -ge 188743680 ]; then
speed_up=7680
else
speed_up=$((($quota_up-$bytes_out)/$secsremain))
fi
if [ $speed_dwn -ge $max_dwn ]; then
newrate_dwn=$max_dwn
else
newrate_dwn=$speed_dwn
fi
if [ $speed_up -ge $max_up ]; then
newrate_up=$max_up
else
newrate_up=$speed_up
fi
lograte_dwn=$(($newrate_dwn*8/1024)) lograte_up=$(($newrate_up*8/1024))
echo "$(date) Speed changed to $lograte_dwn/$lograte_up \
kbit down/up." >> $logfile
parent_class change $imq_up $newrate_up
parent_class change $imq_dwn $newrate_dwn
for user in $user_ip; do
mark=`echo $user | sed -e "s/[^0-9].*//g"`
mark=$(($mark*10));
user_class change $imq_up $newrate_up $mark
user_class change $imq_dwn $newrate_dwn $mark
done
}
#################################################################################
# This is the main class structure. We set up the root qdisc with a
# lan speed class under it for non-internet transfers. A single pfifo
# qdisc and tc filter are part of this scheme to get the LAN traffic into
# the full speed queue.
#
# After this we create the interactive class for applications like voip,
# video conferencing, and delay sensative applications. A pfifo qdisc
# is attached to this as opposed to sfq to keep delay to an absolute
# minimum (we should only have one user in this band at a time, anyway).
# Then we create a low bandwidth reserve for ssh, so that we can work on the
# server during high traffic. Then we create the main internet class,
# which gets the lions share of the bandwidth. Last, we create the lowest
# priority download class, which gets a total of 20% total bandwidth. Be
# careful setting this too high, as it is difficult to slow down a
# screaming download once it reaches a high level of saturation.
###############################################################################
parent_class() {
command=$1 dev=$2 rate=$3
if [ $command == add ]; then
tc qdisc $command dev $dev root handle 1: htb default 5
tc class add dev $dev parent 1: classid 1:1 htb rate \
900000bps quantum 1510 prio 0
tc qdisc add dev $dev parent 1:1 handle 2: pfifo
tc filter $command dev $dev parent 1: protocol ip prio 1 u32 match \
ip protocol 0x6 0xff match ip tos 0x10 0xff flowid 1:1
fi
tc class $command dev $dev parent 1: classid 1:2 htb rate $(($rate))bps \
ceil $(($rate))bps quantum 1510
if [ $dev == $imq_up ]; then
tc class $command dev $dev parent 1:2 classid 1:3 htb rate \
$(($rate*5/100))bps ceil 12800bps quantum 1510 prio 1
else
tc class $command dev $dev parent 1:2 classid 1:3 htb rate \
$(($rate*5/100))bps quantum 1510 prio 1
fi
tc class $command dev $dev parent 1:2 classid 1:4 htb rate \
$(($rate*20/100))bps ceil $(($rate*25/100))bps quantum 1510 prio 3
tc class $command dev $dev parent 1:2 classid 1:5 htb rate \
$(($rate*55/100))bps ceil $(($rate))bps burst 6k cburst 6k quantum 1510 prio 2
tc class $command dev $dev parent 1:2 classid 1:6 htb rate \
$(($rate*20/100))bps ceil $(($rate*25/100))bps quantum 1510 prio 4
if [ $command == add ]; then
tc qdisc add dev $dev parent 1:3 handle 3: pfifo
tc qdisc add dev $dev parent 1:4 handle 4: sfq perturb 10
tc qdisc add dev $dev parent 1:5 handle 5: sfq perturb 10
for class in 1 3 4 5; do
tc filter add dev $dev parent 1: protocol ip prio 2 handle $class \
fw flowid 1:$class
done
fi
}
#################################################################################
# user_class() creates a class under the parent download class. We don't
# allocate everything by individual user, just downloads, as it is
# better to stay out of squid's way. squid has it's own scheme and
# it works well.
################################################################################
user_class() {
command=$1 dev=$2 urate=$(($3*20/100)) mark=$4
tc class $command dev $dev parent 1:6 classid 1:$mark htb rate \
$(($urate/$num_users))bps ceil $(($urate))bps quantum 1510 prio 5
if [ $command == add ]; then
tc qdisc add dev $dev parent 1:$mark handle $mark: sfq perturb 10
tc filter add dev $dev parent 1: protocol ip prio 3 handle $mark \
fw classid 1:$mark
fi
}
################################################################################
# user_rules picks up downloads and marks them with the mark derived from
# the user's IP address. These are then placed in the lowest priority
# class, where 20% of the pipe's total bandwidth is shared by all
# downloads. Be careful with the numbers. 256K works well. Much lower
# and you'll be placing large web pages in the slowest class, much higher
# and downloads run longer than they need to at higher speeds.
#
# upload quotas are also initiated here. We give users their share*2,
# since not all users will use their entire quota every 8 hours. Adjust
# as appropriate. This is reset every 8 hours by the daemon function
# below.
################################################################################
user_rules() {
chain=$1 flow=$2 ip=$3 mark=$4
cmd="iptables -t mangle -I $chain $flow $ip"
$cmd -m connbytes --connbytes 256000: --connbytes-dir original \
--connbytes-mode bytes -j MARK --set-mark $mark
if [ $chain == qos_up ]; then
$cmd -j DROP
$cmd -m quota2 --name $ip --quota $(($quota_up*2/$num_users)) -j ACCEPT
fi
}
################################################################################
# Here is where all of the marking takes place. Things should be self
# explanatory--we are just prioritizing and assgning marks. The reason
# port 6000 is prioritized is for the Vsee application. The
# lower numerically the mark, the higher the priority. Class 1:5 is the
# default, so this traffic needs no mark. Download traffic is pulled
# from 1:5 and placed in 1:6 in user_rules() after 256K has been consumed.
################################################################################
for chain in qos_up qos_dwn; do
cmd="iptables -t mangle -A $chain"
$cmd -s 192.168.1.0 -d 192.168.1.0 -j MARK --set-mark 1
$cmd -s 192.168.1.0 -d 192.168.1.0 -j RETURN
$cmd -p icmp -j MARK --set-mark 3
$cmd -p icmp -j RETURN
$cmd -m layer7 --l7proto sip -j MARK --set-mark 3
$cmd -m layer7 --l7proto sip -j RETURN
$cmd -m layer7 --l7proto skypeout -j MARK --set-mark 3
$cmd -m layer7 --l7proto skypeout -j RETURN
$cmd -m layer7 --l7proto skypetoskype -j MARK --set-mark 3
$cmd -m layer7 --l7proto skypetoskype -j RETURN
$cmd -p tcp --sport 6000 -j MARK --set-mark 3
$cmd -p tcp --sport 6000 -j RETURN
$cmd -p udp --sport 6000 -j MARK --set-mark 3
$cmd -p udp --sport 6000 -j RETURN
$cmd -p tcp --dport 6000 -j MARK --set-mark 3
$cmd -p tcp --dport 6000 -j RETURN
$cmd -p udp --dport 6000 -j MARK --set-mark 3
$cmd -p udp --dport 6000 -j RETURN
$cmd -p tcp --dport ssh -j MARK --set-mark 4
$cmd -p tcp --dport ssh -j RETURN
$cmd -p tcp -m tos ! --tos Minimize-Delay --sport ssh -j MARK --set-mark 4
$cmd -p tcp -m tos ! --tos Minimize-Delay --sport ssh -j RETURN
$cmd -p tcp -m tos ! --tos Minimize-Delay --dport ssh -j MARK --set-mark 4
$cmd -p tcp -m tos ! --tos Minimize-Delay --dport ssh -j RETURN
done
################################################################################
# Here we call the functions to create the classes, queues, rules, and
# filters when the script is first run. The only thing changed after
# the script's initial execution is the bandwidth and quotas.
################################################################################
parent_class add $imq_up $rate_up
parent_class add $imq_dwn $rate_dwn
for user in $user_ip; do
mark=`echo $user | sed -e "s/[^0-9].*//g"`
mark=$(($mark*10));
user_class add $imq_up $rate_up $mark $num_users
user_class add $imq_dwn $rate_dwn $mark $num_users
ip_list=`echo $user | sed -e "s/:/ /g" -e "s/[^0-9 ].*//"`
for ip in $ip_list; do
ip=$lan_sub.$ip chain_up=qos_up chain_dwn=qos_dwn flow_up="-s" flow_dwn="-d"
user_rules $chain_up $flow_up $ip $mark
user_rules $chain_dwn $flow_dwn $ip $mark
done
done
################################################################################
# Here we hook the imqs and qos chains into their respective devices and
# iptables chains.
################################################################################
iptables -t mangle -I PREROUTING -i $dev_lan -j IMQ --todev 0
ip link set $imq_up up &> /dev/null
iptables -t mangle -I POSTROUTING -o $dev_lan -j IMQ --todev 1
ip link set $imq_dwn up &> /dev/null
iptables -t mangle -I POSTROUTING -o $dev_lan -j qos_dwn
iptables -t mangle -I PREROUTING -i $dev_lan -j qos_up
iptables -t mangle -A bytes_in -i $dev_net
iptables -t mangle -A PREROUTING -j bytes_in
iptables -t mangle -A bytes_out -o $dev_net
iptables -t mangle -A POSTROUTING -j bytes_out
sleep 1
################################################################################
# This is the part of the script that runs in memory continuously. The
# timer variable divides by 60 every second. When we have no remainder,
# we are at the top of the minute. At this point, we check to see if
# we are at the top of the hour for the 8 hour quota reset. If so, user
# upload quotas are reset. Then, every minute, the adjust_bandwidth()
# function is run.
################################################################################
while :; do
timer=$(($(date +%s)%60))
if [ $timer -eq 0 ]; then
sectime=$(($(date +%s)%86400))
logfile=/var/log/bandwidth.log
if [ $sectime -eq 24400 -o $sectime -eq 43200 -o $sectime -eq 72000 ]; then
echo "$(date) Bytes in reset at $(($bytes_in/1048576))MB." >> $logfile
echo "$(date) Bytes out reset at $(($bytes_out/1048576))MB." >> $logfile
echo "$(date) Quota counters reset." >> $logfile
iptables -Z
for s in $(seq 8 57); do
quota_up=311427072 num_users=34
echo $(($quota_up*2/$num_users)) > /proc/net/xt_quota/192.168.1.$s
done
fi
adjust_bandwidth
fi
sleep 1
done
