After looking high and low for information on how to do this, I finally gathered enough information to do what I wanted to do. Not to mention, the archives from this list helped out greatly (even though the list is fairly new). So, I figured I'd post the results of my endeavor for those of you who are interested. I hope it helps anyone who is also trying to do this. I'll also be housing this document & changes here: http://bubba.org/cbq.html I look forward to seeing other informative posts on this list involving Linux advanced routing concepts and traffic control issues and appreciate the previous posts which helped me get to this point. Regards, Brian -- Limiting Outgoing and Incoming Bandwidth in Linux (for 2.2+ Kernels) Brian Wilson - wilson@xxxxxxxxxxxxxxxxx - 12/3/2000 Overview ----------- The point of this document is to walk you through setting up bandwidth limiting for outgoing and incoming network packets under Linux 2.2.x+ kernel series. The limiting of the outgoing bandwidth is accomplished using the built-in QoS Class Based Queuing (CBQ) functions in the 2.2.x kernel. The limiting of the incoming bandwidth is accomplished using a stand-alone kernel module called rshaper. NOTE: rshaper also requires slight modififications to your network driver source code and it not officially part of the 2.2 kernel (so use it at your own risk). Before You Begin ------------------- You first need to download a few utilities. - iproute2 ftp://ftp.funet.fi/pub/mirrors/ftp.inr.ac.ru/ip-routing/iproute2-current.tar .gz (and if you wish to limit incoming traffic as well, you'll need rshaper) - rshaper ftp://ftp.prosa.it/pub/software/rshaper-1.07.tar.gz Installation ------------ You'll first need to configure your kernel for QoS and CBQ. This is accomplished under the "Networking Options" menu. Here you need to enable Routing Messages and the Kernel/User netlink socket. Next, under "QoS and/or Fair Queuing" menu, it is probably best to enable all of the options. Below is the Networking Options portion of my kernel config. You should probably mimic this setup (unless you want to build them as modules). If you do choose to build them as modules, then you'll have to load all the specific QoS modules (sch_cbq sch_tbf sch_sfq sch_prio cls_u32) after your machine boots (or allow them to be autoloaded). Since I don't like dealing with a slew of modules, I usually compile everything in statically. <*> Packet socket [*] Kernel/User netlink socket [*] Routing messages <*> Netlink device emulation [*] Network firewalls [*] Socket Filtering <*> Unix domain sockets [*] TCP/IP networking [*] IP: multicasting [*] IP: advanced router [*] IP: policy routing [*] IP: equal cost multipath [*] IP: use TOS value as routing key [ ] IP: verbose route monitoring [ ] IP: large routing tables [*] IP: fast network address translation [ ] IP: kernel level autoconfiguration [*] IP: firewalling [ ] IP: firewall packet netlink device [*] IP: use FWMARK value as routing key [*] IP: masquerading --- Protocol-specific masquerading support will be built as modules. [*] IP: ICMP masquerading --- Protocol-specific masquerading support will be built as modules. [*] IP: masquerading special modules support <*> IP: ipautofw masq support (EXPERIMENTAL) <*> IP: ipportfw masq support (EXPERIMENTAL) <*> IP: ip fwmark masq-forwarding support (EXPERIMENTAL) [ ] IP: optimize as router not host < > IP: tunneling [*] IP: aliasing support [ ] IP: ARP daemon support (EXPERIMENTAL) [*] IP: TCP syncookie support (not enabled per default) --- (it is safe to leave these untouched) <*> IP: Reverse ARP [*] IP: Allow large windows (not recommended if <16Mb of memory) < > The IPv6 protocol (EXPERIMENTAL) --- < > The IPX protocol < > Appletalk DDP < > CCITT X.25 Packet Layer (EXPERIMENTAL) < > LAPB Data Link Driver (EXPERIMENTAL) [ ] Bridging (EXPERIMENTAL) [ ] 802.2 LLC (EXPERIMENTAL) < > Acorn Econet/AUN protocols (EXPERIMENTAL) < > WAN router [*] CPU is too slow to handle full bandwidth QoS and/or fair queueing ---> [*] QoS and/or fair queueing <*> CBQ packet scheduler <*> CSZ packet scheduler <*> The simplest PRIO pseudoscheduler <*> RED queue <*> SFQ queue <*> TEQL queue <*> TBF queue [*] QoS support [*] Rate estimator [*] Packet classifier API <*> Routing table based classifier <*> Firewall based classifier <*> U32 classifier <*> Special RSVP classifier < > Special RSVP classifier for IPv6 [*] Ingres traffic policing While you are in the kernel config, if you have decided to use rshaper and limit incoming traffic, then go ahead and modularize your network device driver (if its not already a module). < M > Sun LANCE support Doing so will be helpful in modifying the driver and setting it up for use with rshaper. If you aren't going to be using rshaper, then you may skip the next step. Installing rshaper ------------------ Unpackage the source, "make" and then read the README! (from the README) (1) in global space add the following line: int (*net_shaper_rx_hook)(struct sk_buff *skb) = NULL; (2) Only for Linux-2.2 (not for Linux-2.0), add this too: EXPORT_SYMBOL(net_shaper_rx_hook) this line must appear after the definition of the hook. (3) in the receive function of the driver change netif_rx(skb) to if (net_shaper_rx_hook) (*net_shaper_rx_hook)(skb); else netif_rx(skb); This should be it unless you have more than one network device or if you are not compiling your network driver as a module. If this is the case, please see the README included with rshaper. Enabling Changes ------------------- Compile your kernel and modules, install them and reboot if necessary. Now, we'll setup our outgoing bandwidth limitations (note, replace eth0 with your network device). Setup the queue, with your max bandwidth, mine is 10Mbit # tc qdisc add dev eth0 root handle 10: cbq bandwidth 10Mbit avpkt 1000 Add your class. Again, bandwidth is 10Mbit. weight should be bandwidth / 10. # tc class add dev eth0 parent 10:0 classid 10:1 cbq bandwidth 10Mbit \ rate 10Mbit allot 1514 weight 1Mbit prio 8 maxburst 20 avpkt 1000 Add your subclass where you'll limit the bandwidth. I'm limiting at a rate of 1024Kbit's, weight should be less than the limit, preferrably by a factor of 10 as above. # tc class add dev eth0 parent 10:1 classid 10:100 cbq bandwidth 10Mbit \ rate 1024kbit allot 1514 weight 128Kbit prio 5 maxburst 20 avpkt 1000 \ bounded We've added the queues, now you must tell the kernel they exist and how to manage them. We'll be using stochastic fairness queuing (sfq) # tc qdisc add dev eth0 parent 10:100 sfq quantum 1514b perturb 15 Now we have to tell the kernel to send packets to the queue. Notice the ip/mask. Change this depending on your target network to filter. # tc filter add dev eth0 parent 10:0 protocol ip prio 100 u32 match ip \ dst 192.168.1.0/24 flowid 10:100 If you get "RTNETLINK answers: Invalid argument" errors, chances are you didn't compile in all the necessary items into the kernel. Check my config above and make sure necessary things are there. For instance, if the last "tc" command above delivers this error, it means that QoS u32 queuing is not compiled into the kernel or the module isn't loaded. If you do not see any errors, then chances are you have correctly limited the outgoing bandwidth of the above device. Issue the "tc -d qdisc" command. It should look similar to this: # tc -d qdisc qdisc sfq 8001: dev eth0 quantum 1514b limit 128p flows 128/1024 perturb 15sec qdisc cbq 10: dev eth0 rate 10Mbit cell 8b (bounded,isolated) prio \ no-transmit/8 weight 10Mbit allot 1514b level 2 ewma 5 avpkt 1000b maxidle 23us You can verify this by connecting to your machine from a remote machine and comparing your download speeds. This is best done on an internal LAN where you're speeds are pretty much guaranteed max speeds. If you have chosen to also limit incoming bandwidth using the rshaper utility, then you'll need to load the rshaper.o module at this time. # insmod /usr/lib/rshaper.o Then you need to setup your incoming bandwidth limitation using the rshaperctl utility. For more information on this, see the rshaper README. In this case my local network is 192.168.1.0/24. # rshaperctl 192.168.1.0/24 128000 This shapes all incoming traffic on the local network to 128kb/s. Running rshapectl with no options will show you the status of shaped hosts. # rshaperctl host mask bytes-per-sec max-queue-len (s) 192.168.1.0 255.255.255.0 128000 4 To test this setup, connect from your local machine to a remote machine and compare your download speeds from this machine. Again, this is best done on an internal LAN where you're speeds are pretty much guaranteed. Script ------ Here is the script I use to start cbq on my machine: ---------------- #!/bin/sh FILTERMASK="192.168.1.0/24" # bandwidth in Kbits IN_BANDWIDTH="1024" # bandwidth in bytes OUT_BANDWIDTH="128000" DEV="eth0" PATH=/usr/sbin:/sbin echo -n "Shaping incoming traffic to ${IN_BANDWIDTH}Kbit/s " tc qdisc add dev $DEV root handle 10: cbq bandwidth 10Mbit avpkt 1000 echo -n . tc class add dev $DEV parent 10:0 classid 10:1 cbq bandwidth 10Mbit rate 10Mbit allot 1514 weight 1Mbit prio 8 maxburst 20 avpkt 1000 echo -n . tc class add dev $DEV parent 10:1 classid 10:100 cbq bandwidth 10Mbit rate ${IN_BANDWIDTH}kbit allot 1514 weight 85kbit prio 5 maxburst 20 avpkt 1000 bounded echo -n . tc qdisc add dev $DEV parent 10:100 sfq quantum 1514b perturb 15 echo -n . tc filter add dev $DEV parent 10:0 protocol ip prio 100 u32 match ip dst $FILTERMASK flowid 10:100 echo -n . echo echo -n "Shaping outgoing traffic to ${OUT_BANDWIDTH}bytes/s " modprobe rshaper rshaperctl $FILTERMASK $OUT_BANDWIDTH echo -n . echo ------------------- Questions/Comments/Information ----------------------------------- The information contained herein was gathered from a weekends worth of searches on google. Hopefully you'll find it useful and usable. I claim to be no expert on this topic by any means, so I don't know if I'll be much help if you run into problems, but I'll do my best to help. So please direct comments and questions to me, Brian Wilson wilson@xxxxxxxxxxxxxxxxxx Credits --------- http://snafu.freedom.org/linux2.2/iproute-notes.html http://www.ibiblio.org/pub/Linux/docs/HOWTO/other-formats/html_single/Adv-Ro uting-HOWTO.html#s8 http://mailman.ds9a.nl/pipermail/lartc/2000q4/ z3d and sh00p for pointing fingers and laughing.
