Re: How many (htb) tc classes and qdiscs are too many?

Linux Advanced Routing and Traffic Control

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello 

I have 4000 users and i use hfsc for shaping them.
Each class has own qdisc(esfq)


tc -s -d qdisc show dev vlan0891 | grep qdisc | wc -l
4355
 
tc -s -d qdisc show dev eth2 | grep qdisc | wc -l
4355

I use hashing filters.
System is:
P4 3.2GHz (HT enabled)
2GB RAM
2xIntel gigabit (Napi enabled)
Machine load is:


12:57:06 up 11:24,  2 users,  load average: 0.00, 0.05, 0.06


mpstat -P ALL 1 (output)

Linux 2.6.12-rc5-git6 (natjawman)       06/03/05

12:57:24     CPU   %user   %nice %system %iowait    %irq   %soft   %idle
   intr/s
12:57:25     all   12.00    0.00   30.50    0.00    0.50   14.50   42.50
  4990.00
12:57:25       0   12.00    0.00   32.00    0.00    1.00   13.00   42.00
  3390.00
12:57:25       1   12.00    0.00   29.00    0.00    0.00   16.00   42.00
  1603.00

12:57:25     CPU   %user   %nice %system %iowait    %irq   %soft   %idle
   intr/s
12:57:26     all   11.50    0.00   30.50    0.00    0.50   16.50   41.00
  4970.00
12:57:26       0   12.00    0.00   29.00    0.00    0.00   17.00   42.00
  3302.00
12:57:26       1   11.00    0.00   33.00    0.00    1.00   16.00   41.00
  1666.00

12:57:26     CPU   %user   %nice %system %iowait    %irq   %soft   %idle
   intr/s
12:57:27     all   12.94    0.00   29.85    0.00    0.50   14.43   42.29
  4998.02
12:57:27       0   12.87    0.00   30.69    0.00    0.99   14.85   40.59
  3324.75
12:57:27       1   13.86    0.00   28.71    0.00    0.00   13.86   42.57
  1674.26

12:57:27     CPU   %user   %nice %system %iowait    %irq   %soft   %idle
   intr/s
12:57:28     all   11.50    0.00   29.00    0.00    0.50   19.00   40.00
  4912.87
12:57:28       0   11.88    0.00   31.68    0.00    0.99   15.84   39.60
  3304.95
12:57:28       1   10.89    0.00   25.74    0.00    0.00   21.78   40.59
  1608.91


Peak bw is 32Mbit/s
Average bw 25Mbit/s

Machine is doing also SNAT to all clients:

iptables -L -n -v -t nat | grep SNAT | wc -l
4465

Some example script which i use for hashing filters is in attachement.






Best Regards
Paweł Staszewski
ART-COM
+48327522333
+480609183038


>>>lartc-request@xxxxxxxxxxxxxxx 06/03/05 8:37 am >>> 
Send LARTC mailing list submissions to 
lartc@xxxxxxxxxxxxxxx 
 
To subscribe or unsubscribe via the World Wide Web, visit 
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc 
or, via email, send a message with subject or body 'help' to 
lartc-request@xxxxxxxxxxxxxxx 
 
You can reach the person managing the list at 
lartc-owner@xxxxxxxxxxxxxxx 
 
When replying, please edit your Subject line so it is more specific 
than "Re: Contents of LARTC digest..." 
 
 
Today's Topics: 
 
  1. Re: how to configure linux in production line (/dev/rob0) 
  2. Re: HTB on loopback gives a bit rate multiplied by 8 
     (Kiruthika Selvamani) 
  3. Re: how to configure linux in production line (Taylor, Grant) 
  4. iproute + xml (Alberto Torres) 
  5. Re: HTB on loopback gives a bit rate multiplied by 8 
     (Andy Furniss) 
  6. How many (htb) tc classes and qdiscs are too many? (Spencer) 
  7. Re: [PATCH] Support module autoloading in iproute2 
     (Stephen Hemminger) 
  8. Re: How many (htb) tc classes and qdiscs are too many? (threaded) 
  9. Re: iproute + xml (cristian_dimache@xxxxxxxxx) 
 10. Re: How many (htb) tc classes and qdiscs are too many? 
     (Szymon Miotk) 
 
 
---------------------------------------------------------------------- 
 
Message: 1 
Date: Thu, 02 Jun 2005 06:34:14 -0500 
From: /dev/rob0 <rob0@xxxxxxxxx> 
Subject: Re:  how to configure linux in production line 
To: LARTC@xxxxxxxxxxxxxxx 
Message-ID: <429EEEB6.8050201@xxxxxxxxx> 
Content-Type: text/plain; charset=ISO-8859-1; format=flowed 
 
Gonn Star wrote: 
>I am new in linux world,basically I'm using red hat 9 
>kernel 2.4.20-8. I need to build a trusted gateway. my 
 
Whoa! You are starting out with something very old and bug-ridden. You 
should scrap that and switch to a current release, whatever distro you 
may choose. 
 
Quite a few of those old bugs can bite very hard, including root 
compromises. Being new, did you know how to update for security? Sure, 
there's Fedora Legacy which may or may not be supporting the old stuff 
with updates, but that is intended for people who have long-running 
stable servers ... not to entice new users to RH 9. 
 
>linux box will be the gateway for several machine PCs 
>to go to the desired server. there will be several 
>subnets under the linux box, I've already assigned 
>static IPs for the PCs . Now my problem is I only need 
>2 PCs from each subnets to connect to certain servers, 
>and those 2 PCs can only have transaction(open) to the 
>specified servers, for others it will 
>drop(firewalled). for other PCs, they can't log on to 
>the outside world. should I use only iptable rules or 
>with the help of squid(ACL) as well ? 
 
You do not seem to understand that HTTP is just one of many TCP/IP 
protocols, and yet you want to set up complex networking controls. 
Anyone who knows more than you do would likely find it a trivial task to

get around your controls. 
 
>please add up the commands as well. Thanks. 
 
Specific questions which show that you have tried will tend to be 
better-received than generalised requests for spoonfeeding. I do things 
like this for a living, and I do not have time to earn your living as
well. 
 
You mention "production" which implies that this is needed in a business

setting. If so it's probably worth it to the business owners to pay for 
expertise. You can't learn everything you need to know, overnight. 
 
For you, I would recommend starting with the basics. There are good 
HOWTOs at netfilter.org which might help. 
-- 
    mail to this address is discarded unless "/dev/rob0" 
    or "not-spam" is in Subject: header 
 
 
------------------------------ 
 
Message: 2 
Date: Thu, 2 Jun 2005 09:40:46 -0400 
From: Kiruthika Selvamani <kiruthika.selvamani@xxxxxxxxx> 
Subject: Re:  HTB on loopback gives a bit rate multiplied by 8 
To: Andy Furniss <andy.furniss@xxxxxxxxxxxxx> 
Cc: lartc@xxxxxxxxxxxxxxx 
Message-ID: <5ee6fe6105060206401e1f1ff@xxxxxxxxxxxxxx> 
Content-Type: text/plain; charset=ISO-8859-1 
 
Hi Andy, 
Thanks for the suggestion. I changed the MTU to 1500 and it started 
working. Is this because HTB shapes traffic based on packet rate 
rather than bit rate? How does it use the rate lookup tables? 
Thanks 
Kiruthika 
 
On 6/1/05, Andy Furniss <andy.furniss@xxxxxxxxxxxxx> wrote: 
>Kiruthika Selvamani wrote: 
>>Hi, 
>> 
>>I am trying to use htb to limit bandwidth on loopback for traffic 
>>through particular port. 
>> 
>>Here is the script I am using. 
>> 
>>tc qdisc add dev lo root handle 1: htb 
>>tc class add dev lo parent 1: classid 1:1 htb rate 100kbit ceil
100kbit 
>>tc class add dev lo parent 1:1 classid 1:10 htb rate 50kbit ceil
50kbit 
>>tc class add dev lo parent 1:1 classid 1:11 htb rate 50kbit ceil
50kbit 
>>tc filter add dev lo protocol ip parent 1:0 prio 0 u32 match ip sport 
>>22 0xffff flowid 1:10 
>>tc filter add dev lo protocol ip parent 1:0 prio 0 u32 match ip dport 
>>22 0xffff flowid 1:11 
>> 
>>When this script is applied across eth0 (when I do a sftp to another 
>>machine) the bandwidth limitation is applied correctly. However if I 
>>use this in loopback (sftp to another directory in the same machine) 
>>then I get bit rate approx 400kbit - i.e. usually it roughly 
>>multiplies the bit rate by 8. Why does this happen? Does HTB work 
>>differently in loopback? Any clue regarding this would be mostl 
>>helpful. 
> 
>It's because the MTU on lo is big and htb uses a small one when it asks

>tc to make it's rate lookup tables. 
> 
>if you do a tc -s class ls dev lo you will see there is a giants
counter 
>, giant packets are only limited as if they are the size of the biggest

>slot in the lookup table. 
> 
>To fix specify the mtu of lo on the htb classes or set the mtu on lo to

>1500. 
> 
>Andy. 
> 
 
 
------------------------------ 
 
Message: 3 
Date: Thu, 02 Jun 2005 10:46:48 -0500 
From: "Taylor, Grant" <gtaylor@xxxxxxxxxxxxxxxxx> 
Subject: Re:  how to configure linux in production line 
To: LARTC@xxxxxxxxxxxxxxx 
Message-ID: <429F29E8.4070805@xxxxxxxxxxxxxxxxx> 
Content-Type: text/plain; charset=ISO-8859-1 
 
Gonn Star wrote: 
>I am new in linux world,basically I'm using red hat 9 
>kernel 2.4.20-8. I need to build a trusted gateway. my 
>linux box will be the gateway for several machine PCs 
>to go to the desired server. there will be several 
>subnets under the linux box, I've already assigned 
>static IPs for the PCs . Now my problem is I only need 
>2 PCs from each subnets to connect to certain servers, 
>and those 2 PCs can only have transaction(open) to the 
>specified servers, for others it will 
>drop(firewalled). for other PCs, they can't log on to 
>the outside world. should I use only iptable rules or 
>with the help of squid(ACL) as well ? please add up 
>the commands as well. Thanks. 
 
 
This sounds like a fairly basic firewall with out Squid in the mix.  In
short you are probably looking at a firewall like this (NOTE:  This
script will be incomplete for just about any scenario, but will give you
the idea.): 
 
iptables -t filter -P FORWARD DROP 
iptables -t filter -F FORWARD 
iptables -t filter -A FORWARD -s 192.168.0.1 -j ACCEPT 
iptables -t filter -A FORWARD -s 192.168.0.2 -j ACCEPT 
iptables -t filter -A FORWARD -s 192.168.1.1 -j ACCEPT 
iptables -t filter -A FORWARD -s 192.168.1.2 -j ACCEPT 
iptables -t filter -A FORWARD -s 192.168.2.1 -j ACCEPT 
iptables -t filter -A FORWARD -s 192.168.2.2 -j ACCEPT 
iptables -t filter -A FORWARD -j REJECT --reject-with
icmp-net-unreachable 
 
This quick and dirty (and incomplete) script will set the default policy
(-P) of the FORWARD chain to DROP all traffic that is to be forwarded
and not handled by any other rule.  Once the default policy has been set
it flushes (-F) the FORWARD chain to make sure that there were not any
old rules lingering arround that could mess things up.  The next six
rules are in place to explicietly allow just the two machines from three
subnets (in this example) to pass traffic through the FORWARD chain on
out to a different network.  Any traffic that is not explicietly handled
by the six rules to allow traffic to be forwarded will meat the last
rule which will reject the traffic with a message saying that there is
no route to the destination thus making the computers think that they
are icolated. 
 
As someone else pointed out if you are new to the Linux community you
might be better off served by finding someone in your area with more
experience at hardening a box and a firewall to help you in this
endevor.  Or if you are not new to unix or firewalling, just Linux and
you need to acclimate your self with the Linux syntax and methodology
you will probably be ok.  Either way it would probably be worth your
time to skim some of the HOW-TOs that are out there, namely the
NetFilter HOW-TO as you are asking questions that are answered in it. 
 
 
 
Grant. . . . 
 
 
------------------------------ 
 
Message: 4 
Date: Thu, 2 Jun 2005 21:22:19 +0200 
From: Alberto Torres <perezoso@xxxxxxxxx> 
Subject:  iproute + xml 
To: lartc@xxxxxxxxxxxxxxx 
Message-ID: <850c9dea05060212225a0d8549@xxxxxxxxxxxxxx> 
Content-Type: text/plain; charset=ISO-8859-1 
 
Hello there, i am continuing with the development of the iproute GUI. 
I was wondering if there is a xml parser for the set up of the queues. 
I have been searching but i cant find any... anyone? 
 
 
------------------------------ 
 
Message: 5 
Date: Thu, 02 Jun 2005 20:32:25 +0100 
From: Andy Furniss <andy.furniss@xxxxxxxxxxxxx> 
Subject: Re:  HTB on loopback gives a bit rate multiplied by 8 
To: Kiruthika Selvamani <kiruthika.selvamani@xxxxxxxxx> 
Cc: lartc@xxxxxxxxxxxxxxx 
Message-ID: <429F5EC9.5020202@xxxxxxxxxxxxx> 
Content-Type: text/plain; charset=ISO-8859-1; format=flowed 
 
Kiruthika Selvamani wrote: 
>Hi Andy, 
>Thanks for the suggestion. I changed the MTU to 1500 and it started 
>working. Is this because HTB shapes traffic based on packet rate 
>rather than bit rate? How does it use the rate lookup tables? 
 
It's not based on packet rate as such, the lookup tables are for the 
time delay for different packet lengths at the different rates. There is

one for each rate and ceil pre calculated for efficiency. 
 
Each table has 256 slots so the mtu is needed to fill it efficiently, 
with normal mtu each slot is 8 bytes apart. If you had told htb the mtu 
of lo (16436) then each slot would have been calculated to cover a 
bigger range of bytes. 
 
I suppose the giants counter is a warning that these packets are not 
being shaped properly as they are too big. I suppose devik decided to do

this in preference to calculating the delay for every giant so it didn't

slow things down too much. 
 
Personally I am glad he didn't just use the interface mtu, as my dsl 
ppp0 gets one of 32k - it never sees a packet bigger than 1500 though, 
so if htb used 32k the shaping of small packets would be too innacurate.

 
Andy. 
 
 
 
------------------------------ 
 
Message: 6 
Date: Thu, 2 Jun 2005 16:07:31 -0600 
From: "Spencer" <spencer@xxxxxxxxxxxxxx> 
Subject:  How many (htb) tc classes and qdiscs are too many? 
To: <lartc@xxxxxxxxxxxxxxx> 
Message-ID: <000501c567bf$792fd700$650fa8c0@hotsitespencer> 
Content-Type: text/plain;charset="iso-8859-1" 
 
We have a Linux box that is acting as the gateway to the internet for
about 
400 people, typically there are not more then 50 of them using the
internet 
at any given time.  We would like to provide different levels of access
to 
different users.  For example 128kbps to some users and 256kbps to
others. 
We have considered creating a class and qdisc for each user (using htb) 
however we don't know how much overhead creating 50-200 classes and 
qdiscs would involve, would this put too much strain on the Linux box? 
Is 
it 
better to create fewer classes and qdisc and assign multiple users to
each? 
I haven't been able to find any test on maximum effect number of qdiscs,
but 
it could be I have just been looking in the wrong place.  If any one has
any 
ideas or could point me in the right direction it would be greatly 
appreciated. 
 
Spencer 
 
 
 
------------------------------ 
 
Message: 7 
Date: Thu, 02 Jun 2005 17:20:44 -0700 
From: Stephen Hemminger <shemminger@xxxxxxxx> 
Subject:  Re: [PATCH] Support module autoloading in iproute2 
To: jt@xxxxxxxxxx 
Cc: lartc@xxxxxxxxxxxxxxx 
Message-ID: <429FA25C.2030804@xxxxxxxx> 
Content-Type: text/plain; charset=ISO-8859-1; format=flowed 
 
Use module aliases and the kernel will do the autoloading. 
Most distros add something like: 
alias eth0 e100 
to /etc/modprobe.conf 
 
 
 
------------------------------ 
 
Message: 8 
Date: Thu, 02 Jun 2005 19:55:37 -0700 
From: threaded <gypsy@xxxxxxxxxx> 
Subject: Re:  How many (htb) tc classes and qdiscs are too 
many? 
To: lartc@xxxxxxxxxxxxxxx 
Cc: Spencer <spencer@xxxxxxxxxxxxxx> 
Message-ID: <429FC6A9.9F1A5BFE@xxxxxxxxxx> 
Content-Type: text/plain; charset=us-ascii 
 
Spencer wrote: 
> 
>Is it 
>better to create fewer classes and qdisc and assign multiple users to
each? 
>I haven't been able to find any test on maximum effect number of
qdiscs, but 
>it could be I have just been looking in the wrong place.  If any one
has any 
>ideas or could point me in the right direction it would be greatly 
>appreciated. 
> 
>Spencer 
 
You're not the first person to ask this.  AFAIK there is no benchmark. 
People 
just do it.  I suggest googling this ML for "hash", "internet cafe", 
"pyshaper", "PaceMaker" and whatever else that leads to.  IIRC "hotel"
may 
also be a good search word. 
 
Tomasz Paszkowski runs a HUGE script for his HFSC setup. 
 
The short answer is that, if you can create a hash that matches, you can

reduce the volume of entries; but that is more a convenience than
something 
necessary for efficiency.  It takes a HELL of a lot to make Linux groan
under 
the load.  I once spent > 1 hour loading ~32K filters, but when the
script 
finished, I could not tell they were there based on the performance of
my AMD 
Duron 1400 CPU, 256Mb RAM equipped Linux box. 
 
The following is probably the most useful single site you'll find: 
http://digriz.org.uk/ 
-- 
gypsy 
 
 
------------------------------ 
 
Message: 9 
Date: Fri, 3 Jun 2005 08:34:31 +0300 (EEST) 
From: cristian_dimache@xxxxxxxxx 
Subject: Re:  iproute + xml 
To: LARTC@xxxxxxxxxxxxxxx 
Message-ID: <50532.194.102.203.14.1117776871.squirrel@xxxxxxxxxxxxxx> 
Content-Type: text/plain;charset=iso-8859-1 
 
Let us look back on the archives: 
 
On 12 Jul 2001 17:41:42 -0500, Nikolai Vladychevski wrote: 
>But what I am trying to do is to release it for 
>production where the end users would point & click for filter creation
& 
>bandwidth definition, so I think it will be an adventure, but I am 
>accepting the risks... after all.... it's free code.... 
 
I've been working on an XML format for describing a traffic control 
configuration in-house.  We're working on a good way to describe the 
ru

Attachment: outbw.sh
Description: Bourne shell script

_______________________________________________
LARTC mailing list
LARTC@xxxxxxxxxxxxxxx
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

[Index of Archives]     [LARTC Home Page]     [Netfilter]     [Netfilter Development]     [Network Development]     [Bugtraq]     [GCC Help]     [Yosemite News]     [Linux Kernel]     [Fedora Users]
  Powered by Linux