Re: big problem with HTB/CBQ and CPU for more than 1.700 customers

Linux Advanced Routing and Traffic Control

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



u32 hash filters is the key, as somebody pointed. You can also tune your
iptables setup, like this

#192.168.1.0/24
iptables -t mangle -N 192-168-1-0-24
iptables -t mangle -A FORWARD -s 192.168.1.0/24 -j 192-168-1-0-24
iptables -t mangle -N 192-168-1-0-25
iptables -t mangle -N 192-168-1-128-25
iptables -t mangle -A 192-168-1-0-24 -s 192.168.1.0/25 -j 192-168-1-0-25
iptables -t mangle -A 192-168-1-0-24 -s 192.168.128.0/25 -j 192-168-1-128-25
.
.
and so on, until (ip 192.168.1.11, which is called in chain created for
192.168.1.10/31)

iptables -t mangle -A 192-168-1-10-31 -s 192.168.1.10 -j CLASSIFY
--set-class 1:10
iptables -t mangle -A 192-168-1-10-31 -s 192.168.1.11 -j CLASSIFY
--set-class 1:11

.. I guess you got the ideea, it requires some RAM, which i belive is
not such a big problem. Similar rules should be made for download.

Pablo Fernandes Yahoo wrote:
>
> Hello,
>
> have HTB „rules“ in 4 different ISPs and i control for each customer
> this way:
>
> Flush and 1:0 class
>
> tc qdisc del dev eth0 root
>
> tc qdisc add dev eth0 root handle 1:0 htb
>
> tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100mbit
>
> tc qdisc del dev eth1 root
>
> tc qdisc add dev eth1 root handle 1:0 htb
>
> tc class add dev eth1 parent 1:0 classid 1:1 htb rate 100mbit
>
> Upload and Download: user1
>
> tc class add dev eth0 parent 1:1 classid 1:5 htb rate 150kbit ceil 150kbit
>
> tc qdisc add dev eth0 parent 1:5 handle 5: sfq perturb 10
>
> tc class add dev eth1 parent 1:1 classid 1:5 htb rate 50kbit ceil 50kbit
>
> tc qdisc add dev eth1 parent 1:5 handle 5: sfq perturb 10
>
> iptables -t mangle -A POSTROUTING --dest x.x.x.x -o eth0 -j CLASSIFY
> --set-class 1:5
>
> iptables -t mangle -A FORWARD --src x.x.x.x -o eth1 -j CLASSIFY
> --set-class 1:5
>
> Upload and Download: user2
>
> tc class add dev eth0 parent 1:1 classid 1:8 htb rate 150kbit ceil 150kbit
>
> tc qdisc add dev eth0 parent 1:8 handle 8: sfq perturb 10
>
> tc class add dev eth1 parent 1:1 classid 1:8 htb rate 50kbit ceil 50kbit
>
> tc qdisc add dev eth1 parent 1:8 handle 8: sfq perturb 10
>
> iptables -t mangle -A POSTROUTING --dest y.y.y.y -o eth0 -j CLASSIFY
> --set-class 1:8
>
> iptables -t mangle -A FORWARD --src y.y.y.y -o eth1 -j CLASSIFY
> --set-class 1:8
>
> (…)
>
> This rules works fine, but just for less than 1.700 customers. More
> than 1.700 customers, i have my load avarage in the sky and Ksoftirqd
> process (top information) in 100% fulltime. I don’t know why. I used
> to use CBQ instead HTB because i had the same problem and Ron (a guy
> in this list) gave this rules and told me that he uses this for more
> than 3.000 customers. I tested it in more than 7 different computers
> (but the same hadware specifications) and i had the same problem with
> either CBQ or HTB rules. The computers that i have are all of them
> DELL PowerEdge 1850. I will put some hardware iformations here:
>
> top
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 3 root 39 19 0 0 0 R 100 0.0 5316:20 ksoftirqd/0
>
> [root@fw ~]# uptime
>
> 10:38:11 up 161 days, 17:21, 3 users, load average: 1.58, 1.65, 1.51
> (unfortunately when i took this, the load average was „pretty good“,
> but minutes ago, it was more than 11.0
>
> [root@fw ~]# lspci
>
> 00:00.0 Host bridge: Intel Corporation E7520 Memory Controller Hub
> (rev 09)
>
> 00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express
> Port A (rev 09)
>
> 00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI Express Port B
> (rev 09)
>
> 00:05.0 PCI bridge: Intel Corporation E7520 PCI Express Port B1 (rev 09)
>
> 00:06.0 PCI bridge: Intel Corporation E7520 PCI Express Port C (rev 09)
>
> 00:1d.0 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB
> UHCI Controller #1 (rev 02)
>
> 00:1d.1 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB
> UHCI Controller #2 (rev 02)
>
> 00:1d.2 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB
> UHCI Controller #3 (rev 02)
>
> 00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2
> EHCI Controller (rev 02)
>
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c2)
>
> 00:1f.0 ISA bridge: Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC
> Interface Bridge (rev 02)
>
> 00:1f.1 IDE interface: Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE
> Controller (rev 02)
>
> 01:00.0 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor
> (A-Segment Bridge) (rev 06)
>
> 01:00.2 PCI bridge: Intel Corporation 80332 [Dobson] I/O processor
> (B-Segment Bridge) (rev 06)
>
> 02:0c.0 Ethernet controller: Intel Corporation 82545GM Gigabit
> Ethernet Controller (rev 04)
>
> 02:0e.0 RAID bus controller: Dell PowerEdge Expandable RAID controller
> 4 (rev 06)
>
> 03:0b.0 Ethernet controller: Intel Corporation 82545GM Gigabit
> Ethernet Controller (rev 04)
>
> 05:00.0 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI
> Bridge A (rev 09)
>
> 05:00.2 PCI bridge: Intel Corporation 6700PXH PCI Express-to-PCI
> Bridge B (rev 09)
>
> 06:07.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit
> Ethernet Controller (rev 05)
>
> 07:08.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit
> Ethernet Controller (rev 05)
>
> 09:0d.0 VGA compatible controller: ATI Technologies Inc Radeon RV100
> QY [Radeon 7000/VE]
>
> [root@fw ~]# free -m
>
> total used free shared buffers cached
>
> Mem: 2021 1479 542 0 400 654
>
> -/+ buffers/cache: 424 1597
>
> Swap: 1027 0 1027
>
> [root@fw ~]# cat /proc/cpuinfo
>
> processor : 0
>
> vendor_id : GenuineIntel
>
> cpu family : 15
>
> model : 4
>
> model name : Intel(R) Xeon(TM) CPU 3.00GHz
>
> stepping : 3
>
> cpu MHz : 2992.674
>
> cache size : 2048 KB
>
> physical id : 0
>
> siblings : 2
>
> core id : 0
>
> cpu cores : 1
>
> fdiv_bug : no
>
> hlt_bug : no
>
> f00f_bug : no
>
> coma_bug : no
>
> fpu : yes
>
> fpu_exception : yes
>
> cpuid level : 5
>
> wp : yes
>
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
> pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm
> constant_tsc pni monitor ds_cpl cid cx16 xtpr
>
> bogomips : 5990.78
>
> processor : 1
>
> vendor_id : GenuineIntel
>
> cpu family : 15
>
> model : 4
>
> model name : Intel(R) Xeon(TM) CPU 3.00GHz
>
> stepping : 3
>
> cpu MHz : 2992.674
>
> cache size : 2048 KB
>
> physical id : 0
>
> siblings : 2
>
> core id : 0
>
> cpu cores : 1
>
> fdiv_bug : no
>
> hlt_bug : no
>
> f00f_bug : no
>
> coma_bug : no
>
> fpu : yes
>
> fpu_exception : yes
>
> cpuid level : 5
>
> wp : yes
>
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
> pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm
> constant_tsc pni monitor ds_cpl cid cx16 xtpr
>
> bogomips : 5985.13
>
> Any help/Tipp/hint will be very welcome.
>
> Thanks in Advance!
>
> Pablo Fernandes
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> LARTC mailing list
> LARTC@xxxxxxxxxxxxxxx
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>   

_______________________________________________
LARTC mailing list
LARTC@xxxxxxxxxxxxxxx
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

[Index of Archives]     [LARTC Home Page]     [Netfilter]     [Netfilter Development]     [Network Development]     [Bugtraq]     [GCC Help]     [Yosemite News]     [Linux Kernel]     [Fedora Users]
  Powered by Linux