Linux QOS and prioritization of real-time data (RTP/VoIP)

["Greg Freeman" <gfreeman@xxxxxxxxxxxx>]

The questions I will be asking are:
How can I prevent the latency on the VoIP traffic I am experiencing when remote users copy files from the server to their local PC?

What rules do I need to Change/add? (I think I have too many rules at this point, and probably not the right ones, since it doesn't seem to help.)
Is it related to the sfq queuing? Ie. should I be using another queuing technique such as PQ?
(Please keep in mind I know very little Linux) and basically cut and paste  :0)

Site details:

Site 1 (Corp)
Firewall (10.0.0.1) running embedded Linux with an Ipsec tunnel to site 2. Internet connection is 640kbps/640kbps. Corporate side has a phone server with a VoIP card at 10.0.0.7. The below rules are being applied, but even with these rules in place I am seeing when data is pulled from the file server (10.0.0.2), to one of the clients at site 2, the latency on the VoIP traffic (measured by sending continuous ICMP packets from the phone server (10.0.0.7) to an IP phone (10.0.1.20) at site 2 -for testing) increases dramtically. With no data being copied from the server (10.0.0.2) to a client at site 2 the average latency is 30ms. When data is being copied from the server, it puts the average latency of the ICMP test packets to about 300ms, with some as high as 600+.

Here are the rules on the Corp side:
tc qdisc del dev ipsec0 root
tc qdisc add dev ipsec0 root handle 1: cbq bandwidth 10Mbit avpkt 1000
tc class add dev ipsec0 parent 1: classid 1:1 cbq rate 580kbit allot 1500 prio 2 bounded isolated
tc class add dev ipsec0 parent 1:1 classid 1:10 cbq rate 240kbit allot 1500 avpkt 1000 prio 1
tc class add dev ipsec0 parent 1:1 classid 1:20 cbq rate 400kbit allot 1500 avpkt 1000 prio 10
tc qdisc add dev ipsec0 parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev ipsec0 parent 1:20 handle 20: sfq perturb 10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 10 u32 match ip protocol 6 0xff match ip src 10.0.0.0/24 flowid 1:20
tc filter add dev ipsec0 parent 1:0 protocol ip prio 10 u32 match ip protocol 17 0xff match ip src 10.0.0.0/24 flowid 1:20
tc filter add dev ipsec0 parent 1:0 protocol ip prio 10 u32 match ip protocol 1 0xff match ip src 10.0.0.0/24 flowid 1:20
tc filter add dev ipsec0 parent 1:0 protocol ip prio 4 u32 match ip protocol 6 0xff match ip src 10.0.0.7 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 17 0xff match ip src 10.0.0.7 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 1 0xff match ip src 10.0.0.7 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 17 0xff match ip dst 10.0.1.20 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 17 0xff match ip dst 10.0.1.21 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 17 0xff match ip dst 10.0.1.22 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 4 u32 match ip protocol 6 0xff match ip dst 10.0.1.20 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 4 u32 match ip protocol 6 0xff match ip dst 10.0.1.21 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 4 u32 match ip protocol 6 0xff match ip dst 10.0.1.22 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 1 0xff match ip dst 10.0.1.20 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 1 0xff match ip dst 10.0.1.21 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 1 0xff match ip dst 10.0.1.22 flowid 1:10

Site 2 (Remote)
Firewall (10.0.1.1) running embedded Linux with an Ipsec tunnel to site 1. Internet connection speed is 640/640kbps. Each firewall has an eth0, and eth1 interface in addition to the Ipsec0 interface. TheIP Phones are at 10.0.1.20-22.

The site 2 rules are:

tc qdisc del dev ipsec0 root
tc qdisc add dev ipsec0 root handle 1: cbq bandwidth 10Mbit avpkt 1000
tc class add dev ipsec0 parent 1: classid 1:1 cbq rate 580kbit allot 1500 prio 1 bounded isolated
tc class add dev ipsec0 parent 1:1 classid 1:10 cbq rate 240kbit allot 1500 avpkt 1000 prio 1
tc class add dev ipsec0 parent 1:1 classid 1:20 cbq rate 400kbit allot 1500 avpkt 1000 prio 10
tc qdisc add dev ipsec0 parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev ipsec0 parent 1:20 handle 20: sfq perturb 10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 10 u32 match ip protocol 6 0xff match ip src 10.0.1.0/24 flowid 1:20
tc filter add dev ipsec0 parent 1:0 protocol ip prio 10 u32 match ip protocol 17 0xff match ip src 10.0.1.0/24 flowid 1:20
tc filter add dev ipsec0 parent 1:0 protocol ip prio 10 u32 match ip protocol 1 0xff match ip src 10.0.1.0/24 flowid 1:20
tc filter add dev ipsec0 parent 1:0 protocol ip prio 4 u32 match ip protocol 6 0xff match ip src 10.0.1.20 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 17 0xff match ip src 10.0.1.20 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 1 0xff match ip src 10.0.1.20 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 4 u32 match ip protocol 6 0xff match ip src 10.0.1.21 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 17 0xff match ip src 10.0.1.21 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 1 0xff match ip src 10.0.1.21 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 4 u32 match ip protocol 6 0xff match ip src 10.0.1.22 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 17 0xff match ip src 10.0.1.22 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 1 0xff match ip src 10.0.1.22 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 4 u32 match ip protocol 6 0xff match ip dst 10.0.0.7 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 17 0xff match ip dst 10.0.0.7 flowid 1:10
tc filter add dev ipsec0 parent 1:0 protocol ip prio 1 u32 match ip protocol 1 0xff match ip dst 10.0.0.7 flowid 1:10

My reasoning behind these rules - the VoIP traffic should be mainly,if not entirely UDP,(there may be some TCP/IP with the session setup but the amount of traffic would be negligiable). Each phone would require a maximum of 80 kbps (and there are three phones). Would like to be able to lock down the bandwidth to keep the server file transfers from disturbing the voice traffic - not sure how to effectively do this.

Additional information:  average packet size of the VoIP traffic is 500bytes.  (I have also tried bumping down the avpkt to 500 on each side, but this had no effect).  I also modified the cpq rate drastically trying to limit lower priority to 100kb, and boost the size of the high priority to 440kbit but this too had no effect.  I did aply the cbq rules at one point on eth1 (wan port and it didn't appear to have any effect, I then tried eth0, which did reduce the speed of Internet browsing but not to the rate I specified, Ie. it exceed the rate so the rate must not be a true cap on what it gives to each class. 

 

Please help.