Hello Martin, Thank you so much for taking time to answer my questions in such a detailed way. You have provided the information I was missing to understand how the priorities work, hence now I understand the importance of the quantum parameter, but also the behavior that I was observing during my HTB tests. For me quantum was mainly applicable in case of the same priorities, thus I have left it to default, i.e. calculated according to configured rates. But given the information you have provided, I definitely have a better understanding and I will see how I can adapt this parameter. Thank you again for your help. Have a great day, Ewa On Tue, Apr 5, 2016 at 7:41 PM, Martin A. Brown <martin@xxxxxxxxxxxx> wrote: > > Greetings Ewa, > > You seem to be exercising HTB quite well. > >>I have questions about how exactly priorities work in HTB. I have >>read the documentation and different websites, but it still appears >>not really clear to me, especially that I do not observe the >>“right” behavior. > > I think the 'prio' is one of the under-explained parameters to HTB. > >>I was trying to give an absolute priority to real-time traffic, but >>concurrent TCP traffic with lower priority was still sometimes >>taking over (even though its rate was limited to minimum so was >>getting almost nothing in the green mode since I know that queues >>in green mode are always sent before those in yellow). > > It's quite good that you have set the 'rate' on the concurrent TCP > traffic class to be quite low, because as you point out, green > classes always get to meet their Assured Rate, the HTB 'rate' > parameter. > > [ > > For others who may be reading this message, here's a review of our > references to class colors: > > * green: the current dequeue rate of an HTB class is below its > 'rate' (in the theory, this is called 'Assured Rate' [1]) > > * yellow: the current dequeue rate of an HTB class is between > its 'rate' and its 'ceil' > > * red: the current dequeue rate of an HTB class equals or exceeds > the 'ceil' (it is possible to go slightly over 'ceil' when > borrowing and dequeuing 'quantum' bytes) > ] > > My main (somewhat rhetorical) question in reply to you is: > > How large is the quantum? > > In the rest of my reply, I will try to describe the interaction > between the 'quantum' and 'prio' parameters in HTB operation. > >>Thus I would like to ask how exactly the priority is treated in >>HTB. >> > >From the documentation, I understand that in case of the same >>priority, the quantum and DRR is used. So no problem here. > > Here's are some framing rules, that may help (it's quite likely to > be review for you). > > * each class is either a leaf class or an ancestor class (a.k.a. > inner class) > > * leaf classes enqueue, hold and dequeue packets (they do all of > the actual work) > > * ancestor classes only offer control over borrowing; they hold no > packets, but rather control the distribution of shared resources > > * the 'prio' and 'quantum' parameters are only used in leaf > classes ('prio' and 'quantum' have no real meaning in parent > classes, even if they accept the parameters) > > * lower numbered values for the 'prio' of a class represent higher > priority > > * borrowing does not kick in until a leaf class has hit its > 'rate' (when the class becomes yellow) > > * for ancestor classes, the 'rate' and 'ceil' parameters are only > used to determine when that ancestor can lend to children (I > think that only 'ceil' is used, maybe others who know can > comment authoritatively) > > * the HTB qdisc will only service the leaf classes looking for > packets to dequeue if at least one leaf is green or if there are > ancestors are capable of lending tokens > > At any point, either there are tokens available, in which case, HTB > can send, or there are no tokens available, in which case HTB waits > until tokens are available again. Here are a couple of (the many) > possible states for an HTB tree: > > * all leaves in an HTB tree may be completely 'red'; every leaf > class is at its 'ceil'; no packets will be dequeued > > * all (or some) leaves in an HTB tree are 'yellow', but all > ancestor classes are 'red'; no packets dequeued > > * all (or some) leaves in an HTB tree are 'yellow', and some > ancestor classes are 'green' or 'yellow'; HTB will allow leaf > classes to borrow from ancestors and dequeue packets up to > 'quantum' bytes in 'prio' order > > * all (or some) leaves in an HTB tree are 'green' (ancestor > classes don't matter here); allow leaf classes to dequeue > packets up to 'quantum' bytes in 'prio' order > > Here's another way to look at it. This is what HTB does in an > infinite loop (of course, this is an inexact approximation in > pseudo-code): > > while exist(green_classes) > for leaf in ordered_by_prio(green_classes) > dequeue_and_charge_parents(leaf, quantum_bytes) > > while exist(yellow_classes) > for leaf in ordered_by_prio(yellow_classes) > if tokens_available(ancestors(leaf)) > dequeue_and_charge_parents(leaf, quantum_bytes) > > So, as you can see, any leaf class that has not spent its 'ceil' yet > gets an opportunity to send 'quantum' bytes, during each time it is > possible to send. The above is, of course, an approximation, since > packets can arrive at any time to any class....but, this is roughly > how HTB behaves. > > And, now to try to answer your questions directly. > >>However, in case of queues with different priorities (but on the >>same level), how is it exactly divided? > > All leaf classes are visited in order (by 'prio'). Each leaf class > gets to send up to 'quantum' bytes. Then, the next leaf class gets > an opportunity. And, this is the magic of 'quantum'. > > The 'quantum' parameter has significant possible effects. By > setting the 'quantum' parameter really large on one class, you can > starve other classes. But, by setting the 'quantum' very low, you > can cause more work, because HTB has to visit each class more times > to dequeue the same number of bytes (packets). > > You must balance the possibility of sibling starvation against the > amount of work incurred. > > For reference, HTB clamps the 'quantum' values at the following > minima and maxima (from 1489ff. in sch_htb.c [0]): > > cl->quantum = 1000; # -- minimum quantum > cl->quantum = 200000; # -- maximum quantum > > If you have not set the 'quantum' on any of your classes, then it is > calculated sanely for you based on the 'rate'. > >>Does the highest priority send packets as long as its queue is full >>or also only for a given quantum? > > Hopefully the above explanation answers your question. The highest > priority leaf class does not get to send as long as its queue is > full. It gets to send 'quantum' bytes for each round of service, > but that highest priority leaf class will always get the first > opportunity to send and to borrow from ancestors. > >>If it sends as long as its queue is full, how come the traffic with >>lower priority does not get stalled? > > And, this is precisely why the 'quantum' parameter exists. It's not > really that important when classes are green, but as soon as classes > turn yellow, it is the interaction of 'prio' and 'quantum' which > offers us control over resource sharing among siblings. > > With 'quantum', you get to determine how much each yellow sibling > class gets to transmit at a single service. With 'prio', you > determine which sibling transmits first. > > Analogy: If your higher priority sister has a prodigious appetite > (large 'quantum' value), then she may eat most of the available > stew! THe horrible beast! But, maybe in a different world, your > higher priority sister eats only enough to keep a bird alive (small > 'quantum' value). Considerate sister! In that case, there's plenty > of stew left for you and even your lower priority siblings. > >>Additionally, is this behavior the same when both queues are in >>yellow and in green mode (thus at parent and leaf level)? > > Also, with the reference to the yellow and green, I see that you > have read Martin Devera's theory page [1] on HTB! Wonderfully > helpful and dense stuff! > > Have a wonderful day, > > -Martin > > [0] http://lxr.free-electrons.com/source/net/sched/sch_htb.c#L1489 > [1] http://luxik.cdi.cz/~devik/qos/htb/manual/theory.htm > > -- > Martin A. Brown > http://linux-ip.net/ -- To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html