Trimming… > On Mar 24, 2020, at 11:57 AM, Grant Taylor <gtaylor@xxxxxxxxxxxxxxxxxx> wrote: > > QoS has the ability to guarantee an SLA of 40 & 8 to production. > > Think about it this way: > > 1) Production gets up to it's SLA. > 2) Guest gets up to it's SLA. > 3) Production and / or guest get any unused bandwidth. > > Each class is guaranteed their SLA, and can optionally use any remaining bandwidth (unused bandwidth of other classes). If they’re both oversubscribed, then how does it get divvied up? > >> Right. In this case I’m limiting (or pacing) the ACKs so that the sender paces his data. > > That's not what I was referring to. > > QoS can rate limit what is sent out the internal interfaces at the 40 / 10 Mbps values. Sure. > > The thing that it can not do is rate limit want comes in the outside interface. There may be ~12 Mbps of incoming traffic for guests. But the router will only send 10 Mbps of that out it's inside interface. Thus the router is rate limiting what guest receives to 10 Mbps. It's just that there is an additional 2 Mbps that the router is dropping on the floor. > > Does that make sense? Well, yeah. That can happen at any time and there’s nothing you can do about it. I’ve been the target of a DDoS reflection attack and 99% of my traffic was TCP RST’s and ICMP Unreachable… and that’s just what was getting through… not what was being dropped upstream. Serenity prayer here… > >> For UDP not at all. For TCP you can apply back pressure, as above. If the sender has filled his window, and I hold back any ACKs, he can’t send anything more until I do send an ACK. > > See above. > > It's possible for a router to use QoS to rate limit any type of traffic. The router quite literally receives the traffic on one interface and sends it out another interface. The rate that the traffic is sent is what is rate limited. > > TCP, UDP, ICMP, it doesn't matter what type of traffic. Sure. And by applying controls inside the firewall, you affect the perceived end-to-end properties as seen by the sender. Which is about the best you can hope for. > >> Correct. Eventually the sender will back off in an attempt to reach a congestion-free steady state. > > I would bet that a "congestion-free steady state" is /never/ achieved. The very design of most protocols is to send as fast as possible. When they detect errors, they /may/ slow down /for/ /a/ /little/ /while/. But they will speed back up. Sure. Because we use congestion control, and not congestion avoidance. It’s assumed that we will periodically hit a congested state… > > Even if a given flow could achieve something resembling a congestion-free steady state, the nature of Internet traffic is so inconsistent that you have flows starting & stopping all the time. Thus you have wildly shifting demands on traffic. Of course. For a relatively small SoHo network, it’s better understood, but also more bursty since they’re less statistical smoothing taking place. > >> My scenario, as I said, is a SoHo router. I don’t have a lot of servers behind it that receive bursts of incoming traffic asynchronously from outside (other than email, which I host locally). > > IMHO servers are actually less of a problem than the average person surfing the web. > > Every single web page you go to is at least one new and short lived flow. Many web pages are 100s of new and short lived flows. Most of them start at about the same time. Although with HTTP 1.1 and pipelining that’s better than the request-per-connection of HTTP 1.0. > > The more web surfers you have, the more of these types of traffic patterns that you have. It's also very random when they will happen. You could have anywhere between 0 and the number of people on your network at the same time. > > Also, multiple windows / tabs mean that more and more of these can happen at the same time. Sure. > >> If my daughter decides to watch an HD movie on an iPad during the day while I’m working, I don’t want that traffic overrunning my network and causing me to not be able to work. In that scenario, the connection is originating internally and going outbound, and it’s long-lived (where "long-lived" is any duration of 20 or more RTT’s). > > That's one of the things that QoS is quite good at dealing with. > > Though I question how long lived your daughter's streams actually are. I know for a fact that YouTube is a series of small downloads. So each download is relatively short lived. It's not one long lived connection that lasts for the duration of the video. > > There also the fact that YouTube prefers QUIC, which is UDP based, to TCP if it can use it. More worried about Netflix, Hulu, and Disney+ which are all TCP-based. All three (and possibly Amazon Prime, I don’t remember) use HTTP byte-ranges, but reuse the same connection. So one connection, but bursty fetches… > >> Only slightly less for me: I did a traffic-shaper plugin for Arno’s Internet Firewall (AIF) about 12 years ago. I’ve since forgotten everything. > > Tempus fugit. Indeed. Although there are moments I wouldn’t recapture even if I could. > >> Well, know you’ve got me confused. Because if each can borrow from the other, where’s the SLA? Where’s the cap? Who gets prioritized? > > I think I explained it above. > > Each is guaranteed the availability of it's SLA. The unused traffic over the SLA is (can be) fair game. > > Meaning that if production is using 15 & 3, there is 25 & 5 that guest could use if allowed to. > > Similarly, if guests are sleeping, there is an additional 10 & 2 that production could take advantage of. And if they both want to go over quota… I guess they can compete. > >> Yeah, and indeed that’s what HTB excels at. > > Yep. > > If memory serves, HTB is one of many that can do it. But HTB was one of the earlier options. I like HTB because it’s very straightforward to model in simulations, etc. > >> Some ISPs were actually squashing the bits, and got spanked severely by the FCC. > > Okay. I don't recall that. I wonder why they wanted to stomp on ECN. Especially seeing as how ECN is to alert about congestion. Lack of congestion notification encourages additional ramp up. > > I'm assuming that ISPs were clearing ECN. Maybe I have this backwards. Maybe they were artificially setting it to induce slowdowns. No, they were clearing it because they thought they were protecting subscribers with not-up-to-date equipment from being confused by seeing markings they didn’t know how to correctly interpret. Odd, considering that customer equipment often moves faster than ISP or RBOC’s. The whole 5 years I was at Cisco, several RBOC’s were still running 12.0S (and insisting on continued support) even as I was writing features for 12.4(T)… And they had only recently migrated off 11.3 Mainline. > >> Also, some older router’s IP stacks were not ECN aware, and had the older bit definitions (remember that RFC 3168 and ECN borrowed the ECT1 bit from TOS/LOWCOST from RFC 791 and 1349). > > My experience has been that most routers ignore QoS / ECN. That’s typically a configuration issue, and not a question of not having current software. https://www.reddit.com/r/linux/comments/933vys/is_tcp_ecn_still_a_problem_today/ https://www.bufferbloat.net/projects/cerowrt/wiki/Enable_ECN/ Especially: https://www.ietf.org/proceedings/89/slides/slides-89-tsvarea-1.pdf slide 14 Unfortunately most of the surveys on how widely deployed ECN marking in transit networks is, is 12-19 years old. > >> I’m assuming a 3.18 kernel or later and iproute2 + iptables. Nothing else. And sch_htb is present. > > Unfortunately there are a LOT of possible combination in that mix. > > I also know that the Ubiquity EdgeRouter Lite uses a 2.6 kernel. I don't know about other EdgeOS (Ubiquity Linux distro). But I wouldn't be surprised to learn that EdgeOS is 2.6 period. Yeah, I was going to work at Ubiquiti on the OS update until they made a salary offer… > >> This is the same problem that ifb solves, right? > > Probably. (See above.) > >> I’m not sure I want to assume that Namespaces are available in all scenarios. > > Fair enough. > > I run old PCs with standard Linux distros as routers. So I can easily add what I want to them. I’m using Supermicro pizza boxes (mostly SYS-5018D’s) that require EFI support… > >> Yeah, for now I’m not concerned about internal traffic. Yet. > > That's something to keep in mind when creating the QoS configuration. As in you might need to take it into account and make sure that you don't artificially slow it down. Sure. Though on Gigabit interfaces, 50mbps is not statistically significant even if I blocked it out… > >> As I remember, some of the newer (model-based) congestion avoidance algorithms (like BBR) were really much better at fairness and avoiding dropped packets… > > My understanding is that BBR is rather aggressive in that it tries to identify what the bandwidth is and then use all of that bandwidth that it can. It's particularly aggressive at finding the bandwidth too. I remember people having the same complaint about Reno back in the day… Ever wake up and realize “I’m old”? Well, my wife wakes up every day and says to me, “You’re old.” But not the same thing… -Philip
