all: After a pretty long layoff from kernel development of any sort of mine... we're pushing the limits again trying to get cake to support 10s of thousands of ISP subscribers, leveraging xdp for reads, and ebpf "pping" for inline monitoring of TCP rtts, then a ton of htb bins + cake for each subscriber. shameless plugs: https://github.com/thebracket/cpumap-pping#cpumap-pping # everybody needs kathie nichols and simon 's pping!!! https://github.com/LibreQoE (even more shameless plug - ask your isp to try libreqos.io out - presently at 10k users, pushing 11gbit/sec, 24% of 16 cores! I'm really, really amazed by all this, I remember struggling to get cake to crack 50Mbits on a 600mhz mips box a decade back. Way to go everyone! - love, rip van winkle) Anyway... after adopting xdp fully over the past couple months... most of the CPU time is now spent in htb, and while I see htb has been successfully offloaded by the mlx5, it's not clear if that can pull from from cake as an underlying qdisc (?), or only a pfifo, nor how much buffering offloads like this introduce. ? Are there other cards with an htb offload now? Secondly - The xdp path is marvelous, but in trying to drive this transparent bridge to 100Gbit, I find myself wanting something old fashioned, and in my mind, simpler than a match pattern - is there any ethernet card that lets you do a TCAM mapping of a large (say, 128 thousand) set of IP addresses to an irq or queue ? 1.2.3.4/29 -> irq 8 (or hw queue 8) 1.2.4.1/32 -> irq 9 (or hw queue 9) a:b:c:d::/56 -> irq 8 (etc * 10s of thousands of other ip addresses, 1 or more LPM matches per subscriber) Needn't be big enough to fill a bgp table... This is different from RPS in that we don't want a rxhash to spread the load "evenly", but to always direct a set of IP addresses to a particular core, on a particular queue - which is setup to do all that htb-ing (32k unique bins per core, e.g. 1.9m bins on 64 cores) and cake-ing. Other ideas for steppingstones on the march to 100gbit forwarding through tons of cake gladly considered. We're going to be fiddling with the metadata stuff in xdp as well - 3 hw hashes for cake, a rx timestamp for codel will probably help there too. -- This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz Dave Täht CEO, TekLibre, LLC
|