On 05/13/24 14:26, Steven Rostedt wrote: > On Mon, 13 May 2024 10:03:59 +0200 > Peter Zijlstra <peterz@xxxxxxxxxxxxx> wrote: > > > > I believe we agree that we want more people contributing to the scheduling > > > area. > > > > I think therein lies the rub -- contribution. If we were to do this > > thing, random loadable BPF schedulers, then how do we ensure people will > > contribute back? > > Hi Peter, > > I'm somewhat agnostic to sched_ext itself, but I have been an advocate > for a plugable scheduler infrastructure. And we are seriously looking > at adding it to ChromeOS. > > > > > That is, from where I am sitting I see $vendor mandate their $enterprise > > product needs their $BPF scheduler. At which point $vendor will have no > > incentive to ever contribute back. > > Believe me they already have their own scheduler, and because its so > different, it's very hard to contribute back. > > > > > And customers of $vendor that want to run additional workloads on > > their machine are then stuck with that scheduler, irrespective of it > > being suitable for them or not. This is not a good experience. > > And $vendor usually has a unique workload that their changes will > likely cause regressions in other workloads, making it even harder to > contribute back. > > > > > So I don't at all mind people playing around with schedulers -- they can > > do so today, there are a ton of out of tree patches to start or learn > > from, or like I said, it really isn't all that hard to just rip out fair > > and write something new. > > For cloud servers, I bet a lot of schedulers are not public. Although, > my company tries to publish the schedulers they use. > > > > > Open source, you get to do your own thing. Have at. > > > > But part of what made Linux work so well, is in my opinion the GPL. GPL > > forces people to contribute back -- to work on the shared project. And I > > see the whole BPF thing as a run-around on that. > > > > Even the large cloud vendors and service providers (Amazon, Google, > > Facebook etc.) contribute back because of rebase pain -- as you well > > know. The rebase pain offsets the 'TIVO hole'. > > From what I understand (I don't work on production, but Chromebooks), a > lot of changes cannot be contributed back because their updates are far > from what is upstream. Having a plugable scheduler would actually allow > them to contribute *more*. > > > > > But with the BPF muck; where is the motivation to help improve things? > > For the same reasons you mention about GPL and why it works. > Collaboration. Sharing ideas helps everyone. If there's some secret > sauce scheduler then they would likely just replace the scheduler, as > its more performant. I don't believe it would be worth while to use BPF > for that purpose. > > > > > Keeping a rando github repo with BPF schedulers is not contributing. > > Agreed, and I would guess having them in the Linux kernel tree would be > more beneficial. > > > That's just a repo with multiple out of tree schedulers to be ignored. > > Who will put in the effort of upsteaming things if they can hack up a > > BPF and throw it over the wall? > > If there's a place in the Linux kernel tree, I'm sure there would be > motivation to place it there. Having it in the kernel proper does give > more visibility of code, and therefore enhancements to that code. This > was the same rationale for putting perf into the kernel proper. > > > > > So yeah, I'm very much NOT supportive of this effort. From where I'm > > sitting there is simply not a single benefit. You're not making my life > > better, so why would I care? > > > > How does this BPF muck translate into better quality patches for me? > > Here's how we will be using it (we will likely be porting sched_ext to > ChromeOS regardless of its acceptance). > > Doing testing of scheduler changes in the field is extremely time > consuming and complex. We tested EEVDF vs CFS by backporting EEVDF to > 5.15 (as that is the kernel version we are using on the chromebooks we > were testing on), and then we need to add a user space "switch" to > change the scheduler. Note, this also risks causing a bug in adding > these changes. Then we push the kernel out, and then start our > experiment that enables our feature to a small percentage, and slowly > increases the number of users until we have a enough for a statistical > result. > > What sched_ext would give us is a easy way to try different scheduling > algorithms and get feedback much quicker. Once we determine a solution > that improves things, we would then spend the time to implement it in > the scheduler, and yes, send it upstream. > > To me, sched_ext should never be the final solution, but it can be > extremely useful in testing various changes quickly in the field. Which > to me would encourage more contributions. I really don't think the problems we have are because of EEVDF vs CFS vs anything else. Other major OSes have one scheduler, but what they exceed on is providing better QoS interfaces and mechanism to handle specific scenarios that Linux lacks. The confusion I see again and again over the years is the fragmentation of Linux eco system and app writers don't know how to do things properly on Linux vs other OSes. Note our CONFIG system is part of this fragmentation. The addition of more flavours which inevitably will lead to custom QoS specific to that scheduler and libraries built on top of it that require that particular extension available is a recipe for more confusion and fragmentation. Not to mention big players are likely to take over, and I wouldn't be surprised if new business models start to spring up on top of that. Add to the lot the potential security issues with the ease to lure people to download sneaky sched extension that gives great promises but full of malware (more dangerous with the greater power of BPF/sudo misused). I really don't buy the rapid development aspect too. The scheduler was heavily influenced by the early contributors which come from server market that had (few) very specific workloads they needed to optimize for and throughput had a heavier weight vs latency. Fast forward to now, things are different. Even on server market latency/responsiveness has become more important. Power and thermal are important on a larger class of systems now too. I'd dare say even on server market. How do you know when it's okay for an app/task to consume too much power and when it is not? Hint hint, you can't unless someone in userspace tells you. Similarly for latency vs throughput. What is the correct way to write an application to provide this info? Then we can ask what is missing in the scheduler to enable this. Note the original min/wakeup_granularity_ns, latency_ns etc were tuned by default for throughput by the way (server market bias). You can manipulate those and get better latencies. And this brings me to the major point, we really need to stop thinking that we must improve everything at system level. Workloads need to evolve to take best out of systems and we need new libraries for performance and power management. And this means they need to get new APIs and libraries do a be able to do a better job and scale well. I agree with Peter it is not hard to write something to make specific workload better. But what we really need is enable workloads to be written better and be more portable to take best of the hardware they run on, AND coexist with other workloads. For example, how do you write a good multi threaded application that can scale well across systems (including big.LITTLE) and not trip over other workloads stealing resources sometimes? You need something like this https://developer.apple.com/documentation/DISPATCH which has a linux port https://github.com/apple/swift-corelibs-libdispatch not a new scheduler. How do you write an app that can manage bad thermal situations? https://developer.android.com/games/optimize/adpf/thermal POSIX is dormant, and every OS has to wing new interfaces to deal with the new realities. And I don't see a lot of these discussions. Linux is lagging behind in general in this aspect. The trend I see is how do I make existing stuff better, and believe me I've seen strcmp(task->comm, ...) to hand pick things. Which I am sure we'll end up down this path if we let things loose. So I am against any custom extension. I think it all has to be part of the kernel tree and adhere to all of its supported interfaces. Which I think what we really ought on focusing to evolve and improve. This is the biggest friction point IMO, not the scheduler algorithm. If the latter need to change, it needs to be as the result of this friction - which what EEVDF came about from to my understanding. To enable implementing a latency interface easier. But Vincent had a working implementation with CFS too which I think would have worked fine by the way. I do hope we can reconsider some of our default behaviors though (that bias to perf and throughput specifically). FWIW IMO the biggest issues I see in the scheduler is that its testability and debuggability is hard. I think BPF can be a good fit for that. For the latter I started this project, yet I am still trying to figure out how to add tracer for the difficult paths to help people more easily report when a bad decision has happened to provide more info about the internal state of the scheduler, in hope to accelerate the process of finding solutions. I think people are getting stuck explaining why things are failing, which makes finding a common solution hard if not impossible. We need better way to understand the problems people are seeing https://github.com/qais-yousef/sched-analyzer Similar methodology can be used to create a BPF based sched test framework. I don't have cycles to start this, but hope to if no one beats me to it. I think it would be great to have a clear list of the current limitations people see in the scheduler. It could be a failure on my end, but I haven't seen specifics of problems and what was tried and failed to the point it is impossible to move forward. From what I see, I am hitting bugs here and there all the time. But they are hard to debug to truly understand where things went wrong. Like this one for example where PTHREAD_PRIO_PI is a NOP for fair tasks. Many thought using this flag doesn't help (rather than buggy).. https://lore.kernel.org/lkml/20240403005930.1587032-1-qyousef@xxxxxxxxxxx/
