Hello, Mel. On Tue, Sep 26, 2023 at 10:20:20AM +0100, Mel Gorman wrote: > Plenty, but I'm not sure how to reconcile this. I view pluggable scheduler > as something that would be a future maintenance nightmare and our "lived > experience" or "exposure bias" with respect to the expertise of users differs > drastically. Some developers will be mostly dealing with users that have > extensive relevant expertise, a strong incentive to maximise performance > and full control of their stack, others do not and the time cost of > supporting such users is high. My experience working for distros is shorter and less extensive than yours but I believe I can appreciate the pain points to some degree. Getting dropped into an enterprise software problem where most of the environment is walled off can get really frustrating and pluggable schedulers can add unfamiliar problems on top. That said, I'd like to reiterate two counter points: First, there are multiple substantial non-distro communities that use and work on the kernel. Hyper-scalers that operate huge fleets of machines such as Google and Meta among others, public cloud operators such as Amazon and Google, hardware projects including smartphones, watches, VR headsets and so on. Because these communities tend to have more vertical integration across the software stack and extremely high usage fan-out, development resource allocation and the boundaries of what can be reasonably tried are different from distros. In addition, the cost of failing to exploit optimization opportunities is very direct and often extremely high. The cost-benefit balance is pretty one sided in the favorable direction for these communities. Second, if a distro chooses to support sched_ext, there will be new kinds of problems but that wouldn't be for nothing. While there will be stupid cases, overall, the new problems will be there because users can now do things that they couldn't do before. As I wrote before, in principle at least, adding more capabilities should be a mutually beneficial proposition for both the distros and their users. > While I can see advantages to having specific > schedulers targeting either a specific workload or hardware configuration, > the proliferation of such schedulers and the inevitable need to avoid > introducing any new regressions in deployed schedulers will be cumbersome. I'm having a bit of a hard time following here. If someone (or group) writes a custom scheduler, they'd be responsible for maintaining that, right? It shouldn't significantly increase the maintenance burden somewhere centrally. FUSE can be an imperfect analogy. There are numerous FUSE implementations. While they sometimes expose shared underlying issues, the proliferation doesn't usually lead to huge extra maintenance overhead. If you're referring to the maintenance of sched_ext itself, yes, people would get sad if things get slower or break and we'd want to avoid such situations if we reasonably can. However, we absolutely can make breaking changes if necessary. Here, BCC tools can serve as the imperfect analogy (other BPF use cases often share similar characteristics). BCC tools are coupled with the kernel in more fragile and opportunistic ways than sched_ext, and some of them do break as the kernel code changes. However, they get patched up pretty quickly and don't have any problem maintaining a thriving ecosystem. This sort of arrangement isn't rare - browser, desktop, even editor plugins are often like this. Another point which may be worth considering is that there is an innate drive towards consolidation even in open ecosystems. In most cases, over time, a large number of experiments will be boiled down to a small number of winners. Many people end up having similar needs and sooner or later something becomes dominant in the area. While I don't have a crystal ball, I don't think sched_ext schedulers are going to be much different - there will probably be a handful that are both useful in some niches and well maintained against the backdrop of many one-off and experimental ones. A bit messy maybe, but not overwhelming. Most projects flourish in similar situations. > I generally worry that certain things may not have existed in the shipped > scheduler if plugging was an option including EAS, throttling control, > schedutil integration, big.Little, adapting to chiplets and picking preferred > SMT siblings for turbo boost. In each case, integrating support was time > consuming painful and a pluggable scheduler would have been a relatively > easy out that would ultimately cost us if it was never properly integrated. > While no one wants the pain, a few of us also want to avoid the problem > of vendors publishing a hacky scheduler for their specific hardware and > discontinuing the work at that point. There are two diametric approaches. Currently, in the scheduler, everyone is forced to work on a single implementation so that all the efforts and energy can be captured in one place. Let's call this the funneling model. The opposite would be the open model, where a common framework is laid out and there are multiple competing implementations. As with anything, there are pros and cons to both approaches and the long term outcome can sometimes be counter-intuitive. For example, if a project is funneled too hard, it's unlikely to grow a large and diverse pool of contributors as there just isn't enough space to accommodate them. This limits the contributor pool which in turn fortifies the justification for funneling everyone into one thing, creating a feedback loop. When development effort is perceived as finite zero-sum resource, this way of describing the situation makes sense - "Had that developer not been forced to work on this code base, we would have lost this feature." Putting the other end into a similar proposition may be useful in seeing the inherent trade-off - "What new ideas and talents are we missing out on by constricting the contributor pool?" As I mentioned before, the closest analogy I can think of is filesystems. It's not an apples-to-apples comparison of course but there easily are an order of magnitude more developers working on filesystems compared to scheduling. Sure, a lot of efforts are diffused and duplicated but in the long run we benefit so much more by letting people explore the problem space, compete, copy from and improve upon each other. Given how much both the machines and workloads have changed, we should be trying wilder things a lot more and we can't do that with the funneled model. > I see that some friction with the current state is due to tuning knobs > moving to debugfs. FWIW, I didn't 100% agree with that move either and > carried an out-of-tree revert that displayed warnings for a time but I > understood the logic behind it. However, if the tuning parameters are > insufficient, and there is good reason to change them then the answer > is to add tuning knobs with defined semantics and document them -- not > pluggable schedulers. We've seen something along those lines recently > with nice_latency even if it turned into EEVDF instead of a new interface, > so I guess we'll see how that pans out. That only works if we assume that most of what we want to try can be reasonably covered by parameterizing EEVDF. The argument for sched_ext is that there are a lot more radical things we can and should be trying. For example, if we have many dozens of CPUs and workloads which are logically grouped, it may make more sense to allocate CPUs to workloads rather than scheduling each task's slice. Or given the close distance within a LLC domain and non-uniform LLC layouts on some chiplet processors, task-to-CPU stickiness maybe doesn’t matter anymore while spilling across multiple CCXs can benefit from application-side hinting. Or given that CPU time has become a lot flimsier as a way to measure CPU utilization, we should experiment with different metrics which may or may not include wallclock times. Or, given the proliferation of async frameworks in userspace, working closely with async runtime could be pretty interesting. That's too big a surface to cover by parametrizing EEVDF in any reasonable way. Once we know what's really useful, some part can likely be incorporated with well-defined interface but we don't know what this should look like yet and it's pretty difficult to find out without an easy way to experiment. > I get most of your points. Maybe most users will not care about a pluggable > scheduler but *some will* and they will the maintenance burden. I get your > point as well that if there is a bug and the pluggable scheduler then the > first step would be "reproduce without the pluggable scheduler" and again, > you'd be right, that is a great first step *except* sometimes they can't or > sometimes they simply won't without significant proof and that's incurs a > maintenance burden. Even if the pluggable schedulers are GPL, there still > is a burden to understood any scheduler that is loaded to see if it's the > source of a problem which means. Instead of understanding a defined number > of schedulers that are developed over time with the history in changelogs, > we may have to understand N schedulers that may be popular and that also > is painful. That's leaving aside the difficulty of what happens when > more than 1 can be loaded and interacting once containers are involved > assuming that such support would exist in the future. It's already known > that interacting IO schedulers are a nightmare so presumably interacting > CPU schedulers within the same host would also be zero fun. We should do a better job of clarifying and enforcing which IO controllers and schedulers can be employed together, but here's the same question turned around. What'd it be like if cfq had been deemed the only allowable IO scheduler while the underlying hardware was going through rapid developments? The IO schedulers and controllers being pluggable makes the overall scene more complicated but that's also what allowed the block layer to quickly adapt to the new hardware reality. > Pluggable schedulers are effectively a change that we cannot walk back > from if it turns out to be a bad idea because it potentially comes under > the "you cannot break userspace" rule if a particular pluggable scheduler > becomes popular. I believe the track record of BPF use cases provides pretty strong evidence against this. There are a lot more tools and usages which depend on kernel internals now than several years ago, but that hasn't really gotten in the way of kernel evolving in any meaningful way. We need to manage the expectations as necessary but I believe the recent history has shown that this concern doesn't really match reality. > As I strongly believe it will be a nightmare to support > within distributions where there is almost no control over the software > stack of managing user expectations, I'm opposed to crossing that line with > pluggable schedulers. While my nightmare scenarios may never be realised > and could be overblown, it'll be hard to convince me it'll not kick me in > the face eventually. I agree that this may add support cost to distros and we should take measures to limit that (e.g. make it evident which scheduler\[s\] have been active and we’d be happy to take any suggestions), but we can't just look at the costs. What benefits other parts of the industry are going to benefit enterprise users too. If enterprise customers want to use sched_ext, their reasons wouldn't be all that different from other industries. Taking a further step back. You're asking "What's the cost that sched_ext will add in terms of maintenance and support?" The question I'd like to counter with is "How much are we missing out on by making it difficult to explore and experiment in the scheduling space?" Opportunity costs are usually more challenging to factor in but they're costs all the same at the end of the day. Thanks. -- tejun
