Hello, On Fri, May 03, 2024 at 10:52:32AM +0200, Peter Zijlstra wrote: > On Thu, May 02, 2024 at 09:20:15AM -1000, Tejun Heo wrote: > > We can resurrect the discussion on that patchset but how is that connected > > to sched_ext? > > I'm absolutely not taking any of this until at the very least the cgroup > situation that's been created is solved. And even then, I fundamentally > believe the approach to be detrimental to the scheduler eco-system. Please see below for more on the flattened hierarchy patchset. However, no matter how that discussion works out, what you seem to be suggesting - suspending discussion or further push for upstream on sched_ext until a mostly unrelated work is done - doesn't seem reasonable, especially when most of the input that you have provided is not constructive. Even if we agree that, for some reason, the two projects are linked and Meta and Google owe to push the flattened hierarchy patchset to land sched_ext upstream, it should be obvious how your proposition puts us in an impossible spot - A is a prerequisite for B but B isn't going to happen. That's not a motivating situation for anyone. If working on the flattened hierarchy patchset is something you want us to commit to as a gesture of good will, we can surely consider that, but that shouldn't block further discussions on sched_ext or its upstreaming. (I reordered your comment about the number of sched_ext schedulers and developer attention towards the end of the reply to avoid jumping back and forth between subjects.) > You guys Google/Facebook got us the cgroup thing, Google did a lot of We can't divorce ourselves completely from the organizations that we work for but the above is still a pretty broad stroke. Neither David nor I was involved in the CPU controller design or implementation and I don't think it's the same group of people on the Google side either. We sure can discuss how to proceed on the flattened hierarchy patchset but I don't think the picture you're painting is a fair depiction of the overall situation. > the work for cpu-cgroup, and now you Facebook say you can't live with it > because it's too expensive. Yes Rik did put a lot of effort into it, but > Google shot it down. What am I to do? You could have encouraged and guided the project if it felt important enough. You didn't have to but that was an option. > You Google/Facebook are touting collaboration, collaborate on fixing it. > Instead of re-posting this over and over. After all, your main > motivation for starting this was the cpu-cgroup overhead. The hierarchical scheduling overhead isn't the main motivation for us. We can't use the CPU controller for all workloads and while it'd be nice to improve that, it's pretty easy to work around especially with constantly increasing number of CPUs. Currently, most sched_ext experiments are without cgroups and even when cgroups are considered, they're just used as grouping hints. In fact, we want to try implementing hierarchical scheduling by dynamically soft-affinitizing cgroups to CPUs, which would be a bridge too far for the in-kernel scheduler, at least for now, as it wouldn't be able to handle custom affinities properly, but it is an idea worth exploring. Enabling experiments like that is definitely one of our main motivations. > From where I'm sitting, you created a problem (cpu-cgroup) and now > you're creating an even bigger problem as a work-around. Very much not > appreciated. I have a hard time agreeing. These projects don't overlap all that much. Their scopes are wildly different. That said, if this is somehow the blocker, we can talk and try to find a solution but such a solution would have to be reasonable from our end too. How else would it work? > Witness the metric ton of toy schedulers written for it, that's all > effort not put into improving the existing code. This view works only if you assume that the entire world contains only a handful of developers who can work on schedulers. The only way that would be the case is if the barrier of entry is raised unreasonably high. Sometimes a high barrier of entry can't be avoided or is beneficial. However, if it's pushed up high enough to leave only a handful of people to work on an area as large as scheduling, something probably is wrong. You know better than anyone that there's no such thing as the perfect scheduler for all, or even most, workloads. There are too many interacting factors and second-order effects for a single implementation, no matter how advanced, to be perfect or even great for the multitudes of situations that scheduling encounters. With hardware and workloads becoming more complex, the situation isn't getting any better. This partially explains why we can easily achieve significantly better behaviors for specific workloads even with a toy scheduler which is just there to demonstrate an idea. The built-in scheduler has to be good enough for everyone, and, thanks to the effort of you and the other sched maintainers, it serves that role admirably. However, that requirement also comes with stringent constraints. Radical ideas are difficult to play with. Each change has to make some sense for every use case. Nothing drastic can be introduced unless the future path can reasonably be forecast. So, development efforts must be highly orchestrated and stay consistent which justifies a higher barrier of entry and strict control. Yet, the many different ways that even simple schedulers can demonstrates sometimes significant behavior and performance benefits for specific workloads suggest that there are a lot of low hanging fruits in the area. Low hanging fruits that we can't easily reach from our current local optimum. A single implementation which has to satisfy all users all the time is unlikely to be an effective vehicle for mapping out such landscape. I believe we agree that we want more people contributing to the scheduling area. We need that. However, I have a hard time seeing how that would be achieved in the current structure. Most people can't afford to sink six months, a year, two years into a project only to eventually be nacked without any way to deploy and prove their ideas and efforts. Unfortunately, that is where we end up today in many cases. There are many smart people with bright ideas just outside the fence who are eager to develop, tune and even just play with schedulers. I believe they will flourish when they can work in an environment where scheduling experimentation is accessible and encouraged. In fact, we are already seeing that. Out of the four non-trivial sched_ext schedulers, three are either primarily driven by or have significant contributions from people who had not and wouldn't have worked on the in-kernel schedulers at all. So, here's my proposition. Let's please open it up. sched_ext hooks into sched infra but the contact surface is limited and we'll try our best to stay out of your way. I can't promise that it won't ever get in your way, but, if it ever does, just ping me and David. Resolving such situations would be our highest priority. Let us and others try out crazy ideas and find out what works. Thanks. -- tejun
