Hello Christian, On 10/17/2024 11:49 PM, Prundeanu, Cristian wrote:
On 2024-10-17, 04:11, "Peter Zijlstra" <peterz@xxxxxxxxxxxxx> wrote:For example, running mysql+hammerdb results in a 12-17% throughputGautham, is this a benchmark you're running?
Most of our testing used sysbench as the benchmark driver. How does mysql+hammerdb work specifically? Do the tasks driving the request are located on a separate server or are co-located with the benchmarks threads on the same server? Most of our testing uses affinity to make sure the drivers do not run on same CPUs as the workload threads. If the two can run on the same CPU, then we have observed interesting behavior with a wide amount of deviation.
Most of my testing for this investigation is on mysql+hammerdb because it simplifies differentiating statistically meaningful results, but performance impact (and improvement from disabling the two features) also shows on workloads based on postgresql and on wordpress+nginx.
Did you see any glaring changes in scheduler statistics with the introduction of EEVDF in v6.6? EEVDF commits up till v6.9 were easy to revert from my experience but I've not tried it on v6.12-rcX with the EEVDF complete series. Is all the regression seen purely attributable to EEVDF alone on the more recent kernels?
How does using SCHED_BATCH compare?I haven't tested with SCHED_BATCH yet, will update the thread with results as they accumulate (each variation of the test takes multiple hours, not counting result processing and evaluation).
Could you also test running with:
echo NO_WAKEUP_PREEMPTION > /sys/kernel/debug/sched/features
In our testing, the using SCHED_BATCH prevents aggressive wakeup
preemption, and those benchmarks also showed improvements with
NO_WAKEUP_PREEMPTION. On a side note, what is the CONFIG_HZ and the
preemption model on your test kernel (most of my testing was with
CONFIG+HZ=250, voluntary preemption)
Looking at man sched for SCHED_BATCH: "the scheduler will apply a small scheduling penalty with respect to wakeup behavior, so that this thread is mildly disfavored in scheduling decisions". Would this correctly translate to "the thread will run more deterministically, but be scheduled less frequently than other threads", i.e. expectedly lower performance in exchange for less variability?So disabling them by default will undoubtedly affect a ton of other workloads.That's very likely either way, as the testing space is near infinite, but it seems more practical to first address the issue we already know about.
RUN_TO_PARITY was introduced when Chenyu discovered that a large regression in blogbench reported by Intel Test Robot (https://lore.kernel.org/all/202308101628.7af4631a-oliver.sang@xxxxxxxxx/) was the result of very aggressive wakeup preemption (https://lore.kernel.org/all/ZNWgAeN%2FEVS%2FvOLi@chenyu5-mobl2.bbrouter/) The data in the latter link helped root-cause the actual issue with the algorithm that the benchmark disliked. Similar information for the database benchmarks you are running, can help narrow down the issue.
At this time, I don't have any data points to indicate a negative impact of disabling them for popular production workloads (as opposed to the flip case). More testing is in progress (looking at the major areas: workloads heavy on CPU, RAM, disk, and networking); so far, the results show no downside.
Analyzing your approach, what you are essentially doing with the two sched features is as follows: o NO_PLACE_LAG - Without place lag, a newly enqueued entity will always start from the avg_vruntime point in the task timeline i.e., it will always be eligible at the time of enqueue. o NO_RUN_TO_PARITY - Do not run the current task until the vruntime meets its deadline after the first pick. Instead, preempt the current running task if it found to be ineligible at the time of wakeup. From what I can tell, your benchmark has a set of threads that like to get cpu time as fast as possible. With EEVDF Complete (I would recommend using current tip:sched/urgent branch to test them out) setting a more aggressive nice value to these threads should enable them to negate the effect of RUN_TO_PARITY thanks to PREEMPT_SHORT. As for NO_PLACE_LAG, the DELAY_DEQUEUE feature should help task shed off any lag it has built up and should very likely start from the zero-lag point unless it is a very short sleeper.
And sysctl is arguably more of an ABI than debugfs, which doesn't really sound suitable for workaround. And I don't see how adding a line to /etc/rc.local is harder than adding a line to /etc/sysctl.confAdding a line is equally difficult both ways, you're right. But aren't most distros better equipped to manage (persist, modify, automate) sysctl parameters in a standardized manner? Whereas rc.local seems more "individual need / edge case" oriented. For instance: changes are done by editing the file, which is poorly scriptable (unlike the sysctl command, which is a unified interface that reconciles changes); the load order is also typically late in the boot stage,
Is there any reason to flip it very early into the boot? Have you seen anything go awry with system processes during boot with EEVDF?
making it not an ideal place for settings that affect system processes.
-- Thanks and Regards, Prateek
|