Le lundi 06 mars 2023 à 17:03:30 (+0530), Shrikanth Hegde a écrit : > > > On 3/5/23 6:33 PM, Vincent Guittot wrote: > > On Sat, 4 Mar 2023 at 16:13, Shrikanth Hegde <sshegde@xxxxxxxxxxxxxxxxxx> wrote: > >> > >> > >> On 3/3/23 10:01 PM, Vincent Guittot wrote: > >>> Le jeudi 02 mars 2023 à 23:37:52 (+0530), Shrikanth Hegde a écrit : > >>>> On 3/2/23 8:30 PM, Shrikanth Hegde wrote: > >>>>> On 3/2/23 6:47 PM, Vincent Guittot wrote: > >>>>>> On Thu, 2 Mar 2023 at 12:00, Shrikanth Hegde <sshegde@xxxxxxxxxxxxxxxxxx> wrote: > >>>>>>> On 3/2/23 1:20 PM, Vincent Guittot wrote: > >>>>>>>> On Wed, 1 Mar 2023 at 19:48, shrikanth hegde <sshegde@xxxxxxxxxxxxxxxxxx> wrote: > >>>>>>>>> On 2/24/23 3:04 PM, Vincent Guittot wrote: > >>> [...] > >>> > >>>>>>>>> Ran the schbench and hackbench with this patch series. Here comparison is > >>>>>>>>> between 6.2 stable tree, 6.2 + Patch and 6.2 + patch + above re-arrange of > >>>>>>>>> latency_node. Ran two cgroups, in one cgroup running stress-ng at 50%(group1) > >>>>>>>>> and other is running these benchmarks (group2). Set the latency nice > >>>>>>>>> of group2 to -20. These are run on Power system with 12 cores with SMT=8. > >>>>>>>>> Total of 96 CPU. > >>>>>>>>> > >>>>>>>>> schbench gets lower latency compared to stabletree. Whereas hackbench seems > >>>>>>>>> to regress under this case. Maybe i am doing something wrong. I will re-run > >>>>>>>>> and attach the numbers to series. > >>>>>>>>> Please suggest if any variation in the test i need to try. > >>>>>>>> hackbench takes advanatge of a latency nice 19 as it mainly wants to > >>>>>>>> run longer slice to move forward rather than preempting others all the > >>>>>>>> time > >>>>>>> hackbench still seems to regress in different latency nice values compared to > >>>>>>> baseline of 6.2 in this case. up to 50% in some cases. > >>>>>>> > >>>>>>> 12 core powerpc system with SMT=8 i.e 96 CPU > >>>>>>> running 2 CPU cgroups. No quota assigned. > >>>>>>> 1st cgroup is running stress-ng with 48 threads. Consuming 50% of CPU. > >>>>>>> latency is not changed for this cgroup. > >>>>>>> 2nd cgroup is running hackbench. This cgroup is assigned the different latency > >>>>>>> nice values of 0, -20 and 19. > >>>>>> According to your other emails, you are using the cgroup interface and > >>>>>> not the task's one. Do I get it right ? > >>>>> right. I create cgroup, attach bash command with echo $$, > >>>>> assign the latency nice to cgroup, and run hackbench from that bash prompt. > >>>>> > >>>>>> I haven't run test such tests in a cgroup but at least the test with > >>>>>> latency_nice == 0 should not make any noticeable difference. Does this > >>>>>> include the re-arrange patch that you have proposed previously ? > >>>>> No. This is only with V12 of the series. > >>>>> > >>>>>> Also, the tests that you did on v6, gave better result. > >>>>>> https://lore.kernel.org/lkml/34112324-de67-55eb-92bc-181a98c4311c@xxxxxxxxxxxxxxxxxx/ > >>>>>> > >>>>>> Are you running same tests or you changed something in the mean time ? > >>>>> Test machine got changed. > >>>>> now i re-read my earlier mail. I see it was slightly different. > >>>>> I had created only one cgroup and stress-ng was run > >>>>> without any cgroup. Let me try that scenario and get the numbers. > >>>> Tried the same method of testing i had done on V7 of the series. on this > >>>> machine hackbench still regress's both on V12 as well as V7 of the series. > >>>> > >>>> Created one cpu cgroup called cgroup1. created two bash prompts. > >>>> assigned "bash $$" to cgroup1 and on other bash prompt running, > >>>> stress-ng --cpu=96 -l 50. Ran hackbench from cgroup1 prompt. > >>>> assigned latency values to the cgroup1. > >>> I have tried to reproduce your results on some of my systems but I can't see > >>> the impacts that you are reporting below. > >>> The fact that your other platform was not impacted as well could imply that > >>> it's specific to this platform. > >>> In particular, the lat nice=0 case should not show any real impact as it > >>> should be similar to a nop. At least that what I can see in the tests on my > >>> platforms and Prateek on his. > >>> > >>> Nevertheless, could you try to run your tests with the changes below ? > >>> These are the only places which could have an impact even with lat nice = 0 > >>> > >>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > >>> index 8137bca80572..979571a98b28 100644 > >>> --- a/kernel/sched/fair.c > >>> +++ b/kernel/sched/fair.c > >>> @@ -4991,8 +4991,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr) > >>> if (delta < offset) > >>> return; > >>> > >>> - if ((delta > ideal_runtime) || > >>> - (delta > get_latency_max())) > >>> + if (delta > ideal_runtime) > >>> resched_curr(rq_of(cfs_rq)); > >>> } > >>> > >>> @@ -7574,9 +7573,10 @@ static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity * > >>> * Otherwise, use the latency weight to evaluate how much scheduling > >>> * delay is acceptable by se. > >>> */ > >>> - if ((latency_offset < 0) || (curr->latency_offset < 0)) > >>> + if ((latency_offset < 0) || (curr->latency_offset < 0)) { > >>> latency_offset -= curr->latency_offset; > >>> - latency_offset = min_t(long, latency_offset, get_latency_max()); > >>> + latency_offset = min_t(long, latency_offset, get_latency_max()); > >>> + } > >>> > >>> return latency_offset; > >>> } > >>> @@ -7635,7 +7635,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) > >>> * for low priority task. Make sure that long sleeping task will get a > >>> * chance to preempt current. > >>> */ > >>> - gran = min_t(s64, gran, get_latency_max()); > >>> > >>> if (vdiff > gran) > >>> return 1; > >>> > >> Above patch helps. thank you. > > Great. At least we have narrow the problem to one f the 3 changes. > > > >> Numbers are comparable to 6.2 and there is slight improvement. Much better than V12 numbers. > >> > >> type groups | v6.2 |v6.2 + V12| v6.2 + V12 | v6.2 + V12 > >> | |lat nice=0| lat nice=-20| lat nice=+19 > >> > >> Process 10 | 0.33 | 0.37 | 0.38 | 0.37 > >> Process 20 | 0.61 | 0.67 | 0.68 | 0.67 > >> Process 30 | 0.85 | 0.95 | 0.95 | 0.96 > >> Process 40 | 1.10 | 1.20 | 1.20 | 1.21 > >> Process 50 | 1.34 | 1.47 | 1.44 | 1.45 > >> Process 60 | 1.57 | 1.70 | 1.71 | 1.70 > >> thread 10 | 0.36 | 0.40 | 0.39 | 0.39 > >> thread 20 | 0.65 | 0.72 | 0.71 | 0.71 > >> Process(Pipe) 10 | 0.18 | 0.31 | 0.31 | 0.33 > >> Process(Pipe) 20 | 0.32 | 0.51 | 0.50 | 0.50 > >> Process(Pipe) 30 | 0.43 | 0.65 | 0.67 | 0.67 > >> Process(Pipe) 40 | 0.57 | 0.82 | 0.83 | 0.83 > >> Process(Pipe) 50 | 0.67 | 1.00 | 0.97 | 0.98 > >> Process(Pipe) 60 | 0.81 | 1.13 | 1.11 | 1.12 > >> thread(Pipe) 10 | 0.19 | 0.33 | 0.33 | 0.33 > >> thread(Pipe) 20 | 0.34 | 0.53 | 0.51 | 0.52 > >> > >> > >> > >> type groups | v6.2 |v6.2+ V12+ | v6.2 + V12+| v6.2 + V12 > >> | |above patch|above patch | above patch > >> | |lat nice=0 |lat nice=-20| lat nice=+19 > >> > >> Process 10 | 0.36 | 0.33 | 0.34 | 0.34 > >> Process 20 | 0.62 | 0.60 | 0.61 | 0.61 > >> Process 30 | 0.87 | 0.84 | 0.85 | 0.84 > >> Process 40 | 1.13 | 1.09 | 1.10 | 1.09 > >> Process 50 | 1.38 | 1.33 | 1.33 | 1.34 > >> Process 60 | 1.64 | 1.56 | 1.58 | 1.56 > >> thread 10 | 0.35 | 0.35 | 0.35 | 0.35 > >> thread 20 | 0.64 | 0.63 | 0.63 | 0.63 > >> Process(Pipe) 10 | 0.20 | 0.18 | 0.18 | 0.18 > >> Process(Pipe) 20 | 0.32 | 0.31 | 0.31 | 0.32 > >> Process(Pipe) 30 | 0.44 | 0.43 | 0.43 | 0.43 > >> Process(Pipe) 40 | 0.56 | 0.57 | 0.56 | 0.55 > >> Process(Pipe) 50 | 0.70 | 0.67 | 0.67 | 0.67 > >> Process(Pipe) 60 | 0.83 | 0.79 | 0.81 | 0.80 > >> thread(Pipe) 10 | 0.21 | 0.19 | 0.19 | 0.19 > >> thread(Pipe) 20 | 0.35 | 0.33 | 0.34 | 0.33 > >> > >> > >> Do you want me to try any other experiment on this further? > > Yes, would be good to know which of the 3 changes in the patch create > > the regression > > > > I suspect the 1st change to be the root cause of your problem but It > > would be good if you can confirm my assumption with some tests > > > > Thanks > > Applied each change individually. 3rd change seems to cause the regression. > Kept only the 3rd change and numbers are same as stable 6.2 for latency nice > value of 0. Ok, it's the patch 1 that aims to prevent some unfairness with low weight waking task. And your platform probably falls in the last part of the commit: " Strictly speaking, we should use cfs->min_vruntime instead of curr->vruntime but it doesn't worth the additional overhead and complexity as the vruntime of current should be close to min_vruntime if not equal." Could you try the patch below on top of v12 ? --- kernel/sched/fair.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e2aeb4511686..77b03a280912 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5049,7 +5049,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) } static int -wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se); +wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct cfs_rq *cfs_rq); /* * Pick the next process, keeping these things in mind, in this order: @@ -5088,16 +5088,16 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr) second = curr; } - if (second && wakeup_preempt_entity(second, left) < 1) + if (second && wakeup_preempt_entity(second, left, cfs_rq) < 1) se = second; } - if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left) < 1) { + if (cfs_rq->next && wakeup_preempt_entity(cfs_rq->next, left, cfs_rq) < 1) { /* * Someone really wants this to run. If it's not unfair, run it. */ se = cfs_rq->next; - } else if (cfs_rq->last && wakeup_preempt_entity(cfs_rq->last, left) < 1) { + } else if (cfs_rq->last && wakeup_preempt_entity(cfs_rq->last, left, cfs_rq) < 1) { /* * Prefer last buddy, try to return the CPU to a preempted task. */ @@ -5107,7 +5107,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr) /* Check for latency sensitive entity waiting for running */ latency = __pick_first_latency(cfs_rq); if (latency && (latency != se) && - wakeup_preempt_entity(latency, se) < 1) + wakeup_preempt_entity(latency, se, cfs_rq) < 1) se = latency; return se; @@ -7808,7 +7808,7 @@ static unsigned long wakeup_gran(struct sched_entity *se) * */ static int -wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) +wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se, struct cfs_rq *cfs_rq) { s64 gran, vdiff = curr->vruntime - se->vruntime; s64 offset = wakeup_latency_gran(curr, se); @@ -7818,6 +7818,8 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) gran = offset + wakeup_gran(se); + if (vdiff > gran) + return 1; /* * At wake up, the vruntime of a task is capped to not be older than * a sched_latency period compared to min_vruntime. This prevents long @@ -7827,9 +7829,8 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) * for low priority task. Make sure that long sleeping task will get a * chance to preempt current. */ - gran = min_t(s64, gran, get_latency_max()); - - if (vdiff > gran) + vdiff = cfs_rq->min_vruntime - se->vruntime; + if (vdiff > get_latency_max()) return 1; return 0; @@ -7933,7 +7934,7 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ return; update_curr(cfs_rq_of(se)); - if (wakeup_preempt_entity(se, pse) == 1) { + if (wakeup_preempt_entity(se, pse, cfs_rq_of(se)) == 1) { /* * Bias pick_next to pick the sched entity that is * triggering this preemption. -- 2.34.1 > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index cdcd991bbcf1..c89c201dd164 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -7827,7 +7827,6 @@ wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) > * for low priority task. Make sure that long sleeping task will get a > * chance to preempt current. > */ > - gran = min_t(s64, gran, get_latency_max()); > > if (vdiff > gran) > return 1; > > > > >>>> I will try to run with only task's set with latency_nice=0 as well. > >>>> > >>>> type groups | v6.2 |v6.2 + V12| v6.2 + V12 | v6.2 + V12 > >>>> | |lat nice=0| lat nice=-20| lat nice=+19 > >>>> > >>>> Process 10 | 0.33 | 0.37 | 0.38 | 0.37 > >>>> Process 20 | 0.61 | 0.67 | 0.68 | 0.67 > >>>> Process 30 | 0.85 | 0.95 | 0.95 | 0.96 > >>>> Process 40 | 1.10 | 1.20 | 1.20 | 1.21 > >>>> Process 50 | 1.34 | 1.47 | 1.44 | 1.45 > >>>> Process 60 | 1.57 | 1.70 | 1.71 | 1.70 > >>>> thread 10 | 0.36 | 0.40 | 0.39 | 0.39 > >>>> thread 20 | 0.65 | 0.72 | 0.71 | 0.71 > >>>> Process(Pipe) 10 | 0.18 | 0.31 | 0.31 | 0.33 > >>>> Process(Pipe) 20 | 0.32 | 0.51 | 0.50 | 0.50 > >>>> Process(Pipe) 30 | 0.43 | 0.65 | 0.67 | 0.67 > >>>> Process(Pipe) 40 | 0.57 | 0.82 | 0.83 | 0.83 > >>>> Process(Pipe) 50 | 0.67 | 1.00 | 0.97 | 0.98 > >>>> Process(Pipe) 60 | 0.81 | 1.13 | 1.11 | 1.12 > >>>> thread(Pipe) 10 | 0.19 | 0.33 | 0.33 | 0.33 > >>>> thread(Pipe) 20 | 0.34 | 0.53 | 0.51 | 0.52 > >>>> > >>>> > >>>> > >>>> type groups | v6.2 |v6.2 + V7 | v6.2 + V7 | v6.2 + V7 > >>>> | |lat nice=0|lat nice=-20| lat nice=+19 > >>>> Process 10 | 0.33 | 0.37 | 0.37 | 0.37 > >>>> Process 20 | 0.61 | 0.67 | 0.67 | 0.67 > >>>> Process 30 | 0.85 | 0.96 | 0.94 | 0.95 > >>>> Process 40 | 1.10 | 1.20 | 1.20 | 1.20 > >>>> Process 50 | 1.34 | 1.45 | 1.46 | 1.45 > >>>> Process 60 | 1.57 | 1.71 | 1.68 | 1.72 > >>>> thread 10 | 0.36 | 0.40 | 0.40 | 0.40 > >>>> thread 20 | 0.65 | 0.71 | 0.71 | 0.71 > >>>> Process(Pipe) 10 | 0.18 | 0.30 | 0.30 | 0.31 > >>>> Process(Pipe) 20 | 0.32 | 0.50 | 0.50 | 0.50 > >>>> Process(Pipe) 30 | 0.43 | 0.67 | 0.67 | 0.66 > >>>> Process(Pipe) 40 | 0.57 | 0.86 | 0.84 | 0.84 > >>>> Process(Pipe) 50 | 0.67 | 0.99 | 0.97 | 0.97 > >>>> Process(Pipe) 60 | 0.81 | 1.10 | 1.13 | 1.13 > >>>> thread(Pipe) 10 | 0.19 | 0.34 | 0.34 | 0.33 > >>>> thread(Pipe) 20 | 0.34 | 0.55 | 0.53 | 0.54 > >>>> > >>>>>>> Numbers are average of 10 runs in each case. Time is in seconds > >>>>>>> > >>>>>>> type groups | v6.2 | v6.2 + V12 | v6.2 + V12 | v6.2 + V12 > >>>>>>> | | lat nice=0 | lat nice=-20| lat nice=+19 > >>>>>>> | | | | > >>>>>>> Process 10 | 0.36 | 0.41 | 0.43 | 0.42 > >>>>>>> Process 20 | 0.62 | 0.76 | 0.75 | 0.75 > >>>>>>> Process 30 | 0.87 | 1.05 | 1.04 | 1.06 > >>>>>>> Process 40 | 1.13 | 1.34 | 1.33 | 1.33 > >>>>>>> Process 50 | 1.38 | 1.62 | 1.66 | 1.63 > >>>>>>> Process 60 | 1.64 | 1.91 | 1.97 | 1.90 > >>>>>>> thread 10 | 0.35 | 0.41 | 0.44 | 0.42 > >>>>>>> thread 20 | 0.64 | 0.78 | 0.77 | 0.79 > >>>>>>> Process(Pipe) 10 | 0.20 | 0.34 | 0.33 | 0.34 > >>>>>>> Process(Pipe) 20 | 0.32 | 0.52 | 0.53 | 0.52 > >>>>>>> Process(Pipe) 30 | 0.44 | 0.70 | 0.70 | 0.69 > >>>>>>> Process(Pipe) 40 | 0.56 | 0.88 | 0.89 | 0.88 > >>>>>>> Process(Pipe) 50 | 0.70 | 1.08 | 1.08 | 1.07 > >>>>>>> Process(Pipe) 60 | 0.83 | 1.27 | 1.27 | 1.26 > >>>>>>> thread(Pipe) 10 | 0.21 | 0.35 | 0.34 | 0.36 > >>>>>>> thread(Pipe) 10 | 0.35 | 0.55 | 0.58 | 0.55 > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>> Re-arrange seems to help the patch series by avoiding an cacheline miss. > >>>>>>>>> > >>>>>>>>> ========================= > >>>>>>>>> schbench > >>>>>>>>> ========================= > >>>>>>>>> 6.2 | 6.2 + V12 | 6.2 + V12 + re-arrange > >>>>>>>>> 1 Thread > >>>>>>>>> 50.0th: 9.00 | 9.00 | 9.50 > >>>>>>>>> 75.0th: 10.50 | 10.00 | 9.50 > >>>>>>>>> 90.0th: 11.00 | 11.00 | 10.50 > >>>>>>>>> 95.0th: 11.00 | 11.00 | 11.00 > >>>>>>>>> 99.0th: 11.50 | 11.50 | 11.50 > >>>>>>>>> 99.5th: 12.50 | 12.00 | 12.00 > >>>>>>>>> 99.9th: 14.50 | 13.50 | 12.00 > >>>>>>>>> 2 Threads > >>>>>>>>> 50.0th: 9.50 | 9.50 | 8.50 > >>>>>>>>> 75.0th: 11.00 | 10.50 | 9.50 > >>>>>>>>> 90.0th: 13.50 | 11.50 | 10.50 > >>>>>>>>> 95.0th: 14.00 | 12.00 | 11.00 > >>>>>>>>> 99.0th: 15.50 | 13.50 | 12.00 > >>>>>>>>> 99.5th: 16.00 | 14.00 | 12.00 > >>>>>>>>> 99.9th: 17.00 | 16.00 | 16.50 > >>>>>>>>> 4 Threads > >>>>>>>>> 50.0th: 11.50 | 11.50 | 10.50 > >>>>>>>>> 75.0th: 13.50 | 12.50 | 12.50 > >>>>>>>>> 90.0th: 15.50 | 14.50 | 14.00 > >>>>>>>>> 95.0th: 16.50 | 15.50 | 14.50 > >>>>>>>>> 99.0th: 20.00 | 17.50 | 16.50 > >>>>>>>>> 99.5th: 20.50 | 18.50 | 17.00 > >>>>>>>>> 99.9th: 22.50 | 21.00 | 19.00 > >>>>>>>>> 8 Threads > >>>>>>>>> 50.0th: 14.00 | 14.00 | 14.00 > >>>>>>>>> 75.0th: 16.00 | 16.00 | 16.00 > >>>>>>>>> 90.0th: 18.00 | 18.00 | 17.50 > >>>>>>>>> 95.0th: 18.50 | 18.50 | 18.50 > >>>>>>>>> 99.0th: 20.00 | 20.00 | 20.00 > >>>>>>>>> 99.5th: 20.50 | 21.50 | 21.00 > >>>>>>>>> 99.9th: 22.50 | 23.50 | 23.00 > >>>>>>>>> 16 Threads > >>>>>>>>> 50.0th: 19.00 | 18.50 | 19.00 > >>>>>>>>> 75.0th: 23.00 | 22.50 | 23.00 > >>>>>>>>> 90.0th: 25.00 | 25.50 | 25.00 > >>>>>>>>> 95.0th: 26.50 | 26.50 | 26.00 > >>>>>>>>> 99.0th: 28.50 | 29.00 | 28.50 > >>>>>>>>> 99.5th: 31.00 | 30.00 | 30.00 > >>>>>>>>> 99.9th: 5626.00 | 4761.50 | 32.50 > >>>>>>>>> 32 Threads > >>>>>>>>> 50.0th: 27.00 | 27.50 | 29.00 > >>>>>>>>> 75.0th: 35.50 | 36.50 | 38.50 > >>>>>>>>> 90.0th: 42.00 | 44.00 | 50.50 > >>>>>>>>> 95.0th: 447.50 | 2959.00 | 8544.00 > >>>>>>>>> 99.0th: 7372.00 | 17032.00 | 19136.00 > >>>>>>>>> 99.5th: 15360.00 | 19808.00 | 20704.00 > >>>>>>>>> 99.9th: 20640.00 | 30048.00 | 30048.00 > >>>>>>>>> > >>>>>>>>> ==================== > >>>>>>>>> hackbench > >>>>>>>>> ==================== > >>>>>>>>> 6.2 | 6.2 + V12 | 6.2+ V12 +re-arrange > >>>>>>>>> > >>>>>>>>> Process 10 Time: 0.35 | 0.42 | 0.41 > >>>>>>>>> Process 20 Time: 0.61 | 0.76 | 0.76 > >>>>>>>>> Process 30 Time: 0.87 | 1.06 | 1.05 > >>>>>>>>> thread 10 Time: 0.35 | 0.43 | 0.42 > >>>>>>>>> thread 20 Time: 0.66 | 0.79 | 0.78 > >>>>>>>>> Process(Pipe) 10 Time: 0.21 | 0.33 | 0.32 > >>>>>>>>> Process(Pipe) 20 Time: 0.34 | 0.52 | 0.52 > >>>>>>>>> Process(Pipe) 30 Time: 0.46 | 0.72 | 0.71 > >>>>>>>>> thread(Pipe) 10 Time: 0.21 | 0.34 | 0.34 > >>>>>>>>> thread(Pipe) 20 Time: 0.36 | 0.56 | 0.56 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> struct list_head group_node; > >>>>>>>>>> unsigned int on_rq; > >>>>>>>>>> > >>>>>>>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c > >>>>>>>>>> index 093cc1af73dc..752fd364216c 100644 > >>>>>>>>>> --- a/kernel/sched/core.c > >>>>>>>>>> +++ b/kernel/sched/core.c > >>>>>>>>>> @@ -4434,6 +4434,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) > >>>>>>>>>> p->se.nr_migrations = 0; > >>>>>>>>>> p->se.vruntime = 0; > >>>>>>>>>> INIT_LIST_HEAD(&p->se.group_node); > >>>>>>>>>> + RB_CLEAR_NODE(&p->se.latency_node); > >>>>>>>>>> > >>>>>>>>>> #ifdef CONFIG_FAIR_GROUP_SCHED > >>>>>>>>>> p->se.cfs_rq = NULL; > >>>>>>>>>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > >>>>>>>>>> index 125a6ff53378..e2aeb4511686 100644 > >>>>>>>>>> --- a/kernel/sched/fair.c > >>>>>>>>>> +++ b/kernel/sched/fair.c > >>>>>>>>>> @@ -680,7 +680,85 @@ struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq) > >>>>>>>>>> > >>>>>>>>>> return __node_2_se(last); > >>>>>>>>>> } > >>>>>>>>>> +#endif > >>>>>>>>>> > >>>>>>>>>> +/************************************************************** > >>>>>>>>>> + * Scheduling class tree data structure manipulation methods: > >>>>>>>>>> + * for latency > >>>>>>>>>> + */ > >>>>>>>>>> + > >>>>>>>>>> +static inline bool latency_before(struct sched_entity *a, > >>>>>>>>>> + struct sched_entity *b) > >>>>>>>>>> +{ > >>>>>>>>>> + return (s64)(a->vruntime + a->latency_offset - b->vruntime - b->latency_offset) < 0; > >>>>>>>>>> +} > >>>>>>>>>> + > >>>>>>>>>> +#define __latency_node_2_se(node) \ > >>>>>>>>>> + rb_entry((node), struct sched_entity, latency_node) > >>>>>>>>>> + > >>>>>>>>>> +static inline bool __latency_less(struct rb_node *a, const struct rb_node *b) > >>>>>>>>>> +{ > >>>>>>>>>> + return latency_before(__latency_node_2_se(a), __latency_node_2_se(b)); > >>>>>>>>>> +} > >>>>>>>>>> + > >>>>>>>>>> +/* > >>>>>>>>>> + * Enqueue an entity into the latency rb-tree: > >>>>>>>>>> + */ > >>>>>>>>>> +static void __enqueue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) > >>>>>>>>>> +{ > >>>>>>>>>> + > >>>>>>>>>> + /* Only latency sensitive entity can be added to the list */ > >>>>>>>>>> + if (se->latency_offset >= 0) > >>>>>>>>>> + return; > >>>>>>>>>> + > >>>>>>>>>> + if (!RB_EMPTY_NODE(&se->latency_node)) > >>>>>>>>>> + return; > >>>>>>>>>> + > >>>>>>>>>> + /* > >>>>>>>>>> + * The entity is always added the latency list at wakeup. > >>>>>>>>>> + * Then, a not waking up entity that is put back in the list after an > >>>>>>>>>> + * execution time less than sysctl_sched_min_granularity, means that > >>>>>>>>>> + * the entity has been preempted by a higher sched class or an entity > >>>>>>>>>> + * with higher latency constraint. In thi case, the entity is also put > >>>>>>>>>> + * back in the latency list so it gets a chance to run 1st during the > >>>>>>>>>> + * next slice. > >>>>>>>>>> + */ > >>>>>>>>>> + if (!(flags & ENQUEUE_WAKEUP)) { > >>>>>>>>>> + u64 delta_exec = se->sum_exec_runtime - se->prev_sum_exec_runtime; > >>>>>>>>>> + > >>>>>>>>>> + if (delta_exec >= sysctl_sched_min_granularity) > >>>>>>>>>> + return; > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + rb_add_cached(&se->latency_node, &cfs_rq->latency_timeline, __latency_less); > >>>>>>>>>> +} > >>>>>>>>>> + > >>>>>>>>>> +/* > >>>>>>>>>> + * Dequeue an entity from the latency rb-tree and return true if it was really > >>>>>>>>>> + * part of the rb-tree: > >>>>>>>>>> + */ > >>>>>>>>>> +static bool __dequeue_latency(struct cfs_rq *cfs_rq, struct sched_entity *se) > >>>>>>>>>> +{ > >>>>>>>>>> + if (!RB_EMPTY_NODE(&se->latency_node)) { > >>>>>>>>>> + rb_erase_cached(&se->latency_node, &cfs_rq->latency_timeline); > >>>>>>>>>> + RB_CLEAR_NODE(&se->latency_node); > >>>>>>>>>> + return true; > >>>>>>>>>> + } > >>>>>>>>>> + > >>>>>>>>>> + return false; > >>>>>>>>>> +} > >>>>>>>>>> + > >>>>>>>>>> +static struct sched_entity *__pick_first_latency(struct cfs_rq *cfs_rq) > >>>>>>>>>> +{ > >>>>>>>>>> + struct rb_node *left = rb_first_cached(&cfs_rq->latency_timeline); > >>>>>>>>>> + > >>>>>>>>>> + if (!left) > >>>>>>>>>> + return NULL; > >>>>>>>>>> + > >>>>>>>>>> + return __latency_node_2_se(left); > >>>>>>>>>> +} > >>>>>>>>>> + > >>>>>>>>>> +#ifdef CONFIG_SCHED_DEBUG > >>>>>>>>>> /************************************************************** > >>>>>>>>>> * Scheduling class statistics methods: > >>>>>>>>>> */ > >>>>>>>>>> @@ -4758,8 +4836,10 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) > >>>>>>>>>> check_schedstat_required(); > >>>>>>>>>> update_stats_enqueue_fair(cfs_rq, se, flags); > >>>>>>>>>> check_spread(cfs_rq, se); > >>>>>>>>>> - if (!curr) > >>>>>>>>>> + if (!curr) { > >>>>>>>>>> __enqueue_entity(cfs_rq, se); > >>>>>>>>>> + __enqueue_latency(cfs_rq, se, flags); > >>>>>>>>>> + } > >>>>>>>>>> se->on_rq = 1; > >>>>>>>>>> > >>>>>>>>>> if (cfs_rq->nr_running == 1) { > >>>>>>>>>> @@ -4845,8 +4925,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) > >>>>>>>>>> > >>>>>>>>>> clear_buddies(cfs_rq, se); > >>>>>>>>>> > >>>>>>>>>> - if (se != cfs_rq->curr) > >>>>>>>>>> + if (se != cfs_rq->curr) { > >>>>>>>>>> __dequeue_entity(cfs_rq, se); > >>>>>>>>>> + __dequeue_latency(cfs_rq, se); > >>>>>>>>>> + } > >>>>>>>>>> se->on_rq = 0; > >>>>>>>>>> account_entity_dequeue(cfs_rq, se); > >>>>>>>>>> > >>>>>>>>>> @@ -4941,6 +5023,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) > >>>>>>>>>> */ > >>>>>>>>>> update_stats_wait_end_fair(cfs_rq, se); > >>>>>>>>>> __dequeue_entity(cfs_rq, se); > >>>>>>>>>> + __dequeue_latency(cfs_rq, se); > >>>>>>>>>> update_load_avg(cfs_rq, se, UPDATE_TG); > >>>>>>>>>> } > >>>>>>>>>> > >>>>>>>>>> @@ -4979,7 +5062,7 @@ static struct sched_entity * > >>>>>>>>>> pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr) > >>>>>>>>>> { > >>>>>>>>>> struct sched_entity *left = __pick_first_entity(cfs_rq); > >>>>>>>>>> - struct sched_entity *se; > >>>>>>>>>> + struct sched_entity *latency, *se; > >>>>>>>>>> > >>>>>>>>>> /* > >>>>>>>>>> * If curr is set we have to see if its left of the leftmost entity > >>>>>>>>>> @@ -5021,6 +5104,12 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr) > >>>>>>>>>> se = cfs_rq->last; > >>>>>>>>>> } > >>>>>>>>>> > >>>>>>>>>> + /* Check for latency sensitive entity waiting for running */ > >>>>>>>>>> + latency = __pick_first_latency(cfs_rq); > >>>>>>>>>> + if (latency && (latency != se) && > >>>>>>>>>> + wakeup_preempt_entity(latency, se) < 1) > >>>>>>>>>> + se = latency; > >>>>>>>>>> + > >>>>>>>>>> return se; > >>>>>>>>>> } > >>>>>>>>>> > >>>>>>>>>> @@ -5044,6 +5133,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev) > >>>>>>>>>> update_stats_wait_start_fair(cfs_rq, prev); > >>>>>>>>>> /* Put 'current' back into the tree. */ > >>>>>>>>>> __enqueue_entity(cfs_rq, prev); > >>>>>>>>>> + __enqueue_latency(cfs_rq, prev, 0); > >>>>>>>>>> /* in !on_rq case, update occurred at dequeue */ > >>>>>>>>>> update_load_avg(cfs_rq, prev, 0); > >>>>>>>>>> } > >>>>>>>>>> @@ -12222,6 +12312,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first) > >>>>>>>>>> void init_cfs_rq(struct cfs_rq *cfs_rq) > >>>>>>>>>> { > >>>>>>>>>> cfs_rq->tasks_timeline = RB_ROOT_CACHED; > >>>>>>>>>> + cfs_rq->latency_timeline = RB_ROOT_CACHED; > >>>>>>>>>> u64_u32_store(cfs_rq->min_vruntime, (u64)(-(1LL << 20))); > >>>>>>>>>> #ifdef CONFIG_SMP > >>>>>>>>>> raw_spin_lock_init(&cfs_rq->removed.lock); > >>>>>>>>>> @@ -12378,6 +12469,7 @@ void init_tg_cfs_entry(struct task_group *tg, struct cfs_rq *cfs_rq, > >>>>>>>>>> se->my_q = cfs_rq; > >>>>>>>>>> > >>>>>>>>>> se->latency_offset = calc_latency_offset(tg->latency_prio); > >>>>>>>>>> + RB_CLEAR_NODE(&se->latency_node); > >>>>>>>>>> > >>>>>>>>>> /* guarantee group entities always have weight */ > >>>>>>>>>> update_load_set(&se->load, NICE_0_LOAD); > >>>>>>>>>> @@ -12529,8 +12621,19 @@ int sched_group_set_latency(struct task_group *tg, int prio) > >>>>>>>>>> > >>>>>>>>>> for_each_possible_cpu(i) { > >>>>>>>>>> struct sched_entity *se = tg->se[i]; > >>>>>>>>>> + struct rq *rq = cpu_rq(i); > >>>>>>>>>> + struct rq_flags rf; > >>>>>>>>>> + bool queued; > >>>>>>>>>> + > >>>>>>>>>> + rq_lock_irqsave(rq, &rf); > >>>>>>>>>> > >>>>>>>>>> + queued = __dequeue_latency(se->cfs_rq, se); > >>>>>>>>>> WRITE_ONCE(se->latency_offset, latency_offset); > >>>>>>>>>> + if (queued) > >>>>>>>>>> + __enqueue_latency(se->cfs_rq, se, ENQUEUE_WAKEUP); > >>>>>>>>>> + > >>>>>>>>>> + > >>>>>>>>>> + rq_unlock_irqrestore(rq, &rf); > >>>>>>>>>> } > >>>>>>>>>> > >>>>>>>>>> mutex_unlock(&shares_mutex); > >>>>>>>>>> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h > >>>>>>>>>> index 9a2e71231083..21dd309e98a9 100644 > >>>>>>>>>> --- a/kernel/sched/sched.h > >>>>>>>>>> +++ b/kernel/sched/sched.h > >>>>>>>>>> @@ -570,6 +570,7 @@ struct cfs_rq { > >>>>>>>>>> #endif > >>>>>>>>>> > >>>>>>>>>> struct rb_root_cached tasks_timeline; > >>>>>>>>>> + struct rb_root_cached latency_timeline; > >>>>>>>>>> > >>>>>>>>>> /* > >>>>>>>>>> * 'curr' points to currently running entity on this cfs_rq. >
