Sagi,
Sorry it took a while to bring my system back online.
With the patch, the IOPS is about the same drop with the 1st patch. I think
the excessive context switches are causing the drop in IOPS.
The following are captured by "perf sched record" for 30 seconds during
tests.
"perf sched latency"
With patch:
fio:(82) | 937632.706 ms | 1782255 | avg: 0.209 ms | max: 63.123
ms | max at: 768.274023 s
without patch:
fio:(82) |2348323.432 ms | 18848 | avg: 0.295 ms | max: 28.446
ms | max at: 6447.310255 s
Without patch means the proposed hard-irq patch?
It means the current upstream code without any patch. But It's prone to soft lockup.
Ming's proposed hard-irq patch gets similar results to "without patch", however it fixes the soft lockup.
Thanks for the clarification.
The problem with what Ming is proposing in my mind (and its an existing
problem that exists today), is that nvme is taking precedence over
anything else until it absolutely cannot hog the cpu in hardirq.
In the thread Ming referenced a case where today if the cpu core has a
net softirq activity it cannot make forward progress. So with Ming's
suggestion, net softirq will eventually make progress, but it creates an
inherent fairness issue. Who said that nvme completions should come
faster then the net rx/tx or another I/O device (or hrtimers or sched
events...)?
As much as I'd like nvme to complete as soon as possible, I might have
other activities in the system that are as important if not more. So
I don't think we can solve this with something that is not cooperative
or fair with the rest of the system.
If we are context switching too much, it means the soft-irq operation is not
efficient, not necessarily the fact that the completion path is running in soft-
irq..
Is your kernel compiled with full preemption or voluntary preemption?
The tests are based on Ubuntu 18.04 kernel configuration. Here are the parameters:
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
I see, so it still seems that irq_poll_softirq is still not efficient in
reaping completions. reaping the completions on its own is pretty much
the same in hard and soft irq, so its really the scheduling part that is
creating the overhead (which does not exist in hard irq).
Question:
when you test with without the patch (completions are coming in
hard-irq), do the fio threads that run on the cpu cores that are
assigned to the cores that are handling interrupts get substantially lower
throughput than the rest of the fio threads? I would expect that
the fio threads that are running on the first 32 cores to get very low
iops (overpowered by the nvme interrupts) and the rest doing much more
given that nvme has almost no limits to how much time it can spend on
processing completions.
If need_resched() is causing us to context switch too aggressively, does
changing that to local_softirq_pending() make things better?
--
diff --git a/lib/irq_poll.c b/lib/irq_poll.c
index d8eab563fa77..05d524fcaf04 100644
--- a/lib/irq_poll.c
+++ b/lib/irq_poll.c
@@ -116,7 +116,7 @@ static void __latent_entropy irq_poll_softirq(struct
softirq_action *h)
/*
* If softirq window is exhausted then punt.
*/
- if (need_resched())
+ if (local_softirq_pending())
break;
}
--
Although, this can potentially cause other threads from making forward
progress.. If it is better, perhaps we also need a time limit as well.
Perhaps we should add statistics/tracing on how many completions we are
reaping per invocation...
