On Mon, Jul 13, 2015 at 2:01 PM, Chris Metcalf <cmetcalf@xxxxxxxxxx> wrote: > On 07/13/2015 04:40 PM, Andy Lutomirski wrote: >> >> On Mon, Jul 13, 2015 at 12:57 PM, Chris Metcalf <cmetcalf@xxxxxxxxxx> >> wrote: >>> >>> The existing nohz_full mode makes tradeoffs to minimize userspace >>> interruptions while still attempting to avoid overheads in the >>> kernel entry/exit path, to provide 100% kernel semantics, etc. >>> >>> However, some applications require a stronger commitment from the >>> kernel to avoid interruptions, in particular userspace device >>> driver style applications, such as high-speed networking code. >>> >>> This change introduces a framework to allow applications to elect >>> to have the stronger semantics as needed, specifying >>> prctl(PR_SET_CPU_ISOLATED, PR_CPU_ISOLATED_ENABLE) to do so. >>> Subsequent commits will add additional flags and additional >>> semantics. >> >> I thought the general consensus was that this should be the default >> behavior and that any associated bugs should be fixed. > > > I think it comes down to dividing the set of use cases in two: > > - "Regular" nohz_full, as used to improve performance and limit > interruptions, possibly for power benefits, etc. But, stray > interrupts are not particularly bad, and you don't want to take > extreme measures to avoid them. > > - What I'm calling "cpu_isolated" mode where when you return to > userspace, you expect that by God, the kernel doesn't interrupt you > again, and if it does, it's a flat-out bug. > > There are a few things that cpu_isolated mode currently does to > accomplish its goals that are pretty heavy-weight: > > Processes are held in kernel space until ticks are quiesced; this is > not necessarily what every nohz_full task wants. If a task makes a > kernel call, there may well be arbitrary timer fallout, and having a > way to select whether or not you are willing to take a timer tick after > return to userspace is pretty important. Then shouldn't deferred work be done immediately in nohz_full mode regardless? What is this delayed work that's being done? > > Likewise, there are things that you may want to do on return to > userspace that are designed to prevent further interruptions in > cpu_isolated mode, even at a possible future performance cost if and > when you return to the kernel, such as flushing the per-cpu free page > list so that you won't be interrupted by an IPI to flush it later. > Why not just kick the per-cpu free page over to whatever cpu is monitoring your RCU state, etc? That should be very quick. > If you're arguing that the cpu_isolated semantic is really the only > one that makes sense for nohz_full, my sense is that it might be > surprising to many of the folks who do nohz_full work. But, I'm happy > to be wrong on this point, and maybe all the nohz_full community is > interested in making the same tradeoffs for nohz_full generally that > I've proposed in this patch series just for cpu_isolated? nohz_full is currently dog slow for no particularly good reasons. I suspect that the interrupts you're seeing are also there for no particularly good reasons as well. Let's fix them instead of adding new ABIs to work around them. --Andy -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
