While doing performance analysis of a new transport protocol for
Linux, I am seeing mysterious gaps where *all* cores in an Intel CPU
(Broadwell, E52640v4) stop processing simultaneously for 200-300
microseconds. I'm writing in the hopes that someone on this list might
know what is causing this. Here are some additional details:
* What's particularly surprising to me is that every core on the
processor seems to pause at exactly the same time.
* I instrumented every interrupt entry point ("__irq_entry") that I
could find in the kernel sources, and none of these are triggering
during the gaps.
* I also instrumented context switches, so I know that those aren't
occurring during the gaps.
* The gaps happen in different places on different cores; in some
cases they occur in the middle of straight-line code; in other cases
they delay the occurrence of interrupts.
* I have disabled c-states when running the tests (but c-states
wouldn't explain gaps in the middle of straight-line code anyway).
* I don't think this is just a Broadwell issue. In previous work we
noticed unexplainable hundred-microsecond gaps in applications running
on different Intel processors; we were never able to find a reason for
these, but I suspect that the same issue might have caused those gaps
as well.
If anyone has any ideas, or knows of places where I could go to get
more information, I'd appreciate it.
-John-
