Hi David,
Let me provide a short history. Last year we installed a system at a
large university research center. It has 1024 cpus and 44 I/O devices.
As it was an early system many problems were found and fixed.
When it got to the GPU support, those devices only support 40 bits of
addressing whereas the system needed 44. So we turned on the IOMMU
support with "intel_iommu=on,forcedac" which was the only method that
worked.
Unfortunately, the 10GigE interfaces slowed down to about 10 - 15% of
line speed. One solution that helped tremendously was to enable the
"pass through" mode, except then the system panic'd. All of these fixes
in this patchset address that performance loss and the system panic.
The patches are critical to us because more customers are encountering
the same circumstances. Since our distros are still based on 2.6.32,
that's where we need acceptance of the patches. The upstream kernel is
only important to the distros when they decide to release their next
version.
You're probably aware of the cycles Chris Wright and Mike Habeck and I
went through to clean up these patches. I believe they are as good as
they can be.
The last patch is important as these types of systems require
"system management" controllers and since we want to set the options
"intel_iommu=on iommu=pt", we can only do that for kernels that have
these fixes. The special symbol (which the memory is released since
it's in the "__init" section) triggers that mgmt s/w to set those
options. If the kernel does not have the fixes, then it must use the
older "on,forcedac" options or the kernel will panic on bootup.
If there is a better way, please let me know. The problem I found
is the SMC needs the info before booting the kernel, and using a
specially named symbol was the best I could think of.
Thanks for looking into this. Please let me know when you have accepted
them and I will then push them to the 2.6.32 stable tree, and then on
to the distros.
Cheers,
Mike
David Woodhouse wrote:
On Thu, 2011-05-26 at 20:32 -0500, Mike Travis wrote:
Prior to these IOMMU patches, the passthrough option
could not be used because it would cause the kernel to
panic.
Could not be used on certain hardware? In certain configurations? It
wasn't just *completely* broken (unless it got broken in a recent
regression).
Provide an indication that a kernel is capable
of handling passthrough mode through the introduction
of a specific variable name. This allows automatic
configuration utilities to set the "iommu=pt" kernel
cmdline option safely.
That implies that there is a follow-up patch to somehow export this
information to userspace? Where the information in question is just the
fact that a certain set of bugs have been fixed?
Since these *are* actually bugs, and not new features, I take it we're
looking at merging them for 2.6.40? I was planning to ask Linus to pull
my tree today, and I'm not entirely happy with merging them at the last
minute.
So I'll plan to merge them into my tree after the outstanding merge, and
then perhaps ask Linus to take them *after* -rc1? Since they're actually
bug fixes, I suspect that's the best way to proceed?
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
