On Tue, 2008-09-09 at 21:12 +0000, I wrote:
> The kdump kernel wouldn't be in danger of being overwritten, it just
> might not be able to set up any IOs that work to its own address space
> if an IOMMU is out there waiting to grab them.
>
> For the calgary case, we'd maybe have to add the Crash Kernel range to
> the list of things sent to iommu_range_reserve in
> calgary_reserve_regions, to prevent those addresses from ever being
> given out.
>
> Bob M.
While this, or something like it, is necessary, it isn't sufficient.
I think what we would really need to do is to have the primary kernel
set up an identity mapping for all pages in the Crash kernel range,
or the subset of pages that could be IO targets when the kdump kernel
is running. This would allow a still-running IOMMU from the primary
kernel to translate kdump IOs to kdump addresses transparently.
And that leads to the Kdump IO Rule:
The primary kernel is responsible for setting up any necessary
conditions to allow the kdump kernel to perform its required
IO without detecting any iommu.
The kdump kernel must refrain from detecting and initializing
any iommu.
This has a these effects:
A) Primary kernel: depending on what it is using for as an IOMMU,
it may have to do some (or considerable) setup, to guarantee
that the kdump kernel can have IO capability to its Crash
kernel address range.
B) Primary kernel: the Crash kernel range must be set up in an address
range whose physical addresses are accessible to IO cards
without address remapping.
C) Kdump kernel: the kdump kernel must ignore any IOMMU hardware that
might be "detectable".
The setup responsibilities for the primary kernel depend on what it is
currently using for dma mapping:
1) no iommu (nommu_map_single): No setup is required for kdump.
Leftover IOs will go to IO buffers allocated by the primary
kernel outside of the Crash kernel area. Kdump IOs will
go to IO buffers allocated by the kdump kernel in the Crash
kernel area.
2) swiotlb (swiotlb_map_single_phys): No setup is required for kdump.
Leftover IOs will go to the primary kernel bounce buffers
outside of the Crash kernel area. Kdump IOs will go to IO
buffers allocated by the kdump kernel in the Crash kernel area.
3) GART (gart_map_single): No setup is required for kdump. Leftover
IOs will be mapped through the GART aperture to IO buffers
allocated by the primary kernel outside of the Crash Kernel
area. Kdump IOs will go to IO buffers allocated by the kdump
kernel in the Crash kernel area.
4) Calgary IOMMU (calgary_map_single): The Crash kernel memory range
must be pre-allocated for IO and identity-mapped, so any IO
operation to an address in the Crash kernel range is allowed
to complete to that same address. To preallocate for a
128MB Crash kernel area, 32K entries (256 Kbytes) are used
from the Calgary table. For a 4GB system, the default size
of the table is 1024K entries (8 Mbytes).
Leftover IOs will go to IO buffers allocated by the primary
kernel and remapped by the Calgary IOMMU. Neither the IO-side
address (iova) nor the physical address of a leftover IO will
be in the Crash kernel area. Kdump IOs will go to IO buffers
allocated by the kdump kernel, remapped by the Calgary IOMMU
to those same addresses (iova equals physical address within
the Crash kernel area).
5) Intel IOMMU (intel_map_single): The Crash kernel memory range must
be pre-allocated and identity-mapped for each hw device that
is needed by the kdump kernel, so any IO operation to an
address in the Crash kernel range is allowed to complete to
that same address.
Leftover IOs will go to IO buffers allocated by the primary
kernel and remapped by the Intel IOMMU. Neither the IO-side
address (iova) nor the physical address of a leftover IO
will be in the Crash kernel area. Kdump IOs will go to IO
buffers allocated by the kdump kernel, remapped by the Intel
IOMMU to those same addresses (iova equals physical address
within the Crash kernel area).
This all assumes no virtual machine stuff yet.
Possible? Comments? Corrections?
Bob Montgomery
