On Fri, 2008-09-05 at 15:12 +0000, Vivek Goyal wrote: > Nice summary Bob. Few thoughts. > > - So until and unless one is reserving memory for crashkernel above 4G, > there is no need for initializing the IOMMU in second kernel (At this > moment I am not too worried about need of isolation in second kernel). If > that's the case, we shouldn't have initialized the calgary iommu in second > kernel and just should have left it alone and things probably would have > been fine? Did you ever try booting the kdump kernel with iommu=off? That should have prevented the detection and initialization of the calgary iommu from the kdump kernel, which (I ass-u-me) is what you had problems with, if leftover IOs from the crashed kernel were still in progress when the kdump kernel initialized it? As for 4G, the info in 2.6.26 Documentation/x86_64/boot-options.txt mentions 3G as the decision for using an iommu in several places. The test in our kernel for using the GART IOMMU is (end_pfn > MAX_DMA32_PFN) where MAX_DMA32_PFN is: #define MAX_DMA32_PFN ((4UL*1024*1024*1024) >> PAGE_SHIFT) > > The only issue is that how do you make sure that first kernel has not > setup an IOMMU entry with same bus address which falls in crash kernel > reserved area. I am not very familiar with the dma/iommu code and how > bus addresses are selected. Because if there is bus address overlap in > first kernel and second kernel, IOMMU will trap the second kernel's DMA > attempts and redirect it somewhere else. If we don't run into this issue > then it is fine otherwise we will be forced to use IOMMU in second kernel > and try to find free bus addresses/entries so that we don't conflict with > the first kernel's settings. I finally thought about this long enough to figure out what you meant, I think :-). If the existence of a real IOMMU in the first kernel allows the use of completely virtualized addresses on the IO side, then there's no reason that they would have to avoid real RAM addresses when handing out addresses to IO cards with that IOMMU. In particular, our little joke about allocating the crash kernel under the GART aperture (with CPU-side access turned off) would prevent the kdump kernel from doing non-iommu IO to the crash kernel address range, because the still-active GART from the old kernel would grab any of those addresses coming in from the IO side. The kdump kernel wouldn't be in danger of being overwritten, it just might not be able to set up any IOs that work to its own address space if an IOMMU is out there waiting to grab them. For the calgary case, we'd maybe have to add the Crash Kernel range to the list of things sent to iommu_range_reserve in calgary_reserve_regions, to prevent those addresses from ever being given out. Bob M.
