On 10/09/2012 05:03 AM, Takao Indoh wrote: > (2012/10/03 22:23), Don Dutile wrote: >> On 10/02/2012 03:49 AM, Takao Indoh wrote: >>> These patches reset PCIe devices at boot time to address DMA problem on >>> kdump with iommu. When "reset_devices" is specified, a hot reset is >>> triggered on each PCIe root port and downstream port to reset its >>> downstream endpoint. >>> >>> Background: >>> A kdump problem about DMA has been discussed for a long time. That is, >>> when a kernel is switched to the kdump kernel DMA derived from first >>> kernel affects second kernel. Recently this problem surfaces when iommu >>> is used for PCI passthrough on KVM guest. In the case of the machine I >>> use, when intel_iommu=on is specified, DMAR error is detected in kdump >>> kernel and PCI SERR is also detected. Finally kdump fails because some >>> devices does not work correctly. >>> >>> The root cause is that ongoing DMA from first kernel causes DMAR fault >>> because page table of DMAR is initialized while kdump kernel is booting >>> up. Therefore to address this problem DMA needs to be stopped before >>> DMAR is initialized at kdump kernel boot time. By these patches, PCIe >>> devices are reset by hot reset and its DMA is stopped when reset_devices >>> is specified. One problem of this solution is that the monitor blacks >>> out when VGA controller is reset. So this patch does not reset the port >>> whose child endpoint is VGA device. >>> >>> v2: >>> Reset devices in setup_arch() because reset need to be done before >>> interrupt remapping is initialized. >>> >>> v1: >>> https://lkml.org/lkml/2012/8/3/160 >>> >>> Thanks, >>> Takao Indoh >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-pci" in >>> the body of a message to majordomo at vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> >> Maybe you've tried the following, and I missed a thread on it, >> but instead of a somewhat-large, reset hammer, did any one try >> just reading all the endpoint-only device CMD register, flip the MasterEnable >> bit off, and write it back? .... that would stop all DMA (should stop >> all MSI writes as well since they are just another DMA), and then >> restart the system? >> May also have to do PCI INT Disable as well... and note, that's a PCI 2.3 >> optional feature.... so devices using INT signalling vs MSI >> is just borked on IOMMU/intr-remapping systems... which I would expect >> are few. Then again, if this is foolishly done, then reset legacy PCI >> busses as the fallback. > > Just clearing bus master bit and INTx disable bit in setup_arch() did > not solve this problem. I still got DMAR error on devices(for exmaple, > igb and megaraid_sas). > > Clearing bus master in setup_arch() and resetting devices in fixup_final > like v1 patch is better, DMAR error was not detected. But on certain > machine kdump kernel hung up when resetting devices. It seems to be a > problem specific to the platform. > > And, resetting devices in setup_arch() like v2 patch solves all problems > I found so far. > > Thanks, > Takao Indoh > this summary should be in the patch set, so others know/learn what was attempted, what failed, and how you reached this working conclusion.
