> -----Original Message----- > From: linux-pci-owner at vger.kernel.org > [mailto:linux-pci-owner at vger.kernel.org] On Behalf Of Takao Indoh > Sent: Wednesday, September 05, 2012 8:10 PM > To: vgoyal at redhat.com > Cc: kexec at lists.infradead.org; linux-kernel at vger.kernel.org; > linux-pci at vger.kernel.org; bhelgaas at google.com; hbabu at us.ibm.com; Ishii, > Hironobu/?? ??; martin.wilck at ts.fujitsu.com > Subject: Re: [RFC][PATCH] Reset PCIe devices to address DMA problem on kdump > with iommu > > (2012/08/07 5:39), Vivek Goyal wrote: > > On Mon, Aug 06, 2012 at 01:30:47PM +0900, Takao Indoh wrote: > >> Hi Vivek, > >> > >> (2012/08/03 20:46), Vivek Goyal wrote: > >>> On Fri, Aug 03, 2012 at 08:24:31PM +0900, Takao Indoh wrote: > >>>> Hi all, > >>>> > >>>> This patch adds kernel parameter "reset_pcie_devices" which resets > PCIe > >>>> devices at boot time to address DMA problem on kdump with iommu. When > >>>> this parameter is specified, a hot reset is triggered on each PCIe > root > >>>> port and downstream port to reset its downstream endpoint. > >>> > >>> Hi Takao, > >>> > >>> Why not use existing "reset_devices" parameter instead of introducing > >>> a new one? > >> > >> "reset_devices" is used for each driver to reset their own device, and > >> this patch resets all devices forcibly, so I thought they were different > >> things. > > > > Yes reset_devices currently is used for driver to reset its device. I > > thought one could very well extend its reach to reset pci express devices > > at bus level. > > > > Having them separate is not going to be much useful from kdump > > perspective. We will end up passing both reset_devices and > > reset_pcie_devices to second kernel whill lead to bus level reset as well > > as device level reset. > > > > Ideal situation would be that somehow detect that bus level reset has > been > > done and skip device level reset (assuming bus level reset obviates the > > need of device level reset, please correct me if that's not the case). > > > > After pcie reset, can we store the state in a variable and drivers can > > use that variable to check if PCIe level reset was done or not. If yes, > > skip device level reset (Assuming driver knows that device is on a > > PCIe slot). > > > > In that case we will not have to introduce new kernel command line, and > > also avoid double reset? > > I found a problem when testing my patch on some machines. > > Originally there are two problems in kdump kernel when iommu is enabled; > DMAR error and PCI SERR. I thought they are fixed by my patch, but I > noticed that PCI SERR is still detected after applying the patch. It > seems that something happens when Interrupt Remapping is initialized in > kdump kernel. I'm not sure, but I guess the PCI SERR might be caused as follows. - The 1st kernel enables interrupt remapping. The MSI(-X) address and data registers of PCI devices are programmed in remappable format. - At the beginning of 2nd kernel, interrupt remapping is still active. And then it is disabled by enable_IR() function for initialization. - PCI device generate an interrupt. At this moment, interrupt remapping is not enabled yet. On the other hand, MSI(-X) address and data of this interrupt is in remappable format because those are programmed by 1st kernel. I guess this might be a cause of PCI SERR. I guess clearing command register or disabling MSI before interrupt remapping initialization in 2nd kernel might solve the PCI SERR problem. Regards, Kenji Kaneshige > > Therefore resetting devices has to be done before enable_IR() is > called. I have three ideas for it. > > (i) Resetting devices in 1st kernel(panic kernel) > We can reset devices before jumping into 2nd kernel. Of course it may > be dangerous to scan pci device tree and call PCI functions in panic'd > kernel. Beforehand we need to collect device information so that only > minimal code could run on panic. > > (ii) Resetting devices in purgatory > It seems to be be appropriate place to do this, but I'm not sure > where I can save/restore PCI config when resetting devices in > purgatory. > > (iii) Resetting devices in 2nd kernel(kdump kernel) > Important point is to do reset before enable_IR() is called as I wrote > above. I think I should add new function to do reset into > arch/x86/pci/early.c and call it in setup_arch like > early_dump_pci_devices() or early_quirks(). > > Any comments? > > Thanks, > Takao Indoh > > -- > To unsubscribe from this list: send the line "unsubscribe linux-pci" in > the body of a message to majordomo at vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html
