On Tue, 2021-10-05 at 21:33 -0500, Bjorn Helgaas wrote: > On Wed, Oct 06, 2021 at 12:37:02AM +0000, Kelvin.Cao@xxxxxxxxxxxxx > wrote: > > On Tue, 2021-10-05 at 15:11 -0500, Bjorn Helgaas wrote: > > > On Mon, Oct 04, 2021 at 08:51:06PM +0000, > > > Kelvin.Cao@xxxxxxxxxxxxx > > > wrote: > > > > On Sat, 2021-10-02 at 10:11 -0500, Bjorn Helgaas wrote: > > > > > I *thought* the problem was that the PCIe Memory Read failed > > > > > and the Root Complex fabricated ~0 data to complete the CPU > > > > > read. But now I'm not sure, because it sounds like it might > > > > > be that the PCIe transaction succeeds, but it reads data that > > > > > hasn't been updated by the firmware, i.e., it reads 'in > > > > > progress' because firmware hasn't updated it to 'done'. > > > > > > > > The original message was sort of misleading. After a firmware > > > > reset, CPU getting ~0 for the PCIe Memory Read doesn't explain > > > > the hang. In a MRPC execution (DMA MRPC mode), the MRPC status > > > > which is located in the host memory, gets initialized by the > > > > CPU > > > > and updated/finalized by the firmware. In the situation of a > > > > firmware reset, any MRPC initiated afterwards will not get the > > > > status updated by the firmware per the reason you pointed out > > > > above (or similar, to my understanding, firmware can no longer > > > > DMA data to host memory in such cases), therefore the MRPC > > > > execution will never end. > > > > > > I'm glad this makes sense to you, because it still doesn't to me. > > > > > > check_access() does an MMIO read to something in BAR0. If that > > > read returns ~0, it means either the PCIe Memory Read was > > > successful and the Switchtec device supplied ~0 data (maybe > > > because firmware has not initialized that part of the BAR) or the > > > PCIe Memory Read failed and the root complex fabricated the ~0 > > > data. > > > > > > I'd like to know which one is happening so we can clarify the > > > commit log text about "MRPC command executions hang indefinitely" > > > and "host wil fail all GAS reads." It's not clear whether these > > > are PCIe protocol issues or driver/firmware interaction issues. > > > > I think it's the latter case, the ~0 data was fabricated by the > > root > > complex, as the MMIO read in check_access() always returns ~0 until > > a reboot or a rescan happens. > > If the root complex fabricates ~0, that means a PCIe transaction > failed, i.e., the device didn't respond. Rescan only does config > reads and writes. Why should that cause the PCIe transactions to > magically start working? I took a closer look. What I observed was like this. A firmware reset cleared some CSR settings including the MSE and MBE bits and the Base Address Registers. With a rescan (removing the switch to which the management EP was binded from root port and rescan), the management EP was re-enumerated and driver was re-probed, so that the settings cleared by the firmware reset was properly setup again, therefore PCIe transactions start working. Kelvin
