On Mon, Jan 07, 2019 at 09:01:29PM -0700, Jason Gunthorpe wrote: > On Sun, Jan 06, 2019 at 09:43:46AM +1100, Benjamin Herrenschmidt wrote: > > On Sat, 2019-01-05 at 10:51 -0700, Jason Gunthorpe wrote: > > > > > > > Interesting. I've investigated this further, though I don't have as > > > > many new clues as I'd like. The problem occurs reliably, at least on > > > > one particular type of machine (a POWER8 "Garrison" with ConnectX-4). > > > > I don't yet know if it occurs with other machines, I'm having trouble > > > > getting access to other machines with a suitable card. I didn't > > > > manage to reproduce it on a different POWER8 machine with a > > > > ConnectX-5, but I don't know if it's the difference in machine or > > > > difference in card revision that's important. > > > > > > Make sure the card has the latest firmware is always good advice.. > > > > > > > So possibilities that occur to me: > > > > * It's something specific about how the vfio-pci driver uses D3 > > > > state - have you tried rebinding your device to vfio-pci? > > > > * It's something specific about POWER, either the kernel or the PCI > > > > bridge hardware > > > > * It's something specific about this particular type of machine > > > > > > Does the EEH indicate what happend to actually trigger it? > > > > In a very cryptic way that requires manual parsing using non-public > > docs sadly but yes. From the look of it, it's a completion timeout. > > > > Looks to me like we don't get a response to a config space access > > during the change of D state. I don't know if it's the write of the D3 > > state itself or the read back though (it's probably detected on the > > read back or a subsequent read, but that doesn't tell me which specific > > one failed). > > If it is just one card doing it (again, check you have latest > firmware) I wonder if it is a sketchy PCI-E electrical link that is > causing a long re-training cycle? Can you tell if the PCI-E link is > permanently gone or does it eventually return? > > Does the card work in Gen 3 when it starts? Is there any indication of > PCI-E link errors? > > Everytime or sometimes? > > POWER 8 firmware is good? If the link does eventually come back, is > the POWER8's D3 resumption timeout long enough? > > If this doesn't lead to an obvious conclusion you'll probably need to > connect to IBM's Mellanox support team to get more information from > the card side. +1, I tried to find any Mellanox-internal bugs related to your issue and didn't find anything concrete. Thanks > > Jason
Attachment:
signature.asc
Description: PGP signature
