On Tue, 2014-06-24 at 01:44 +0300, Yishai Hadas wrote: > On 6/23/2014 11:12 PM, Don Dutile wrote: > > On 06/23/2014 03:09 PM, Bjorn Helgaas wrote: > >> [+cc linux-pci, Don] > >> > > Adding Alex Williamson in case he can add more to this conversation... > > > >> On Mon, Jun 23, 2014 at 8:29 AM, Yishai Hadas > >> <yishaih@xxxxxxxxxxxxxxxxxx> wrote: > >>> Hi Vijay, > >>> Trying to add AER support for Mellanox NIC in SRIOV environment, while > >>> evaluating/testing encountered a problem which led me to your > >>> patch accepted as part of kernel 3.8, commit ID > >>> "918b4053184c0ca22236e70e299c5343eea35304". > >>> > >>> Have some concerns/questions on: > >>> When working in SRIOV environment VFs may be un-attached, having no > >>> driver > >>> assigned to, or may be attached to Virtual machine to work in some > >>> pass-through mode. > >>> Once working in KVM setup there is pci-stub driver which is loaded > >>> in the > >>> HYP/PF for a given attached VF. > > huh? 'loaded in the hyp/pf? .... um, loaded in the host, and a VF is > > detached from its host driver -- a VF can be used in the host w/o any > > virtualization, > > i.e., that's how guest VM is driving the VF: as if it was used by a > > guest (host) OS directly -- > > and attached to pci-stub driver, when assigned to a KVM guest in > > pre-VFIO days/ways. > > If VFIO used, then VF is attached to vfio-pci driver. > > > >>> > >>> I'm using the aer-inject kernel module and its corresponding > >>> aer-inject tool > >>> to simulate an error in the HYP. > >>> In both cases your commit will cause the AER recovery to fail as > >>> there is no > >>> driver assigned to PF's VFs that supports AER, comparing the code > >>> before > >>> your change. > >>> > > Without VFIO, I believe that's correct. There was no AER-to-VF support > > pre-VFIO days. > > I believe with the recent VFIO support, > > and modifications to KVM, an AER that is associated with an assigned > > VF will > > force the crash/halt of the KVM guest -- can't depend on a guest VF > > driver clearing > > the AER in the hyp/host -- guest isn't privileged enough to clear the > > error. > > So, crashing the guest is the simple option at the moment, to contain > > the error. > > Alex: do I have that (vfio aer default) correct, or is that still > > site-under-construction? > How about the case that the VF is not attached to a KVM guest and > has no driver loaded on host ? in such a case from code review and some > testing the recovery will > fail as there is no AER aware driver here. What is the expected > solution here ? > Any special qemu /stuff is needed to activate the VFIO support ? > would like to give it a try for a case that VF is attached. Just use recent QEMU (>=1.6, the newer the better) and it should be automatic. Note that the VM won't exit on error, it's stopped with state RUN_STATE_IO_ERROR to allow the possibility of collecting data. Thanks, Alex > > > >>> How such cases should work ? my expectation was that the PF will > >>> get the > >>> error detected message then will recognize whether > >>> issue is its own or one of its VFs > > The AER packet will have the tag of the VF in if it was the source of > > the error; > > so the PF will never see it; although one could argue it should be > > 'promoted' > > to the PF if PF/VF needs to clear some state it has wrt the VF (the > > SRIOV spec is > > lacking of info in this space); _but_, VFIO resets the VF (sets FLR > > bit) when the > > device is deassigned and before re-attachment to the host, so that > > should clear out > > any state btwn PF & VF ('should' ... famous last words...). > In my test I have used the aer-inject tool simulating an error to > the BUS that both PF/VF are residing on, putting the function number to > be the PF one, looks like both should be called by the aer driver as part > of the pci_walk_bus(). As mentioned I got a call only on the PF and > recovery failed as of the VF doesn't include an AER aware driver, once > removed the VF recovery succeeded. > I believe that packet should include some info about the source of > the error isn't it ? > In addition, looking at IXGBE upstream source code at > ixgbe_error_detected() looks like there is some code running on the PF > that checks whether the source was a VF. > > By the way: when tried to simulate a VF error using its FN got > below error: > "Error: Failed to write, Inappropriate ioctl for device", any idea > about that error ? > > > >> > >> I'm really not an AER expert, so help me understand this question of > >> recognizing whether an error is associated with a PF or a VF. > >> > >> In terms of hardware, it looks like the device that detects an error > >> logs some information and sends an Error Message upstream. The Root > >> Complex receives the message, captures the source ID from the Error > >> Message, and may generate an interrupt. I expect this source ID can > >> be either a PF or a VF; there's no requirement that a VF error must be > >> reported as though it's from the PF, is there? > >> > >>> and work accordingly, in current code > >>> looks like recovery failed as part of "voting" once there is no AER > >>> handler > >>> assigned to the VFs. > >> > >> The commit you mentioned has to do with PCI_ERS_RESULT_NO_AER_DRIVER. > >> We use pci_walk_bus() to figure out whether all the devices in a > >> subtree have a driver. What subtree is involved here? I would expect > >> the VFs to be siblings of the PF, not children of it, so I'm not sure > >> where things went wrong. > > Well, VFs could be on virtual busses (ARI turned on), so not > > necessarily a > > sibling to PF ... and then we have the problem in PCI code of not > > being able > > to traverse these virtual busses (in some cases; not sure if > > pci_walk_bus(), > > which is going down the tree vs up the tree, has any problems here > > w/VFs on > > virtual busses). > > > >> > >> Can you collect "lspci -vvv" output and maybe add some debug so we can > >> see exactly where the error is detected and what devices we're looking > >> at to conclude that one of them doesn't have a driver? > lspci -vvv for both PF & VF is attached, we can see that VF > (21:00.1) has no driver loaded comparing the PF (Kernel driver in use: > mlx4_core). > >> > >> Bjorn > >> > > > -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html