[+to Nishank] On Thu, Jan 9, 2014 at 2:58 PM, Alex Williamson <alex.williamson@xxxxxxxxxx> wrote: > On Thu, 2014-01-09 at 14:39 -0700, Bjorn Helgaas wrote: >> On Thu, Jan 9, 2014 at 11:25 AM, Alex Williamson >> <alex.williamson@xxxxxxxxxx> wrote: >> > On Thu, 2014-01-09 at 11:08 -0700, Bjorn Helgaas wrote: >> >> On Thu, Jan 9, 2014 at 8:36 AM, Alex Williamson >> >> <alex.williamson@xxxxxxxxxx> wrote: >> >> > Per the SR-IOV spec rev 1.1: >> >> > >> >> > 3.4.1.9 Header Type (Offset 0Eh) >> >> > >> >> > "... For VFs, this register must be RO Zero." >> >> > >> >> > Unfortunately some devices get this wrong, ex. Emulex OneConnect 10Gb >> >> > NIC. When they do it makes us handle ACS testing and therefore IOMMU >> >> > groups as if they were actual multifunction devices and require ACS >> >> > capabilities to make sure there's no peer-to-peer between functions. >> >> > VFs are never traditional multifunction devices, so simply clear this >> >> > bit before we get any further into setup. >> >> >> >> This seems reasonable. Do you have "lspci -vvxxxx" output for this >> >> device? I'd like to save it for future reference. >> > >> > Sure, here's a VF: >> > >> > 09:04.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01) >> > Subsystem: Emulex Corporation Device e722 >> >> Thanks! I put this in >> https://bugzilla.kernel.org/show_bug.cgi?id=68431, and I'll add a >> reference to the changelog. >> >> But I wonder if we can make this slightly more generic by doing >> something like this in pci_setup_device(): >> >> dev->multifunction = (PCI_FUNC(dev->devfn) == 0) && (hdr_type & 0x80); >> >> That's basically what lspci does in pci_generic_scan_bus(), and >> section 3.2.2.3.4 of the PCI 3.0 spec sort of implies that we should >> only look at the bit 7 of the header type for function 0: >> >> If a single function device is detected (i.e., bit 7 in the Header >> Type register of function 0 is 0), no more functions for that >> Device Number will be checked. If a multi-function device is >> detected (i.e., bit 7 in the Header Type register of function 0 >> is 1), then all remaining Function Numbers will be checked. > > We could do that and rely only on pci_scan_slot() to set multifunction=1 > for the other functions, but that doesn't completely solve this problem. > VFs can occupy function zero and the example device would still set > multifunction with that test. Thanks, Duh, it would help if I actually paid attention to your lspci output... The reason I'm thinking about this is that virtfn_add() is only used when we enable SR-IOV. If we clear dev->multifunction there, we only end up with the correct value if we start with SR-IOV disabled, and then enable it. If SR-IOV were enabled by the firmware before Linux boots, we wouldn't go through the virtfn_add() path, and dev->multifunction might still be wrong. I'm pretty sure Nishank said there were Cisco boxes that enable SR-IOV in the firmware, but I don't know how that works. It looks like we would disable SR-IOV during enumeration in the path below: pci_scan_slot pci_scan_single_device pci_device_add pci_init_capabilities pci_iov_init sriov_init pci_read_config_word(dev, pos + PCI_SRIOV_CTRL, &ctrl) if (ctrl & PCI_SRIOV_CTRL_VFE) pci_write_config_word(dev, pos + PCI_SRIOV_CTRL, 0) >From that path, it *looks* like it doesn't really matter whether SR-IOV is enabled at handoff, because we disable it anyway. So I'm not sure if I misunderstood Nishank or what. I think it would be cool if we could enumerate previously-enabled VFs, but maybe there are other issues that would make that impossible. Bjorn -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
