Joyce Yu - System Software wrote:
Hello,
I am working on a SR-IOV capable network device. It can be configured in
two ways. One way is that each device node can have four pci functions
in pci function mode. Another way is that each device node can have one
PF and three VFs in the SR-IOV mode. There are total four of these
devices in my setup (maximal configuration of the device). So there are
total 16 pci functions in the pci function mode, and total 4 PFs and 12
VFs in the SR-IOV mode. I have no problem in the pci function mode, but
got "PCI: mem resource #9:300000@f9d00000 for 0000:16:00.0 was not
allocated" in the SR-IOV mode. The requested resource are all the same
for each pci function or PF/VF. The resource of each pci function/PF/VF
= (msix 0x3fff + pio 0xfffff), so total 16 of them = (msix 0x3fff + pio
0xfffff) * 16 = 23.5 m. It is not a lot. Does it exceed max resource
allocation? Is there a tunable in the Linux kernel that can give more
resource to one kind of device? Any idea how to deal with it?
Thanks in advance,
Joyce
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
For system tools that want to be able to balance network utilization,
it's important to know whether there are 4 separate, physical network connections,
or is there only 1 network connection per four-fcn device.
If 4 devices are using a single, physical network connection, then
SR-IOV should be used.
If there are 4 separate, physical network connections, then 4 (std) PCI functions
should be used.
In SRIOV mode, you are seeing the resource problem b/c the platform BIOS
is not reserving enough PCI bridge (memory-mapped IO) space for the VF devices.
VF device space needs are determined by scanning extended PCI cfg space,
specifically looking for SRIOV defined structs. Many BIOS dont have this
extended resource reservation algs to make SRIOV resource allocation the same
as std PCI fcn resource allocation.
Since PCI bridges must allocate memory resources in MB granularity,
sometimes SRIOV VFs can mapped successfully (a bit by luck) since the VF's
don't require significant mem-mapped resources, and, there is enough leftover
space from the PF requested resources from the MB-block granularity.
Trying a newer, known-to-support SRIOV BIOS system will probably get you
out of the SRIOV resource errors you are reporting. A system with an IOMMU
in the northbridge(s) typically have SRIOV-savvy BIOS, but not always (some
system vendors use chipsets w/IOMMU, but not for that feature, so they don't
enable/use SRIOV extensions in the BIOS).
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
