On Tue, May 31, 2011 at 6:51 PM, Yinghai Lu <yinghai@xxxxxxxxxx> wrote:
> On 05/31/2011 03:52 PM, Ben Hutchings wrote:
>> Following commit da7822e5ad71ec9b745b412639f1e5e0ba795a20 ('PCI: update
>> bridge resources to get more big ranges when allocating space (again)'),
>> SFC9000-family network controllers in a Dell PE R905 are getting their
>> memory BARs disabled.
>>
>> These devices have:
>> BAR 0: I/O, 256 bytes
>> BAR 2: memory, 64-bit, 16 MB (for general registers)
>> BAR 4: memory, 64-bit, 64 KB (for MSI-X tables)
>>
>> Here is a diff of 'lspci -vn' output before and after this commit:
>>
>> --- /home/bwh/tmp/lspci-good-init.log 2011-05-31 23:30:39.496353000 +0100
>> +++ /home/bwh/tmp/lspci-bad-init.log 2011-05-31 23:22:02.507796000 +0100
>> @@ -41,7 +41,7 @@
>> Flags: bus master, fast devsel, latency 0
>> Bus: primary=00, secondary=0c, subordinate=0c, sec-latency=0
>> I/O behind bridge: 0000b000-0000bfff
>> - Memory behind bridge: ec000000-eeffffff
>> + Prefetchable memory behind bridge: 00000000ec000000-00000000ec000000
>> Capabilities: <access denied>
>>
>> 00:09.0 0604: 1166:0142 (rev a2)
>> @@ -270,8 +270,8 @@
>> Subsystem: 1924:6102
>> Flags: bus master, fast devsel, latency 0, IRQ 11
>> I/O ports at b800 [size=256]
>> - Memory at ed000000 (64-bit, non-prefetchable) [size=16M]
>> - Memory at ecfe0000 (64-bit, non-prefetchable) [size=64K]
>> + Memory at <ignored> (64-bit, non-prefetchable)
>> + Memory at <ignored> (64-bit, non-prefetchable)
>> Expansion ROM at ec000000 [disabled] [size=128K]
>> Capabilities: <access denied>
>>
>> @@ -279,8 +279,8 @@
>> Subsystem: 1924:6102
>> Flags: bus master, fast devsel, latency 0, IRQ 11
>> I/O ports at bc00 [size=256]
>> - Memory at ee000000 (64-bit, non-prefetchable) [size=16M]
>> - Memory at ecff0000 (64-bit, non-prefetchable) [size=64K]
>> + Memory at <ignored> (64-bit, non-prefetchable)
>> + Memory at <ignored> (64-bit, non-prefetchable)
>> Expansion ROM at ec020000 [disabled] [size=128K]
>> Capabilities: <access denied>
>>
>> @@ -288,7 +288,7 @@
>> Flags: bus master, fast devsel, latency 0
>> Bus: primary=20, secondary=21, subordinate=21, sec-latency=0
>> I/O behind bridge: 00009000-00009fff
>> - Memory behind bridge: d5000000-d7ffffff
>> + Prefetchable memory behind bridge: 00000000d5000000-00000000d5000000
>> Capabilities: <access denied>
>>
>> 20:09.0 0604: 1166:0142 (rev a2)
>> @@ -315,8 +315,8 @@
>> Subsystem: 1924:6205
>> Flags: bus master, fast devsel, latency 0, IRQ 5
>> I/O ports at 9800 [size=256]
>> - Memory at d6000000 (64-bit, non-prefetchable) [size=16M]
>> - Memory at d5fe0000 (64-bit, non-prefetchable) [size=64K]
>> + Memory at <ignored> (64-bit, non-prefetchable)
>> + Memory at <ignored> (64-bit, non-prefetchable)
>> Expansion ROM at d5000000 [disabled] [size=128K]
>> Capabilities: <access denied>
>>
>> @@ -324,8 +324,8 @@
>> Subsystem: 1924:6205
>> Flags: bus master, fast devsel, latency 0, IRQ 5
>> I/O ports at 9c00 [size=256]
>> - Memory at d7000000 (64-bit, non-prefetchable) [size=16M]
>> - Memory at d5ff0000 (64-bit, non-prefetchable) [size=64K]
>> + Memory at <ignored> (64-bit, non-prefetchable)
>> + Memory at <ignored> (64-bit, non-prefetchable)
>> Expansion ROM at d5020000 [disabled] [size=128K]
>> Capabilities: <access denied>
>>
>> --- END ---
>>
>> Below is a boot log of a kernel built from this commit, without the sfc
>> driver loaded. The devices in question are on buses 0c and 21.
>>
>> Ben.
>>
>>
> ...
>> PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
>> ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-1e])
>> pci_root PNP0A08:00: host bridge window [io 0x0000-0x0cf7]
>> pci_root PNP0A08:00: host bridge window [io 0xa000-0xffff]
>> pci_root PNP0A08:00: host bridge window [io 0x0d00-0x0fff]
>> pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
>> pci_root PNP0A08:00: host bridge window [mem 0xf0000000-0xf1ffffff]
>> pci_root PNP0A08:00: host bridge window [mem 0xe4000000-0xef4fffff]
>> pci_root PNP0A08:00: host bridge window [mem 0xd8000000-0xdfffffff]
>> pci_root PNP0A08:00: host bridge window [mem 0xfed40000-0xfed44fff]
> ...
>> pci 0000:0c:00.0: [1924:0813] type 0 class 0x000200
>> pci 0000:0c:00.0: reg 10: [io 0xb800-0xb8ff]
>> pci 0000:0c:00.0: reg 18: [mem 0xed000000-0xedffffff 64bit]
>> pci 0000:0c:00.0: reg 20: [mem 0xecfe0000-0xecfeffff 64bit]
>> pci 0000:0c:00.0: reg 30: [mem 0xec000000-0xec01ffff pref]
>> pci 0000:0c:00.0: PME# supported from D0 D3hot
>> pci 0000:0c:00.0: PME# disabled
>> pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
>> pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
>> pci 0000:0c:00.1: [1924:0813] type 0 class 0x000200
>> pci 0000:0c:00.1: reg 10: [io 0xbc00-0xbcff]
>> pci 0000:0c:00.1: reg 18: [mem 0xee000000-0xeeffffff 64bit]
>> pci 0000:0c:00.1: reg 20: [mem 0xecff0000-0xecffffff 64bit]
>> pci 0000:0c:00.1: reg 30: [mem 0xec000000-0xec01ffff pref]
>> pci 0000:0c:00.1: PME# supported from D0 D3hot
>> pci 0000:0c:00.1: PME# disabled
>> pci 0000:0c:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit]
>> pci 0000:0c:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
>> pci 0000:00:08.0: PCI bridge to [bus 0c-0c]
>> pci 0000:00:08.0: bridge window [io 0xb000-0xbfff]
>> pci 0000:00:08.0: bridge window [mem 0xec000000-0xeeffffff]
>> pci 0000:00:08.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
> ...
>> ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 20-3e])
>> pci_root PNP0A08:01: host bridge window [io 0x9000-0x9fff]
>> pci_root PNP0A08:01: host bridge window [mem 0xf2000000-0xf3ffffff]
>> pci_root PNP0A08:01: host bridge window [mem 0xd5000000-0xd7ffffff]
> ...
>> pci 0000:21:00.0: [1924:0803] type 0 class 0x000200
>> pci 0000:21:00.0: reg 10: [io 0x9800-0x98ff]
>> pci 0000:21:00.0: reg 18: [mem 0xd6000000-0xd6ffffff 64bit]
>> pci 0000:21:00.0: reg 20: [mem 0xd5fe0000-0xd5feffff 64bit]
>> pci 0000:21:00.0: reg 30: [mem 0xd5000000-0xd501ffff pref]
>> pci 0000:21:00.0: PME# supported from D0 D3hot
>> pci 0000:21:00.0: PME# disabled
>> pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
>> pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
>> pci 0000:21:00.1: [1924:0803] type 0 class 0x000200
>> pci 0000:21:00.1: reg 10: [io 0x9c00-0x9cff]
>> pci 0000:21:00.1: reg 18: [mem 0xd7000000-0xd7ffffff 64bit]
>> pci 0000:21:00.1: reg 20: [mem 0xd5ff0000-0xd5ffffff 64bit]
>> pci 0000:21:00.1: reg 30: [mem 0xd5000000-0xd501ffff pref]
>> pci 0000:21:00.1: PME# supported from D0 D3hot
>> pci 0000:21:00.1: PME# disabled
>> pci 0000:21:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit]
>> pci 0000:21:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
>> pci 0000:20:08.0: PCI bridge to [bus 21-21]
>> pci 0000:20:08.0: bridge window [io 0x9000-0x9fff]
>> pci 0000:20:08.0: bridge window [mem 0xd5000000-0xd7ffffff]
>> pci 0000:20:08.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
> ..
>> pci 0000:0c:00.1: address space collision: [mem 0xec000000-0xec01ffff pref] conflicts with 0000:0c:00.0 [mem 0xec000000-0xec01ffff pref]
>> pci 0000:21:00.1: address space collision: [mem 0xd5000000-0xd501ffff pref] conflicts with 0000:21:00.0 [mem 0xd5000000-0xd501ffff pref]
>
> your system is 4 sockets AMD quad cores system.
> two peer root bus: one to cpu 0, and one to cpu 3.
>
> 1. BIOS does assign same resource to func0 and func1.
> 2. BIOS does not assign resource to SR-IOV BAR...
> 3. BIOS does not preserve big enough allocation to peer root buses.
>
> then new code, try to assign resource to those unassigned or wrong assigned BARs, can not find enough resource for them.
>
> solution would be
> make SRIOV register BAR to be in good to have list
I don't think that's the right solution. Here's the path leading to
0c:00.0 and .1:
pci_root PNP0A08:00: host bridge window [mem 0xe4000000-0xef4fffff]
pci 0000:00:08.0: PCI bridge to [bus 0c-0c]
pci 0000:00:08.0: bridge window [mem 0xec000000-0xeeffffff] (48MB)
pci 0000:0c:00.0: reg 18: [mem 0xed000000-0xedffffff 64bit] (16MB)
pci 0000:0c:00.0: reg 20: [mem 0xecfe0000-0xecfeffff 64bit] (64KB)
pci 0000:0c:00.0: reg 30: [mem 0xec000000-0xec01ffff pref] (128KB)
pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit] (8KB)
pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit] (64KB)
pci 0000:0c:00.1: reg 18: [mem 0xee000000-0xeeffffff 64bit]
pci 0000:0c:00.1: reg 20: [mem 0xecff0000-0xecffffff 64bit]
pci 0000:0c:00.1: reg 30: [mem 0xec000000-0xec01ffff pref]
pci 0000:0c:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit]
pci 0000:0c:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
Each function needs 16MB + 264KB. Both functions together will easily
fit in the 48MB bridge window, even if we allocate separate ROM space
and all the SR-IOV BARs.
But we're doing something wrong when assigning the second SR-IOV BAR:
pci 0000:0c:00.0: BAR 9: assigned [mem 0xec040000-0xec82ffff 64bit]
This has a size of 0x7f0000 when it should only be 0x10000 (64KB). I
don't think 0x7f0000 is even a legal size for a PCI BAR; it should be
a power of two.
Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
