On 05/31/2011 03:52 PM, Ben Hutchings wrote:
> Following commit da7822e5ad71ec9b745b412639f1e5e0ba795a20 ('PCI: update
> bridge resources to get more big ranges when allocating space (again)'),
> SFC9000-family network controllers in a Dell PE R905 are getting their
> memory BARs disabled.
>
> These devices have:
> BAR 0: I/O, 256 bytes
> BAR 2: memory, 64-bit, 16 MB (for general registers)
> BAR 4: memory, 64-bit, 64 KB (for MSI-X tables)
>
> Here is a diff of 'lspci -vn' output before and after this commit:
>
> --- /home/bwh/tmp/lspci-good-init.log 2011-05-31 23:30:39.496353000 +0100
> +++ /home/bwh/tmp/lspci-bad-init.log 2011-05-31 23:22:02.507796000 +0100
> @@ -41,7 +41,7 @@
> Flags: bus master, fast devsel, latency 0
> Bus: primary=00, secondary=0c, subordinate=0c, sec-latency=0
> I/O behind bridge: 0000b000-0000bfff
> - Memory behind bridge: ec000000-eeffffff
> + Prefetchable memory behind bridge: 00000000ec000000-00000000ec000000
> Capabilities: <access denied>
>
> 00:09.0 0604: 1166:0142 (rev a2)
> @@ -270,8 +270,8 @@
> Subsystem: 1924:6102
> Flags: bus master, fast devsel, latency 0, IRQ 11
> I/O ports at b800 [size=256]
> - Memory at ed000000 (64-bit, non-prefetchable) [size=16M]
> - Memory at ecfe0000 (64-bit, non-prefetchable) [size=64K]
> + Memory at <ignored> (64-bit, non-prefetchable)
> + Memory at <ignored> (64-bit, non-prefetchable)
> Expansion ROM at ec000000 [disabled] [size=128K]
> Capabilities: <access denied>
>
> @@ -279,8 +279,8 @@
> Subsystem: 1924:6102
> Flags: bus master, fast devsel, latency 0, IRQ 11
> I/O ports at bc00 [size=256]
> - Memory at ee000000 (64-bit, non-prefetchable) [size=16M]
> - Memory at ecff0000 (64-bit, non-prefetchable) [size=64K]
> + Memory at <ignored> (64-bit, non-prefetchable)
> + Memory at <ignored> (64-bit, non-prefetchable)
> Expansion ROM at ec020000 [disabled] [size=128K]
> Capabilities: <access denied>
>
> @@ -288,7 +288,7 @@
> Flags: bus master, fast devsel, latency 0
> Bus: primary=20, secondary=21, subordinate=21, sec-latency=0
> I/O behind bridge: 00009000-00009fff
> - Memory behind bridge: d5000000-d7ffffff
> + Prefetchable memory behind bridge: 00000000d5000000-00000000d5000000
> Capabilities: <access denied>
>
> 20:09.0 0604: 1166:0142 (rev a2)
> @@ -315,8 +315,8 @@
> Subsystem: 1924:6205
> Flags: bus master, fast devsel, latency 0, IRQ 5
> I/O ports at 9800 [size=256]
> - Memory at d6000000 (64-bit, non-prefetchable) [size=16M]
> - Memory at d5fe0000 (64-bit, non-prefetchable) [size=64K]
> + Memory at <ignored> (64-bit, non-prefetchable)
> + Memory at <ignored> (64-bit, non-prefetchable)
> Expansion ROM at d5000000 [disabled] [size=128K]
> Capabilities: <access denied>
>
> @@ -324,8 +324,8 @@
> Subsystem: 1924:6205
> Flags: bus master, fast devsel, latency 0, IRQ 5
> I/O ports at 9c00 [size=256]
> - Memory at d7000000 (64-bit, non-prefetchable) [size=16M]
> - Memory at d5ff0000 (64-bit, non-prefetchable) [size=64K]
> + Memory at <ignored> (64-bit, non-prefetchable)
> + Memory at <ignored> (64-bit, non-prefetchable)
> Expansion ROM at d5020000 [disabled] [size=128K]
> Capabilities: <access denied>
>
> --- END ---
>
> Below is a boot log of a kernel built from this commit, without the sfc
> driver loaded. The devices in question are on buses 0c and 21.
>
> Ben.
>
>
...
> PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
> ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-1e])
> pci_root PNP0A08:00: host bridge window [io 0x0000-0x0cf7]
> pci_root PNP0A08:00: host bridge window [io 0xa000-0xffff]
> pci_root PNP0A08:00: host bridge window [io 0x0d00-0x0fff]
> pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
> pci_root PNP0A08:00: host bridge window [mem 0xf0000000-0xf1ffffff]
> pci_root PNP0A08:00: host bridge window [mem 0xe4000000-0xef4fffff]
> pci_root PNP0A08:00: host bridge window [mem 0xd8000000-0xdfffffff]
> pci_root PNP0A08:00: host bridge window [mem 0xfed40000-0xfed44fff]
...
> pci 0000:0c:00.0: [1924:0813] type 0 class 0x000200
> pci 0000:0c:00.0: reg 10: [io 0xb800-0xb8ff]
> pci 0000:0c:00.0: reg 18: [mem 0xed000000-0xedffffff 64bit]
> pci 0000:0c:00.0: reg 20: [mem 0xecfe0000-0xecfeffff 64bit]
> pci 0000:0c:00.0: reg 30: [mem 0xec000000-0xec01ffff pref]
> pci 0000:0c:00.0: PME# supported from D0 D3hot
> pci 0000:0c:00.0: PME# disabled
> pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:0c:00.1: [1924:0813] type 0 class 0x000200
> pci 0000:0c:00.1: reg 10: [io 0xbc00-0xbcff]
> pci 0000:0c:00.1: reg 18: [mem 0xee000000-0xeeffffff 64bit]
> pci 0000:0c:00.1: reg 20: [mem 0xecff0000-0xecffffff 64bit]
> pci 0000:0c:00.1: reg 30: [mem 0xec000000-0xec01ffff pref]
> pci 0000:0c:00.1: PME# supported from D0 D3hot
> pci 0000:0c:00.1: PME# disabled
> pci 0000:0c:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:0c:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:00:08.0: PCI bridge to [bus 0c-0c]
> pci 0000:00:08.0: bridge window [io 0xb000-0xbfff]
> pci 0000:00:08.0: bridge window [mem 0xec000000-0xeeffffff]
> pci 0000:00:08.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
...
> ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 20-3e])
> pci_root PNP0A08:01: host bridge window [io 0x9000-0x9fff]
> pci_root PNP0A08:01: host bridge window [mem 0xf2000000-0xf3ffffff]
> pci_root PNP0A08:01: host bridge window [mem 0xd5000000-0xd7ffffff]
...
> pci 0000:21:00.0: [1924:0803] type 0 class 0x000200
> pci 0000:21:00.0: reg 10: [io 0x9800-0x98ff]
> pci 0000:21:00.0: reg 18: [mem 0xd6000000-0xd6ffffff 64bit]
> pci 0000:21:00.0: reg 20: [mem 0xd5fe0000-0xd5feffff 64bit]
> pci 0000:21:00.0: reg 30: [mem 0xd5000000-0xd501ffff pref]
> pci 0000:21:00.0: PME# supported from D0 D3hot
> pci 0000:21:00.0: PME# disabled
> pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:21:00.1: [1924:0803] type 0 class 0x000200
> pci 0000:21:00.1: reg 10: [io 0x9c00-0x9cff]
> pci 0000:21:00.1: reg 18: [mem 0xd7000000-0xd7ffffff 64bit]
> pci 0000:21:00.1: reg 20: [mem 0xd5ff0000-0xd5ffffff 64bit]
> pci 0000:21:00.1: reg 30: [mem 0xd5000000-0xd501ffff pref]
> pci 0000:21:00.1: PME# supported from D0 D3hot
> pci 0000:21:00.1: PME# disabled
> pci 0000:21:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:21:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:20:08.0: PCI bridge to [bus 21-21]
> pci 0000:20:08.0: bridge window [io 0x9000-0x9fff]
> pci 0000:20:08.0: bridge window [mem 0xd5000000-0xd7ffffff]
> pci 0000:20:08.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
..
> pci 0000:0c:00.1: address space collision: [mem 0xec000000-0xec01ffff pref] conflicts with 0000:0c:00.0 [mem 0xec000000-0xec01ffff pref]
> pci 0000:21:00.1: address space collision: [mem 0xd5000000-0xd501ffff pref] conflicts with 0000:21:00.0 [mem 0xd5000000-0xd501ffff pref]
your system is 4 sockets AMD quad cores system.
two peer root bus: one to cpu 0, and one to cpu 3.
1. BIOS does assign same resource to func0 and func1.
2. BIOS does not assign resource to SR-IOV BAR...
3. BIOS does not preserve big enough allocation to peer root buses.
then new code, try to assign resource to those unassigned or wrong assigned BARs, can not find enough resource for them.
solution would be
make SRIOV register BAR to be in good to have list
will try to produce one patch for that.
Thanks
Yinghai
> PCI: max bus depth: 2 pci_try_num: 3
> pci 0000:00:02.0: BAR 15: can't assign mem pref (size 0x100000)
> pci 0000:00:08.0: BAR 15: can't assign mem pref (size 0x100000)
> pci 0000:00:0d.0: BAR 6: assigned [mem 0xef000000-0xef01ffff pref]
> pci 0000:01:0e.0: BAR 6: assigned [mem 0xef100000-0xef11ffff pref]
> pci 0000:00:02.0: PCI bridge to [bus 01-01]
> pci 0000:00:02.0: bridge window [io 0xf000-0xffff]
> pci 0000:00:02.0: bridge window [mem 0xef100000-0xef1fffff]
> pci 0000:00:02.0: bridge window [mem pref disabled]
> pci 0000:00:03.0: PCI bridge to [bus 02-02]
> pci 0000:00:03.0: bridge window [io 0xd000-0xefff]
> pci 0000:00:03.0: bridge window [mem 0xef200000-0xef2fffff]
> pci 0000:00:03.0: bridge window [mem pref disabled]
> pci 0000:03:00.0: PCI bridge to [bus 04-04]
> pci 0000:03:00.0: bridge window [io disabled]
> pci 0000:03:00.0: bridge window [mem 0xe4000000-0xe5ffffff]
> pci 0000:03:00.0: bridge window [mem pref disabled]
> pci 0000:00:04.0: PCI bridge to [bus 03-04]
> pci 0000:00:04.0: bridge window [io disabled]
> pci 0000:00:04.0: bridge window [mem 0xe4000000-0xe5ffffff]
> pci 0000:00:04.0: bridge window [mem pref disabled]
> pci 0000:05:00.0: PCI bridge to [bus 06-06]
> pci 0000:05:00.0: bridge window [io disabled]
> pci 0000:05:00.0: bridge window [mem 0xe6000000-0xe7ffffff]
> pci 0000:05:00.0: bridge window [mem pref disabled]
> pci 0000:00:05.0: PCI bridge to [bus 05-06]
> pci 0000:00:05.0: bridge window [io disabled]
> pci 0000:00:05.0: bridge window [mem 0xe6000000-0xe7ffffff]
> pci 0000:00:05.0: bridge window [mem pref disabled]
> pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:0c:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:0c:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:0c:00.1: BAR 6: assigned [mem 0xec020000-0xec03ffff pref]
> pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:0c:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:0c:00.0: BAR 9: assigned [mem 0xec040000-0xec82ffff 64bit]
> pci 0000:0c:00.0: BAR 9: set to [mem 0xec040000-0xec82ffff 64bit] (PCI address [0xec040000-0xec82ffff])
> pci 0000:0c:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:0c:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:0c:00.1: BAR 9: can't assign mem (size 0x7f0000)
> pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:0c:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:0c:00.0: BAR 7: assigned [mem 0xec830000-0xec92dfff 64bit]
> pci 0000:0c:00.0: BAR 7: set to [mem 0xec830000-0xec92dfff 64bit] (PCI address [0xec830000-0xec92dfff])
> pci 0000:0c:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:0c:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:0c:00.1: BAR 7: assigned [mem 0xec92e000-0xeca2bfff 64bit]
> pci 0000:0c:00.1: BAR 7: set to [mem 0xec92e000-0xeca2bfff 64bit] (PCI address [0xec92e000-0xeca2bfff])
> pci 0000:00:08.0: PCI bridge to [bus 0c-0c]
> pci 0000:00:08.0: bridge window [io 0xb000-0xbfff]
> pci 0000:00:08.0: bridge window [mem 0xec000000-0xeeffffff]
> pci 0000:00:08.0: bridge window [mem pref disabled]
> pci 0000:07:00.0: PCI bridge to [bus 08-08]
> pci 0000:07:00.0: bridge window [io disabled]
> pci 0000:07:00.0: bridge window [mem 0xe8000000-0xe9ffffff]
> pci 0000:07:00.0: bridge window [mem pref disabled]
> pci 0000:00:09.0: PCI bridge to [bus 07-08]
> pci 0000:00:09.0: bridge window [io disabled]
> pci 0000:00:09.0: bridge window [mem 0xe8000000-0xe9ffffff]
> pci 0000:00:09.0: bridge window [mem pref disabled]
> pci 0000:09:00.0: PCI bridge to [bus 0a-0a]
> pci 0000:09:00.0: bridge window [io disabled]
> pci 0000:09:00.0: bridge window [mem 0xea000000-0xebffffff]
> pci 0000:09:00.0: bridge window [mem pref disabled]
> pci 0000:00:0a.0: PCI bridge to [bus 09-0a]
> pci 0000:00:0a.0: bridge window [io disabled]
> pci 0000:00:0a.0: bridge window [mem 0xea000000-0xebffffff]
> pci 0000:00:0a.0: bridge window [mem pref disabled]
> pci 0000:00:0b.0: PCI bridge to [bus 0b-0b]
> pci 0000:00:0b.0: bridge window [io 0xc000-0xcfff]
> pci 0000:00:0b.0: bridge window [mem 0xef300000-0xef4fffff]
> pci 0000:00:0b.0: bridge window [mem pref disabled]
> pci 0000:00:0c.0: PCI bridge to [bus 0d-0d]
> pci 0000:00:0c.0: bridge window [io disabled]
> pci 0000:00:0c.0: bridge window [mem disabled]
> pci 0000:00:0c.0: bridge window [mem pref disabled]
> pci 0000:20:08.0: BAR 15: can't assign mem pref (size 0x100000)
> pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:21:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:21:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:21:00.1: BAR 6: assigned [mem 0xd5020000-0xd503ffff pref]
> pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:21:00.0: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:21:00.0: BAR 9: assigned [mem 0xd5040000-0xd582ffff 64bit]
> pci 0000:21:00.0: BAR 9: set to [mem 0xd5040000-0xd582ffff 64bit] (PCI address [0xd5040000-0xd582ffff])
> pci 0000:21:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:21:00.1: reg 18c: [mem 0x00000000-0x0000ffff 64bit]
> pci 0000:21:00.1: BAR 9: can't assign mem (size 0x7f0000)
> pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:21:00.0: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:21:00.0: BAR 7: assigned [mem 0xd5830000-0xd592dfff 64bit]
> pci 0000:21:00.0: BAR 7: set to [mem 0xd5830000-0xd592dfff 64bit] (PCI address [0xd5830000-0xd592dfff])
> pci 0000:21:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:21:00.1: reg 184: [mem 0x00000000-0x00001fff 64bit]
> pci 0000:21:00.1: BAR 7: assigned [mem 0xd592e000-0xd5a2bfff 64bit]
> pci 0000:21:00.1: BAR 7: set to [mem 0xd592e000-0xd5a2bfff 64bit] (PCI address [0xd592e000-0xd5a2bfff])
> pci 0000:20:08.0: PCI bridge to [bus 21-21]
> pci 0000:20:08.0: bridge window [io 0x9000-0x9fff]
> pci 0000:20:08.0: bridge window [mem 0xd5000000-0xd7ffffff]
> pci 0000:20:08.0: bridge window [mem pref disabled]
> pci 0000:20:09.0: PCI bridge to [bus 22-22]
> pci 0000:20:09.0: bridge window [io disabled]
> pci 0000:20:09.0: bridge window [mem disabled]
> pci 0000:20:09.0: bridge window [mem pref disabled]
> pci 0000:20:0a.0: PCI bridge to [bus 23-23]
> pci 0000:20:0a.0: bridge window [io disabled]
> pci 0000:20:0a.0: bridge window [mem disabled]
> pci 0000:20:0a.0: bridge window [mem pref disabled]
> pci 0000:20:0b.0: PCI bridge to [bus 24-24]
> pci 0000:20:0b.0: bridge window [io disabled]
> pci 0000:20:0b.0: bridge window [mem disabled]
> pci 0000:20:0b.0: bridge window [mem pref disabled]
> pci 0000:20:0c.0: PCI bridge to [bus 25-25]
> pci 0000:20:0c.0: bridge window [io disabled]
> pci 0000:20:0c.0: bridge window [mem disabled]
> pci 0000:20:0c.0: bridge window [mem pref disabled]
> PCI: No. 2 try to assign unassigned res
> release child resource [mem 0xd5000000-0xd501ffff pref]
> release child resource [mem 0xd5020000-0xd503ffff pref]
> release child resource [mem 0xd5040000-0xd582ffff 64bit]
> release child resource [mem 0xd5830000-0xd592dfff 64bit]
> release child resource [mem 0xd592e000-0xd5a2bfff 64bit]
> release child resource [mem 0xd5fe0000-0xd5feffff 64bit]
> release child resource [mem 0xd5ff0000-0xd5ffffff 64bit]
> release child resource [mem 0xd6000000-0xd6ffffff 64bit]
> release child resource [mem 0xd7000000-0xd7ffffff 64bit]
...
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
