Hello,
AMCC PPC460EX Canyonlands development board with PCI Express x1 card
in PCIe x4 slot labeled 'PCIE-1'. The PCI Express card has a
soft-programmable PCIe end point.
Problem: The end point with only non-prefetchable BAR does not appear
behind the PPC SoC internal bridge. It does show up on two other
setups: MPC8315E-RDB PPC as well as a PC x86.
It has BARs 0-5 each as a 32-bit non-prefetchable memory region. This
is the kernel boot:
pci 0001:80:00.0: PCI bridge, secondary bus 0001:81
pci 0001:80:00.0: IO window: disabled
pci 0001:80:00.0: MEM window: disabled
pci 0001:80:00.0: PREFETCH window: disabled
...
bus: 81 index 0 mmio: [0xfffe0000-0xfffe0fff]
bus: 81 index 1 mmio: [0xeb8000000-0xeb80fffff]
bus: 81 index 2 mmio: [0x0-0x0]
bus: 81 index 3 mmio: [0x0-0x0]
lspci -v does not show the device.
The default end point design does have a prefetchable region and does
show up properly:
pci 0001:80:00.0: PCI bridge, secondary bus 0001:81
pci 0001:80:00.0: IO window: disabled
pci 0001:80:00.0: MEM window: 0x81000000-0x810fffff
pci 0001:80:00.0: PREFETCH window: 0x00000080000000-0x00000080ffffff
...
bus: 81 index 0 mmio: [0xfffe0000-0xfffe0fff]
bus: 81 index 1 mmio: [0xe81000000-0xe810fffff]
bus: 81 index 2 mmio: [0xe80000000-0xe80ffffff]
bus: 81 index 3 mmio: [0x0-0x0]
Working and non-working behaviour is identical on 2.6.27.7 and today's
-next tree.
Full u-boot, kernel and lspci -v output attached. Also put online on:
http://www.sidebranch.com/leon/next_broken.txt
http://www.sidebranch.com/leon/next_works.txt
Any ideas why/where this might go wrong? I'm diving into the code now,
especially seeing why the memory behind the bridge was "disabled".
Regards,
--
Leon
u-boot
U-Boot 1.3.3-00249-ga524e11 (Jun 30 2008 - 16:05:51)
CPU: AMCC PowerPC 460EX Rev. A at 800 MHz (PLB=200, OPB=100, EBC=100 MHz)
Security/Kasumi support
Bootstrap Option H - Boot ROM Location I2C (Addr 0x52)
Internal PCI arbiter disabled
32 kB I-Cache 32 kB D-Cache
Board: Canyonlands - AMCC PPC460EX Evaluation Board, 2*PCIe, Rev. 16
I2C: ready
DTT: 1 is 46 C
DRAM: 512 MB (ECC not enabled, 400 MHz, CL3)
FLASH: 64 MB
NAND: 128 MiB
PCI: Bus Dev VenId DevId Class Int
PCIE0: link is not up.
PCIE0: initialization as root-complex failed
PCIE1: successfully set as root-complex
02 00 1a0b 5331 ff00 00
Net: ppc_4xx_eth0, ppc_4xx_eth1
kernel
Linux version 2.6.28-rc6-next-20081125-dirty (leon@bigtee) (gcc version 4.2.4) #1 Wed Nov 26 15:02:52 CET 2008
Zone PFN ranges:
DMA 0x00000000 -> 0x00020000
Normal 0x00020000 -> 0x00020000
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
0: 0x00000000 -> 0x00020000
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
Kernel command line: root=/dev/nfs rw nfsroot=172.16.0.50:/nfsroot/canyonlands ip=172.16.0.46:172.16.0.50:::canyonlands:eth0:off panic=1 console=ttyS0,115200
UIC0 (32 IRQ sources) at DCR 0xc0
UIC1 (32 IRQ sources) at DCR 0xd0
UIC2 (32 IRQ sources) at DCR 0xe0
UIC3 (32 IRQ sources) at DCR 0xf0
PID hash table entries: 2048 (order: 11, 8192 bytes)
clocksource: timebase mult[500000] shift[22] registered
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 516736k/524288k available (2432k kernel code, 7232k reserved, 100k data, 119k bss, 144k init)
SLUB: Genslabs=10, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Calibrating delay loop... 1597.44 BogoMIPS (lpj=3194880)
Mount-cache hash table entries: 512
net_namespace: 300 bytes
NET: Registered protocol family 16
256k L2-cache enabled
PCIE0: Checking link...
PCIE0: No device detected.
PCI host bridge /plb/pciex@d00000000 (primary) ranges:
MEM 0x0000000e00000000..0x0000000e7fffffff -> 0x0000000080000000
IO 0x0000000f80000000..0x0000000f8000ffff -> 0x0000000000000000
4xx PCI DMA offset set to 0x00000000
PCIE0: successfully set as root-complex
PCIE1: Checking link...
PCIE1: Device detected, waiting for link...
PCIE1: link is up !
PCI host bridge /plb/pciex@d20000000 (primary) ranges:
MEM 0x0000000e80000000..0x0000000effffffff -> 0x0000000080000000
IO 0x0000000f80010000..0x0000000f8001ffff -> 0x0000000000000000
4xx PCI DMA offset set to 0x00000000
PCIE1: successfully set as root-complex
PCI host bridge /plb/pci@c0ec00000 (primary) ranges:
MEM 0x0000000d80000000..0x0000000dffffffff -> 0x0000000080000000
IO 0x0000000c08000000..0x0000000c0800ffff -> 0x0000000000000000
4xx PCI DMA offset set to 0x00000000
PCI: Probing PCI hardware
PCI: Hiding 4xx host bridge resources 0000:40:00.0
PCI: Hiding 4xx host bridge resources 0001:80:00.0
pci 0001:81:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force'
pci 0000:40:00.0: PCI bridge, secondary bus 0000:41
pci 0000:40:00.0: IO window: disabled
pci 0000:40:00.0: MEM window: disabled
pci 0000:40:00.0: PREFETCH window: disabled
pci 0001:80:00.0: PCI bridge, secondary bus 0001:81
pci 0001:80:00.0: IO window: disabled
pci 0001:80:00.0: MEM window: 0x81000000-0x810fffff
pci 0001:80:00.0: PREFETCH window: 0x00000080000000-0x00000080ffffff
bus: 40 index 0 io port: [0xfffc0000-0xfffcffff]
bus: 40 index 1 mmio: [0xe00000000-0xe7fffffff]
bus: 41 index 0 mmio: [0xfffc0000-0xfffc0fff]
bus: 41 index 1 mmio: [0xd80000000-0xd800fffff]
bus: 41 index 2 mmio: [0xd80000000-0xd800fffff]
bus: 41 index 3 mmio: [0x0-0x0]
bus: 80 index 0 io port: [0xfffe0000-0xfffeffff]
bus: 80 index 1 mmio: [0xe80000000-0xeffffffff]
bus: 81 index 0 mmio: [0xfffe0000-0xfffe0fff]
bus: 81 index 1 mmio: [0xe81000000-0xe810fffff]
bus: 81 index 2 mmio: [0xe80000000-0xe80ffffff]
bus: 81 index 3 mmio: [0x0-0x0]
bus: 00 index 0 io port: [0x00-0xffff]
bus: 00 index 1 mmio: [0xd80000000-0xdffffffff]
NET: Registered protocol family 2
IP route cache hash table entries: 16384 (order: 4, 65536 bytes)
TCP established hash table entries: 65536 (order: 7, 524288 bytes)
TCP bind hash table entries: 65536 (order: 6, 262144 bytes)
TCP: Hash tables configured (established 65536 bind 65536)
TCP reno registered
NET: Registered protocol family 1
msgmni has been set to 1009
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
pcieport-driver 0000:40:00.0: found MSI capability
pcieport-driver 0001:80:00.0: found MSI capability
aer: probe of 0000:40:00.0:pcie01 failed with error -38
aer: probe of 0001:80:00.0:pcie01 failed with error -38
Serial: 8250/16550 driver4 ports, IRQ sharing enabled
serial8250.0: ttyS0 at MMIO 0x4ef600300 (irq = 19) is a 16550A
console [ttyS0] enabled
serial8250.0: ttyS1 at MMIO 0x4ef600400 (irq = 20) is a 16550A
serial8250.0: ttyS2 at MMIO 0x4ef600500 (irq = 29) is a 16550A
serial8250.0: ttyS3 at MMIO 0x4ef600600 (irq = 21) is a 16550A
4ef600300.serial: ttyS0 at MMIO 0x4ef600300 (irq = 19) is a 16550A
4ef600400.serial: ttyS1 at MMIO 0x4ef600400 (irq = 20) is a 16550A
4ef600500.serial: ttyS2 at MMIO 0x4ef600500 (irq = 29) is a 16550A
4ef600600.serial: ttyS3 at MMIO 0x4ef600600 (irq = 21) is a 16550A
brd: module loaded
PPC 4xx OCP EMAC driver, version 3.54
MAL v2 /plb/mcmal, 2 TX channels, 16 RX channels
ZMII /plb/opb/emac-zmii@ef600d00 initialized
RGMII /plb/opb/emac-rgmii@ef601500 initialized with MDIO support
TAH /plb/opb/emac-tah@ef601350 initialized
TAH /plb/opb/emac-tah@ef601450 initialized
/plb/opb/emac-rgmii@ef601500: input 0 in RGMII mode
eth0 (emac): not using net_device_ops yet
eth0: EMAC-0 /plb/opb/ethernet@ef600e00, MAC 00:10:ec:01:06:54
eth0: found Generic MII PHY (0x00)
/plb/opb/emac-rgmii@ef601500: input 1 in RGMII mode
eth1 (emac): not using net_device_ops yet
eth1: EMAC-1 /plb/opb/ethernet@ef600f00, MAC 00:10:ec:81:06:54
eth1: found Generic MII PHY (0x01)
TCP cubic registered
NET: Registered protocol family 17
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
eth0: link is down
IP-Config: Guessing netmask 255.255.0.0
IP-Config: Complete:
device=eth0, addr=172.16.0.46, mask=255.255.0.0, gw=255.255.255.255,
host=canyonlands, domain=, nis-domain=(none),
bootserver=172.16.0.50, rootserver=172.16.0.50, rootpath=
Looking up port of RPC 100003/2 on 172.16.0.50
eth0: link is up, 1000 FDX, pause enabled
Looking up port of RPC 100005/1 on 172.16.0.50
VFS: Mounted root (nfs filesystem).
Freeing unused kernel memory: 144k init
lspci -v
0000:40:00.0 Class 0604: Unknown device aaa0:bed0 (rev 01)
Flags: bus master, fast devsel, latency 0
Memory at <unassigned> (32-bit, prefetchable)
Bus: primary=40, secondary=41, subordinate=7f, sec-latency=0
Capabilities: [7c] Vital Product Data
Capabilities: [40] Power Management version 3
Capabilities: [48] Message Signalled Interrupts: Mask- 64bit+ Queue=0/2 Enable-
Capabilities: [58] Express Root Port (Slot-) IRQ 0
Capabilities: [100] Advanced Error Reporting
Capabilities: [1f8] Unknown (11)
0001:80:00.0 Class 0604: Unknown device aaa1:bed1 (rev 01)
Flags: bus master, fast devsel, latency 0
Memory at <unassigned> (32-bit, prefetchable)
Bus: primary=80, secondary=81, subordinate=bf, sec-latency=0
Memory behind bridge: 81000000-810fffff
Prefetchable memory behind bridge: 0000000080000000-0000000080ffffff
Capabilities: [7c] Vital Product Data
Capabilities: [40] Power Management version 3
Capabilities: [48] Message Signalled Interrupts: Mask- 64bit+ Queue=0/2 Enable-
Capabilities: [58] Express Root Port (Slot-) IRQ 0
Capabilities: [100] Advanced Error Reporting
Capabilities: [1f8] Unknown (11)
0001:81:00.0 Class ff00: Unknown device 1a0b:5331 (rev 01)
Subsystem: Unknown device 1172:0007
Flags: bus master, fast devsel, latency 0, IRQ 18
Memory at e80000000 (64-bit, prefetchable) [size=16M]
Memory at e81000000 (32-bit, non-prefetchable) [size=256K]
Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ Queue=0/2 Enable-
Capabilities: [78] Power Management version 2
Capabilities: [80] Express Endpoint IRQ 0
Capabilities: [100] Virtual Channel
u-boot
U-Boot 1.3.3-00249-ga524e11 (Jun 30 2008 - 16:05:51)
CPU: AMCC PowerPC 460EX Rev. A at 800 MHz (PLB=200, OPB=100, EBC=100 MHz)
Security/Kasumi support
Bootstrap Option H - Boot ROM Location I2C (Addr 0x52)
Internal PCI arbiter disabled
32 kB I-Cache 32 kB D-Cache
Board: Canyonlands - AMCC PPC460EX Evaluation Board, 2*PCIe, Rev. 16
I2C: ready
DTT: 1 is 46 C
DRAM: 512 MB (ECC not enabled, 400 MHz, CL3)
FLASH: 64 MB
NAND: 128 MiB
PCI: Bus Dev VenId DevId Class Int
PCIE0: link is not up.
PCIE0: initialization as root-complex failed
PCIE1: successfully set as root-complex
02 00 2071 2071 ff00 00
Net: ppc_4xx_eth0, ppc_4xx_eth1
kernel
Linux version 2.6.28-rc6-next-20081125-dirty (leon@bigtee) (gcc version 4.2.4) #1 Wed Nov 26 15:02:52 CET 2008
Zone PFN ranges:
DMA 0x00000000 -> 0x00020000
Normal 0x00020000 -> 0x00020000
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
0: 0x00000000 -> 0x00020000
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
Kernel command line: root=/dev/nfs rw nfsroot=172.16.0.50:/nfsroot/canyonlands ip=172.16.0.46:172.16.0.50:::canyonlands:eth0:off panic=1 console=ttyS0,115200
UIC0 (32 IRQ sources) at DCR 0xc0
UIC1 (32 IRQ sources) at DCR 0xd0
UIC2 (32 IRQ sources) at DCR 0xe0
UIC3 (32 IRQ sources) at DCR 0xf0
PID hash table entries: 2048 (order: 11, 8192 bytes)
clocksource: timebase mult[500000] shift[22] registered
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 516736k/524288k available (2432k kernel code, 7232k reserved, 100k data, 119k bss, 144k init)
SLUB: Genslabs=10, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Calibrating delay loop... 1597.44 BogoMIPS (lpj=3194880)
Mount-cache hash table entries: 512
net_namespace: 300 bytes
NET: Registered protocol family 16
256k L2-cache enabled
PCIE0: Checking link...
PCIE0: No device detected.
PCI host bridge /plb/pciex@d00000000 (primary) ranges:
MEM 0x0000000e00000000..0x0000000e7fffffff -> 0x0000000080000000
IO 0x0000000f80000000..0x0000000f8000ffff -> 0x0000000000000000
4xx PCI DMA offset set to 0x00000000
PCIE0: successfully set as root-complex
PCIE1: Checking link...
PCIE1: Device detected, waiting for link...
PCIE1: link is up !
PCI host bridge /plb/pciex@d20000000 (primary) ranges:
MEM 0x0000000e80000000..0x0000000effffffff -> 0x0000000080000000
IO 0x0000000f80010000..0x0000000f8001ffff -> 0x0000000000000000
4xx PCI DMA offset set to 0x00000000
PCIE1: successfully set as root-complex
PCI host bridge /plb/pci@c0ec00000 (primary) ranges:
MEM 0x0000000d80000000..0x0000000dffffffff -> 0x0000000080000000
IO 0x0000000c08000000..0x0000000c0800ffff -> 0x0000000000000000
4xx PCI DMA offset set to 0x00000000
PCI: Probing PCI hardware
PCI: Hiding 4xx host bridge resources 0000:40:00.0
PCI: Hiding 4xx host bridge resources 0001:80:00.0
pci 0000:40:00.0: PCI bridge, secondary bus 0000:41
pci 0000:40:00.0: IO window: disabled
pci 0000:40:00.0: MEM window: disabled
pci 0000:40:00.0: PREFETCH window: disabled
pci 0001:80:00.0: PCI bridge, secondary bus 0001:81
pci 0001:80:00.0: IO window: disabled
pci 0001:80:00.0: MEM window: disabled
pci 0001:80:00.0: PREFETCH window: disabled
bus: 40 index 0 io port: [0xfffc0000-0xfffcffff]
bus: 40 index 1 mmio: [0xe00000000-0xe7fffffff]
bus: 41 index 0 mmio: [0xfffc0000-0xfffc0fff]
bus: 41 index 1 mmio: [0xd80000000-0xd800fffff]
bus: 41 index 2 mmio: [0xd80000000-0xd800fffff]
bus: 41 index 3 mmio: [0x0-0x0]
bus: 80 index 0 io port: [0xfffe0000-0xfffeffff]
bus: 80 index 1 mmio: [0xe80000000-0xeffffffff]
bus: 81 index 0 mmio: [0xfffe0000-0xfffe0fff]
bus: 81 index 1 mmio: [0xeb8000000-0xeb80fffff]
bus: 81 index 2 mmio: [0x0-0x0]
bus: 81 index 3 mmio: [0x0-0x0]
bus: 00 index 0 io port: [0x00-0xffff]
bus: 00 index 1 mmio: [0xd80000000-0xdffffffff]
NET: Registered protocol family 2
IP route cache hash table entries: 16384 (order: 4, 65536 bytes)
TCP established hash table entries: 65536 (order: 7, 524288 bytes)
TCP bind hash table entries: 65536 (order: 6, 262144 bytes)
TCP: Hash tables configured (established 65536 bind 65536)
TCP reno registered
NET: Registered protocol family 1
msgmni has been set to 1009
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
pcieport-driver 0000:40:00.0: found MSI capability
pcieport-driver 0001:80:00.0: found MSI capability
aer: probe of 0000:40:00.0:pcie01 failed with error -38
aer: probe of 0001:80:00.0:pcie01 failed with error -38
Serial: 8250/16550 driver4 ports, IRQ sharing enabled
serial8250.0: ttyS0 at MMIO 0x4ef600300 (irq = 18) is a 16550A
console [ttyS0] enabled
serial8250.0: ttyS1 at MMIO 0x4ef600400 (irq = 19) is a 16550A
serial8250.0: ttyS2 at MMIO 0x4ef600500 (irq = 29) is a 16550A
serial8250.0: ttyS3 at MMIO 0x4ef600600 (irq = 20) is a 16550A
4ef600300.serial: ttyS0 at MMIO 0x4ef600300 (irq = 18) is a 16550A
4ef600400.serial: ttyS1 at MMIO 0x4ef600400 (irq = 19) is a 16550A
4ef600500.serial: ttyS2 at MMIO 0x4ef600500 (irq = 29) is a 16550A
4ef600600.serial: ttyS3 at MMIO 0x4ef600600 (irq = 20) is a 16550A
brd: module loaded
PPC 4xx OCP EMAC driver, version 3.54
MAL v2 /plb/mcmal, 2 TX channels, 16 RX channels
ZMII /plb/opb/emac-zmii@ef600d00 initialized
RGMII /plb/opb/emac-rgmii@ef601500 initialized with MDIO support
TAH /plb/opb/emac-tah@ef601350 initialized
TAH /plb/opb/emac-tah@ef601450 initialized
/plb/opb/emac-rgmii@ef601500: input 0 in RGMII mode
eth0 (emac): not using net_device_ops yet
eth0: EMAC-0 /plb/opb/ethernet@ef600e00, MAC 00:10:ec:01:06:54
eth0: found Generic MII PHY (0x00)
/plb/opb/emac-rgmii@ef601500: input 1 in RGMII mode
eth1 (emac): not using net_device_ops yet
eth1: EMAC-1 /plb/opb/ethernet@ef600f00, MAC 00:10:ec:81:06:54
eth1: found Generic MII PHY (0x01)
TCP cubic registered
NET: Registered protocol family 17
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
eth0: link is down
IP-Config: Guessing netmask 255.255.0.0
IP-Config: Complete:
device=eth0, addr=172.16.0.46, mask=255.255.0.0, gw=255.255.255.255,
host=canyonlands, domain=, nis-domain=(none),
bootserver=172.16.0.50, rootserver=172.16.0.50, rootpath=
Looking up port of RPC 100003/2 on 172.16.0.50
eth0: link is up, 1000 FDX, pause enabled
Looking up port of RPC 100005/1 on 172.16.0.50
VFS: Mounted root (nfs filesystem).
Freeing unused kernel memory: 144k init
lspci -v
0000:40:00.0 Class 0604: Unknown device aaa0:bed0 (rev 01)
Flags: bus master, fast devsel, latency 0
Memory at <unassigned> (32-bit, prefetchable)
Bus: primary=40, secondary=41, subordinate=7f, sec-latency=0
Capabilities: [7c] Vital Product Data
Capabilities: [40] Power Management version 3
Capabilities: [48] Message Signalled Interrupts: Mask- 64bit+ Queue=0/2 Enable-
Capabilities: [58] Express Root Port (Slot-) IRQ 0
Capabilities: [100] Advanced Error Reporting
Capabilities: [1f8] Unknown (11)
0001:80:00.0 Class 0604: Unknown device aaa1:bed1 (rev 01)
Flags: bus master, fast devsel, latency 0
Memory at <unassigned> (32-bit, prefetchable)
Bus: primary=80, secondary=81, subordinate=bf, sec-latency=0
Capabilities: [7c] Vital Product Data
Capabilities: [40] Power Management version 3
Capabilities: [48] Message Signalled Interrupts: Mask- 64bit+ Queue=0/2 Enable-
Capabilities: [58] Express Root Port (Slot-) IRQ 0
Capabilities: [100] Advanced Error Reporting
Capabilities: [1f8] Unknown (11)
