On Fri, Nov 06, 2009 at 02:47:54PM -0600, Bjorn Helgaas wrote:
> On Friday 06 November 2009 12:57:17 pm Ira W. Snyder wrote:
> > On Thu, Nov 05, 2009 at 06:34:56PM -0600, Bjorn Helgaas wrote:
> >
> > [ big snip ]
> >
> > >
> > > Of course, this doesn't help if you actually power off the slot or
> > > press your reset button. In that case, we have to assign things from
> > > scratch, and we really need to know what the host bridge apertures are.
> > > It'd be best if the BIOS gave us an ACPI description, so we could just
> > > use "pci=use_crs". Without that, I think we'll need a host bridge
> > > driver to read them out of the hardware, and it sounds like we don't
> > > have the specs required to write such a driver.
> > >
> >
> > What would actually be needed to write this driver? I ran an the command
> > "lspci -MH1 -vv -xxx", which I found in a very old email. It manages to
> > dump out all 0x100 bytes of configuration space on the bridges. Using
> > this and lspci, it is clear what memory lives behind the bridge, I
> > think.
> >
> > Here is a dump from my Force computer's Broadcom host bridges. Look
> > closely at the 4 16-bit registers starting at 0xC0. If you correlate
> > them with the BAR dump, shown below, it shows:
> >
> > Bridge 00:00.0
> > non-prefetchable memory from 0xf9000000 - 0xf93fffff
> > prefectchable memory from 0xf9400000 - 0xf96fffff
> >
> > Bridge 00:00.1
> > non-prefetchable memory from 0xf98c0000 - 0xfaffffff
> > prefectchable memory from NONE PRESENT
>
> Hey, this is cool! Nice detective work :-)
>
> There should also be a bus number range: it looks like the 00.0
> host bridge has bus 00 below it, and 00.1 has at least buses 01-02
> below it.
>
Ok, these are registers 0x44 and 0x45. I dug this out of an ancient post
by Linus to LKML from the 2.4 days. I have a number of these CPU's in
the lab, one of them has lots of PCI bridges. They all agree with the
"lspci -t" output :)
http://lkml.indiana.edu/hypermail/linux/kernel/0102.1/0137.html
> And there should be an I/O port range. Your Freescale device doesn't
> use any I/O ports, but presumably you could plug in a card that did,
> and the BIOS should allocate space for it.
>
Yep, I've got boards with I/O ports. These appear to be at 0xd0. They
work pretty much the same as the memory numbers, except there are no
implicit 0x0000 or 0xFFFF. The first 16-bit value is the starting port,
the second 16-bit value is the last port.
> That's basically all the info we should need to write a driver. There's
> not really a good model for one yet because we don't have any kind of
> generic PCI host bridge structure. We do have drivers/acpi/pci_root.c
> and arch/x86/pci/amd_bus.c, which are two different approaches to this.
> amd_bus.c is closer to what we need here, but it's not well integrated
> into PCI, so I don't really want to add more stuff like it.
>
I think arch/x86/pci/intel_bus.c is another example.
I'd be happy to test a quick and dirty driver along their lines in the
meantime, to make sure we've got everything right.
I've included the information I've figured out now in the dumps below,
and I've attached a dump from the "many bridges" system.
> > 00:00.0 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
> > Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
> > Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
> > Latency: 48, Cache Line Size: 32 bytes
> > 00: 66 11 09 00 06 01 00 22 06 00 00 06 08 30 80 00
> > 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 40: 00 00 00 00 00 01 04 00 81 14 00 2f 00 00 00 6e
First busno: 00
Last busno: 01
> > 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 70: 20 aa 80 aa 00 00 00 00 01 0c 81 00 88 00 00 00
> > 80: 00 10 10 20 00 00 00 00 00 00 00 00 00 00 00 00
> > 90: ff 00 09 10 ff 6f 70 31 00 00 00 00 00 00 00 00
> > a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > b0: cf ff fc fb ff df ee ff bf dd d7 fe bf f7 ff df
> > c0: 00 f9 3f f9 40 f9 6f f9 00 00 00 00 0d 07 00 48
non-prefetchable mem: 0xf9000000 - 0xf93fffff
prefetchable mem: 0xf9400000 - 0xf96fffff
> > d0: 00 10 0c 14 00 00 00 00 00 00 00 00 01 80 10 93
I/O ports: 0x1000 - 0x140c
> > e0: 04 90 00 07 00 00 00 00 81 00 00 00 00 00 00 00
> > f0: 48 45 00 00 00 00 00 00 00 00 00 00 60 00 00 00
> >
> > 00:00.1 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
> > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
> > Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
> > Latency: 64, Cache Line Size: 32 bytes
> > 00: 66 11 09 00 07 01 00 02 06 00 00 06 08 40 80 00
> > 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 40: 00 00 00 00 02 02 00 00 00 00 00 4b 00 00 00 00
First busno: 02
Last busno: 02
> > 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > c0: 8c f9 ff fa 00 00 00 00 00 00 00 00 01 00 00 00
non-prefetchable mem: 0xf8c00000 - 0xfaffffff
prefetchable mem: NONE
> > d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
I/O ports: NONE
> > e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > f0: 80 81 19 09 00 00 00 00 00 00 00 00 61 00 00 00
> >
> > And here is the full listing of PCI BAR addresses:
> > 00:00.0 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
> > Flags: bus master, medium devsel, latency 48
> >
> > 00:00.1 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
> > Flags: bus master, medium devsel, latency 64
> >
> > 00:01.0 Ethernet controller: Intel Corporation 8255xER/82551IT Fast Ethernet Controller (rev 09)
> > Subsystem: Intel Corporation Device 100c
> > Flags: bus master, medium devsel, latency 66, IRQ 11
> > Memory at f9020000 (32-bit, non-prefetchable) [size=4K]
> > I/O ports at 1080 [size=64]
> > Memory at f9000000 (32-bit, non-prefetchable) [size=128K]
> > [virtual] Expansion ROM at 20000000 [disabled] [size=1M]
> > Capabilities: <access denied>
> > Kernel driver in use: e100
> >
> > 00:02.0 Ethernet controller: Intel Corporation 8255xER/82551IT Fast Ethernet Controller (rev 09)
> > Subsystem: Intel Corporation Device 100c
> > Flags: bus master, medium devsel, latency 66, IRQ 10
> > Memory at f9021000 (32-bit, non-prefetchable) [size=4K]
> > I/O ports at 10c0 [size=64]
> > Memory at f9040000 (32-bit, non-prefetchable) [size=128K]
> > [virtual] Expansion ROM at 20100000 [disabled] [size=1M]
> > Capabilities: <access denied>
> > Kernel driver in use: e100
> >
> > 00:04.0 PCI bridge: Force Computers Device 0001 (rev 01) (prog-if 00 [Normal decode])
> > Flags: bus master, medium devsel, latency 64
> > Memory at f9022000 (32-bit, non-prefetchable) [size=4K]
> > I/O ports at 1000 [size=128]
> > Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
> > Memory behind bridge: f9100000-f91fffff
> > Prefetchable memory behind bridge: f9400000-f96fffff
> > Capabilities: <access denied>
> >
> > 00:0f.0 ISA bridge: Broadcom OSB4 South Bridge (rev 50)
> > Subsystem: Broadcom OSB4 South Bridge
> > Flags: bus master, medium devsel, latency 0
> > Kernel driver in use: piix4_smbus
> >
> > 00:0f.1 IDE interface: Broadcom OSB4 IDE Controller (prog-if 8a [Master SecP PriP])
> > Flags: medium devsel
> > [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [size=8]
> > [virtual] Memory at 000003f0 (type 3, non-prefetchable) [size=1]
> > [virtual] Memory at 00000170 (32-bit, non-prefetchable) [size=8]
> > [virtual] Memory at 00000370 (type 3, non-prefetchable) [size=1]
> > I/O ports at 1400 [size=16]
> >
> > 00:0f.2 USB Controller: Broadcom OSB4/CSB5 OHCI USB Controller (rev 04) (prog-if 10 [OHCI])
> > Subsystem: Broadcom OSB4/CSB5 OHCI USB Controller
> > Flags: medium devsel, IRQ 7
> > Memory at f9023000 (32-bit, non-prefetchable) [size=4K]
> >
> > 01:0b.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
> > Flags: 66MHz, fast devsel, IRQ 9
> > Memory at f9100000 (32-bit, non-prefetchable) [size=1M]
> > Memory at f9600000 (32-bit, prefetchable) [size=4K]
> > Memory at f9500000 (64-bit, prefetchable) [size=1M]
> > Memory at f9400000 (64-bit, prefetchable) [size=1M]
> > Capabilities: <access denied>
> >
> > 02:03.0 VGA compatible controller: Chips and Technologies F69030 (rev 61) (prog-if 00 [VGA controller])
> > Subsystem: Chips and Technologies F69030
> > Flags: stepping, 66MHz, medium devsel, IRQ 9
> > Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
> > [virtual] Expansion ROM at 20200000 [disabled] [size=256K]
> >
Here is an lspci -t for the above system:
-+-[0000:02]---03.0
\-[0000:00]-+-00.0
+-00.1
+-01.0
+-02.0
+-04.0-[0000:01]----0b.0
+-0f.0
+-0f.1
\-0f.2
Hopefully the above should be enough to get a driver working. Detective work is
fun! :)
Ira
00:00.0 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
Latency: 32, Cache Line Size: 32 bytes
00: 66 11 09 00 06 00 00 22 06 00 00 06 08 20 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
40: 00 00 00 00 00 00 00 00 80 14 00 eb 00 00 00 4a
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 40 2a 00 af aa 0f ff 1f 01 1e 81 00 44 00 00 00
80: 00 20 20 40 00 00 00 00 00 00 00 00 00 00 00 00
90: ff 0a 29 10 c1 6f 52 35 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: ff ff f7 fc b5 ff 7f de fb fb 00 c0 ff 7b fe ff
c0: a0 fb 2f fe a0 fa af fa 48 70 2d 30 0d 07 00 48
d0: 00 b0 fc bf 64 40 a8 a5 a0 01 00 3c 01 80 10 93
e0: 04 9b 20 00 00 00 00 00 cd 11 00 00 00 00 00 00
f0: 91 25 20 0b 00 00 00 00 2d 00 00 00 60 10 00 00
00:00.1 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 16, Cache Line Size: 32 bytes
00: 66 11 09 00 07 00 00 02 06 00 00 06 08 10 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
40: 00 00 00 00 01 03 00 00 00 00 00 0b 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 30 fe bf fe b0 fa 9f fb 40 9b 90 24 0d 07 00 00
d0: 00 c0 fc ef 00 00 00 00 00 00 00 00 01 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 52 02 0c 00 00 00 00 6d 00 00 00 61 10 00 00
Summary of buses:
00: Primary host bus
-+-[0000:01]---04.0-[0000:02-03]--+-04.0-[0000:03]--+-0b.0
| | \-0e.0
| \-0e.0
\-[0000:00]-+-00.0
+-00.1
+-05.0
+-06.0
+-07.0
+-0f.0
+-0f.1
\-0f.2
00:00.0 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
Flags: bus master, medium devsel, latency 32
00:00.1 Host bridge: Broadcom CNB20LE Host Bridge (rev 06)
Flags: bus master, medium devsel, latency 16
00:05.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 08)
Flags: bus master, medium devsel, latency 64, IRQ 17
Memory at fe0bd000 (32-bit, non-prefetchable) [size=4K]
I/O ports at be80 [size=64]
Memory at fcd00000 (32-bit, non-prefetchable) [size=1M]
Expansion ROM at fcc00000 [disabled] [size=1M]
Capabilities: [dc] Power Management version 2
Kernel driver in use: e100
00:06.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 08)
Flags: bus master, medium devsel, latency 64, IRQ 18
Memory at fe0be000 (32-bit, non-prefetchable) [size=4K]
I/O ports at bf00 [size=64]
Memory at fcf00000 (32-bit, non-prefetchable) [size=1M]
Expansion ROM at fce00000 [disabled] [size=1M]
Capabilities: [dc] Power Management version 2
Kernel driver in use: e100
00:07.0 VGA compatible controller: Chips and Technologies F69030 (rev 61) (prog-if 00 [VGA controller])
Subsystem: Chips and Technologies F69030
Flags: stepping, 66MHz, medium devsel, IRQ 11
Memory at fd000000 (32-bit, non-prefetchable) [size=16M]
Expansion ROM at fe0c0000 [disabled] [size=256K]
00:0f.0 ISA bridge: Broadcom OSB4 South Bridge (rev 50)
Subsystem: Broadcom OSB4 South Bridge
Flags: bus master, medium devsel, latency 0
Kernel driver in use: piix4_smbus
00:0f.1 IDE interface: Broadcom OSB4 IDE Controller (prog-if 8a [Master SecP PriP])
Flags: bus master, medium devsel, latency 64
[virtual] Memory at 000001f0 (32-bit, non-prefetchable) [size=8]
[virtual] Memory at 000003f0 (type 3, non-prefetchable) [size=1]
[virtual] Memory at 00000170 (32-bit, non-prefetchable) [size=8]
[virtual] Memory at 00000370 (type 3, non-prefetchable) [size=1]
I/O ports at ffa0 [size=16]
00:0f.2 USB Controller: Broadcom OSB4/CSB5 OHCI USB Controller (rev 04) (prog-if 10 [OHCI])
Subsystem: Broadcom OSB4/CSB5 OHCI USB Controller
Flags: bus master, medium devsel, latency 64, IRQ 10
Memory at fe0bf000 (32-bit, non-prefetchable) [size=4K]
01:04.0 PCI bridge: Intel Corporation 21154 PCI-to-PCI Bridge (prog-if 00 [Normal decode])
Flags: bus master, 66MHz, medium devsel, latency 64
Bus: primary=01, secondary=02, subordinate=03, sec-latency=64
I/O behind bridge: 0000c000-0000dfff
Memory behind bridge: fe300000-feafffff
Prefetchable memory behind bridge: 00000000fab00000-00000000fb8fffff
Capabilities: [dc] Power Management version 1
02:04.0 PCI bridge: Intel Corporation 21154 PCI-to-PCI Bridge (prog-if 00 [Normal decode])
Flags: bus master, 66MHz, medium devsel, latency 64
Bus: primary=02, secondary=03, subordinate=03, sec-latency=64
I/O behind bridge: 0000c000-0000cfff
Memory behind bridge: fe300000-fe7fffff
Prefetchable memory behind bridge: 00000000fab00000-00000000fb3fffff
Capabilities: [dc] Power Management version 1
02:0e.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
Flags: bus master, 66MHz, fast devsel, latency 64, IRQ 22
Memory at fea00000 (32-bit, non-prefetchable) [size=1M]
Memory at fb8ff000 (32-bit, prefetchable) [size=4K]
Memory at fb700000 (64-bit, prefetchable) [size=1M]
Memory at fb600000 (64-bit, prefetchable) [size=1M]
Capabilities: [48] CompactPCI hot-swap <?>
Kernel driver in use: pnet-host
03:0b.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
Flags: bus master, 66MHz, fast devsel, latency 64, IRQ 23
Memory at fe700000 (32-bit, non-prefetchable) [size=1M]
Memory at fb3ff000 (32-bit, prefetchable) [size=4K]
Memory at fb200000 (64-bit, prefetchable) [size=1M]
Memory at fb100000 (64-bit, prefetchable) [size=1M]
Capabilities: [48] CompactPCI hot-swap <?>
Kernel driver in use: pnet-host
03:0e.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
Flags: bus master, 66MHz, fast devsel, latency 64, IRQ 22
Memory at fe600000 (32-bit, non-prefetchable) [size=1M]
Memory at fb3fe000 (32-bit, prefetchable) [size=4K]
Memory at fb000000 (64-bit, prefetchable) [size=1M]
Memory at faf00000 (64-bit, prefetchable) [size=1M]
Capabilities: [48] CompactPCI hot-swap <?>
Kernel driver in use: pnet-host
