The kernel source code is 4.4. I have not done any modification except
pre-allocated bus numbers and memory window for a DS port to allow
successful hot plug operation of a PCIe port.
0.611919] acpi PNP0A08:00: host bridge window [io 0x0cf8-0x0cff] (ignored)
[ 0.611921] acpi PNP0A08:00: host bridge window [io 0x0000-0x0cf7
window] (ignored)
[ 0.611922] acpi PNP0A08:00: host bridge window [io 0x1000-0x7fff
window] (ignored)
[ 0.611924] acpi PNP0A08:00: host bridge window [mem
0x000a0000-0x000bffff window] (ignored)
[ 0.611926] acpi PNP0A08:00: host bridge window [mem
0xfedb0000-0xfedb000f window] (ignored)
[ 0.611928] acpi PNP0A08:00: host bridge window [mem
0xfedc0000-0xfedc000f window] (ignored)
[ 0.611929] acpi PNP0A08:00: host bridge window [mem
0x90000000-0xc7ffbfff window] (ignored)
[ 0.611931] acpi PNP0A08:00: host bridge window [mem
0x380000000000-0x383fffffffff window] (ignored)
[ 0.611932] PCI: root bus 00: using default resources
[ 0.612217] PCI host bridge to bus 0000:00
[ 0.612219] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 0.612220] pci_bus 0000:00: root bus resource [mem
0x00000000-0x3fffffffffff]
[ 0.612222] pci_bus 0000:00: root bus resource [bus 00-7e]
kernel command line:
BOOT_IMAGE=/boot/vmlinuz-4.4.8-kb
root=UUID=c6f326f6-7cc4-49f5-a3be-c511566165b4 ro
pci=realloc,nocrs,assign-busses pcie_ports=native
pciehp.pciehp_debug=1 pci=hpmemsize=0M console=ttyS1,115200n8 quiet
splash crashkernel=384M-:128M vt.handoff=7
On Mon, Aug 15, 2016 at 12:06 PM, Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote:
> On Mon, Aug 15, 2016 at 11:39:56AM -0700, Kallol Biswas wrote:
>> Thank you for the explanation.
>>
>>
>> Here is the relevant dmesg content:
>>
>> [ 0.790457] PCI: root bus ff: using default resources
>> [ 0.790491] PCI host bridge to bus 0000:ff
>> [ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff]
>> [ 0.790495] pci_bus 0000:ff: root bus resource [mem
>> 0x00000000-0x3fffffffffff]
>> [ 0.790497] pci_bus 0000:ff: root bus resource [bus ff]
>>
>> 0.795379] PCI host bridge to bus 0000:7f
>> [ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff]
>> [ 0.795383] pci_bus 0000:7f: root bus resource [mem
>> 0x00000000-0x3fffffffffff]
>> [ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f]
>>
>> 0.809149] PCI host bridge to bus 0000:00
>> [ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
>> [ 0.809153] pci_bus 0000:00: root bus resource [mem
>> 0x00000000-0x3fffffffffff]
>> [ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e]
>
> This is the host bridge that claims to lead to bus 0000:00.
>
> But the I/O and memory resources of all these bridges are wrong. The
> ranges they show are CPU addresses, not PCI bus addresses, so they
> cannot overlap.
>
> It could be that you're using a non-standard host bridge driver that
> did not apply the translation between CPU addresses and PCI bus
> addresses. In ACPI, this is the _TRA property.
>
>> [ 0.852071] PCI host bridge to bus 0000:80
>> [ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff]
>> [ 0.852075] pci_bus 0000:80: root bus resource [mem
>> 0x00000000-0x3fffffffffff]
>> [ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe]
>>
>> Following lines are from output of /proc/iomem
>>
>> c8000000-c8000000 : <BAD>
>> c8000001-c8000001 : <BAD>
>> c8000002-c8000002 : <BAD>
>> c8000003-c8000003 : <BAD>
>> c8100000-c91fffff : PCI Bus 0000:02
>> c8100000-c90fffff : PCI Bus 0000:03
>> c8100000-c90fffff : PCI Bus 0000:04
>> c8100000-c82fffff : PCI Bus 0000:05
>> c8100000-c81fffff : PCI Bus 0000:06
>> c8100000-c8100fff : 0000:06:00.0
>> c8200000-c82fffff : PCI Bus 0000:0a
>> c8200000-c8203fff : 0000:0a:00.0
>> c9100000-c913ffff : 0000:02:00.0
>> fb000000-fb8fffff : PCI Bus 0000:81
>> fb000000-fb8fffff : PCI Bus 0000:82
>> fb000000-fb7fffff : PCI Bus 0000:8c
>> fb000000-fb7fffff : 0000:8c:00.0
>> fb000000-fb7fffff : Plx_Mgr
>> fb800000-fb8fffff : PCI Bus 0000:87
>> fb800000-fb800fff : 0000:87:00.0
>> fb900000-fb900fff : 0000:80:05.4
>>
>>
>> I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This
>> device is available after reboot, and initial PCIe bus enumeration
>> time.
>>
>> The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug
>> operation is done.
>>
>> Attached is the complete /proc/iomem and do not find the line
>> c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for
>> PCI Bus 0000:80? BAR access works fine through this bus.
>>
>>
>>
>> On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote:
>> > Hi Kallol,
>> >
>> > Sorry I didn't respond to this earlier; I was on vacation for most of
>> > last week.
>> >
>> > On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote:
>> >> I am trying to debug an issue that the PCIe memory
>> >> transactions are not routed through a host bridge.
>> >>
>> >> The config transactions work fine. The primary, secondary and
>> >> subordinate bus numbers look fine for the PCIe tree under the host
>> >> bridge. The memory windows are programmed fine.
>> >>
>> >> Can someone point me to the code that programs the chipset/root port
>> >> registers those direct processor transactions to a specific host
>> >> bridge?
>> >
>> > It looks like you have an x86 system with ACPI. In that case, we use
>> > the ACPI pci_root.c host bridge driver. The chipset configuration for
>> > routing processor memory transactions to host bridges is done by the
>> > firmware, and the kernel doesn't touch it. There should be
>> > information in the chipset spec about how to do this, but there's no
>> > code in Linux itself that does it.
>> >
>> >> Cat of /proc/mem is copied below.
>> >>
>> >> /root/physmem -p c8100000 -s 4 -r
>> >> Operation:read to Addr: 0xc8100000 Value: 0
>> >> ffffffff ffffffff ffffffff ffffffff
>> >>
>> >> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and
>> >> external switches/devices does not capture any mem transactions.
>> >> However, config transactions are visible in the trace.
>> >
>> > 00:03.0 is not a host bridge; it's a Root Port, which logically
>> > connects two PCI buses. A host bridge logically connects a processor
>> > bus to a PCI bus.
>> >
>> >> root@xlink-target:~# cat /proc/iomem
>> >> 00000000-00000fff : reserved
>> >> 00001000-0009abff : System RAM
>> >> 0009ac00-0009ffff : reserved
>> >> 000c0000-000c7fff : Video ROM
>> >> 000c8000-000c8fff : Adapter ROM
>> >> 000e0000-000fffff : reserved
>> >> 000f0000-000fffff : System ROM
>> >> 00100000-78f36fff : System RAM
>> >> 01000000-01828f14 : Kernel code
>> >> 01828f15-01f3fb3f : Kernel data
>> >> 020bb000-021fdfff : Kernel bss
>> >> 30000000-37ffffff : Crash kernel
>> >> 78f37000-79839fff : reserved
>> >> 79116018-79116018 : APEI ERST
>> >> 7911601c-79116021 : APEI ERST
>> >> 79116028-79116039 : APEI ERST
>> >> 79116040-7911604c : APEI ERST
>> >> 79116050-7911804f : APEI ERST
>> >> 7983a000-79d4afff : ACPI Non-volatile Storage
>> >> 79d4b000-8fffffff : reserved
>> >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
>> >> c6000000-c70fffff : PCI Bus 0000:2f
>> >> c6000000-c70fffff : PCI Bus 0000:30
>> >> c6000000-c6ffffff : 0000:30:00.0
>> >> c7000000-c701ffff : 0000:30:00.0
>> >> c7100000-c71fffff : PCI Bus 0000:31
>> >> c7100000-c711ffff : 0000:31:00.1
>> >> c7100000-c711ffff : igb
>> >> c7120000-c713ffff : 0000:31:00.0
>> >> c7120000-c713ffff : igb
>> >> c7140000-c7143fff : 0000:31:00.1
>> >> c7140000-c7143fff : igb
>> >> c7144000-c7147fff : 0000:31:00.0
>> >> c7144000-c7147fff : igb
>> >> c7300000-c730ffff : 0000:00:14.0
>> >> c7300000-c730ffff : xhci-hcd
>> >> c7310000-c73107ff : 0000:00:1f.2
>> >> c7310000-c73107ff : ahci
>> >> c7311000-c73113ff : 0000:00:1d.0
>> >> c7311000-c73113ff : ehci_hcd
>> >> c7312000-c73123ff : 0000:00:1a.0
>> >> c7312000-c73123ff : ehci_hcd
>> >> c7314000-c731400f : 0000:00:16.1
>> >> c7315000-c731500f : 0000:00:16.0
>> >> c7316000-c73167ff : 0000:00:11.4
>> >> c7316000-c73167ff : ahci
>> >> c7317000-c7317fff : 0000:00:05.4
>> >> c7ffc000-c7ffcfff : dmar1
>> >> c8000000-c8000000 : <BAD>
>> >> c8000001-c8000001 : <BAD>
>> >> c8000002-c8000002 : <BAD>
>> >> c8000003-c8000003 : <BAD>
>> >
>> > Interesting, what are these from?
>> >
>> >> c8100000-c91fffff : PCI Bus 0000:02
>> >> c8100000-c90fffff : PCI Bus 0000:03
>> >> c8100000-c90fffff : PCI Bus 0000:04
>> >> c8100000-c82fffff : PCI Bus 0000:05
>> >> c8100000-c81fffff : PCI Bus 0000:06
>> >> c8100000-c8100fff : 0000:06:00.0
>> >
>> > You're trying to read the 06:00.0 BAR at 0xc8100000. The windows
>> > above look fine, but the top-level host bridge window is missing.
>> >
>> > The c8100000-c91fffff window is a window through the 00:03.0 Root
>> > Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards
>> > memory transactions in the c8100000-c91fffff range from its primary
>> > bus (bus 00) to its secondary bus (bus 02).
>> >
>> > But we don't know how those transactions get to bus 00 in the first
>> > place. The host bridge is the connection between the processor and
>> > bus 00, and it should have a window that encloses c8100000-c91fffff.
>> >
>> > Your dmesg log should have information about this bridge, e.g.,
>> > something like this:
>> >
>> > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f])
>> > PCI host bridge to bus 0000:00
>> > pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window]
>> >
>> > and there should be a line in /proc/iomem something like this:
>> >
>> > c8100000-c9ffffff : PCI Bus 0000:00
>> >
>> > Bjorn
>
>> 00000000-00000fff : reserved
>> 00001000-0009abff : System RAM
>> 0009ac00-0009ffff : reserved
>> 000c0000-000c7fff : Video ROM
>> 000c8000-000c8fff : Adapter ROM
>> 000e0000-000fffff : reserved
>> 000f0000-000fffff : System ROM
>> 00100000-78f36fff : System RAM
>> 01000000-01828f14 : Kernel code
>> 01828f15-01f3fb3f : Kernel data
>> 020bb000-021fdfff : Kernel bss
>> 30000000-37ffffff : Crash kernel
>> 78f37000-79839fff : reserved
>> 79116018-79116018 : APEI ERST
>> 7911601c-79116021 : APEI ERST
>> 79116028-79116039 : APEI ERST
>> 79116040-7911604c : APEI ERST
>> 79116050-7911804f : APEI ERST
>> 7983a000-79d4afff : ACPI Non-volatile Storage
>> 79d4b000-8fffffff : reserved
>> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
>> c6000000-c70fffff : PCI Bus 0000:2f
>> c6000000-c70fffff : PCI Bus 0000:30
>> c6000000-c6ffffff : 0000:30:00.0
>> c7000000-c701ffff : 0000:30:00.0
>> c7100000-c71fffff : PCI Bus 0000:31
>> c7100000-c711ffff : 0000:31:00.1
>> c7100000-c711ffff : igb
>> c7120000-c713ffff : 0000:31:00.0
>> c7120000-c713ffff : igb
>> c7140000-c7143fff : 0000:31:00.1
>> c7140000-c7143fff : igb
>> c7144000-c7147fff : 0000:31:00.0
>> c7144000-c7147fff : igb
>> c7300000-c730ffff : 0000:00:14.0
>> c7300000-c730ffff : xhci-hcd
>> c7310000-c73107ff : 0000:00:1f.2
>> c7310000-c73107ff : ahci
>> c7311000-c73113ff : 0000:00:1d.0
>> c7311000-c73113ff : ehci_hcd
>> c7312000-c73123ff : 0000:00:1a.0
>> c7312000-c73123ff : ehci_hcd
>> c7314000-c731400f : 0000:00:16.1
>> c7315000-c731500f : 0000:00:16.0
>> c7316000-c73167ff : 0000:00:11.4
>> c7316000-c73167ff : ahci
>> c7317000-c7317fff : 0000:00:05.4
>> c7ffc000-c7ffcfff : dmar1
>> c8000000-c8000000 : <BAD>
>> c8000001-c8000001 : <BAD>
>> c8000002-c8000002 : <BAD>
>> c8000003-c8000003 : <BAD>
>> c8100000-c91fffff : PCI Bus 0000:02
>> c8100000-c90fffff : PCI Bus 0000:03
>> c8100000-c90fffff : PCI Bus 0000:04
>> c8100000-c82fffff : PCI Bus 0000:05
>> c8100000-c81fffff : PCI Bus 0000:06
>> c8100000-c8100fff : 0000:06:00.0
>> c8200000-c82fffff : PCI Bus 0000:0a
>> c8200000-c8203fff : 0000:0a:00.0
>> c9100000-c913ffff : 0000:02:00.0
>> fb000000-fb8fffff : PCI Bus 0000:81
>> fb000000-fb8fffff : PCI Bus 0000:82
>> fb000000-fb7fffff : PCI Bus 0000:8c
>> fb000000-fb7fffff : 0000:8c:00.0
>> fb000000-fb7fffff : Plx_Mgr
>> fb800000-fb8fffff : PCI Bus 0000:87
>> fb800000-fb800fff : 0000:87:00.0
>> fb900000-fb900fff : 0000:80:05.4
>> fbffc000-fbffcfff : dmar0
>> fec00000-fecfffff : PNP0003:00
>> fec00000-fec003ff : IOAPIC 0
>> fec01000-fec013ff : IOAPIC 1
>> fec40000-fec403ff : IOAPIC 2
>> fed00000-fed003ff : HPET 0
>> fed00000-fed003ff : PNP0103:00
>> fed12000-fed1200f : pnp 00:01
>> fed12010-fed1201f : pnp 00:01
>> fed1b000-fed1bfff : pnp 00:01
>> fed1c000-fed44fff : reserved
>> fed1c000-fed3ffff : pnp 00:01
>> fed1f410-fed1f414 : iTCO_wdt.0.auto
>> fed45000-fed8bfff : pnp 00:01
>> fee00000-feefffff : pnp 00:01
>> fee00000-fee00fff : Local APIC
>> ff000000-ffffffff : reserved
>> ff000000-ffffffff : pnp 00:01
>> 100000000-47fffffff : System RAM
>> 480000000-4807fffff : PCI Bus 0000:02
>> 480000000-4807fffff : PCI Bus 0000:03
>> 480000000-4807fffff : PCI Bus 0000:04
>> 480000000-4800fffff : PCI Bus 0000:05
>> 480000000-4800fffff : PCI Bus 0000:06
>> 480000000-480003fff : 0000:06:00.0
>> 383fffe00000-383fffefffff : PCI Bus 0000:31
>> 383fffe00000-383fffe1ffff : 0000:31:00.1
>> 383fffe20000-383fffe3ffff : 0000:31:00.1
>> 383fffe40000-383fffe5ffff : 0000:31:00.0
>> 383fffe60000-383fffe7ffff : 0000:31:00.0
>> 383ffff00000-383ffff03fff : 0000:00:04.7
>> 383ffff00000-383ffff03fff : ioatdma
>> 383ffff04000-383ffff07fff : 0000:00:04.6
>> 383ffff04000-383ffff07fff : ioatdma
>> 383ffff08000-383ffff0bfff : 0000:00:04.5
>> 383ffff08000-383ffff0bfff : ioatdma
>> 383ffff0c000-383ffff0ffff : 0000:00:04.4
>> 383ffff0c000-383ffff0ffff : ioatdma
>> 383ffff10000-383ffff13fff : 0000:00:04.3
>> 383ffff10000-383ffff13fff : ioatdma
>> 383ffff14000-383ffff17fff : 0000:00:04.2
>> 383ffff14000-383ffff17fff : ioatdma
>> 383ffff18000-383ffff1bfff : 0000:00:04.1
>> 383ffff18000-383ffff1bfff : ioatdma
>> 383ffff1c000-383ffff1ffff : 0000:00:04.0
>> 383ffff1c000-383ffff1ffff : ioatdma
>> 383ffff21000-383ffff210ff : 0000:00:1f.3
>> 387fffe00000-387fffefffff : PCI Bus 0000:81
>> 387fffe00000-387fffefffff : PCI Bus 0000:82
>> 387fffe00000-387fffefffff : PCI Bus 0000:87
>> 387fffe00000-387fffe03fff : 0000:87:00.0
>> 387ffff00000-387ffff03fff : 0000:80:04.7
>> 387ffff00000-387ffff03fff : ioatdma
>> 387ffff04000-387ffff07fff : 0000:80:04.6
>> 387ffff04000-387ffff07fff : ioatdma
>> 387ffff08000-387ffff0bfff : 0000:80:04.5
>> 387ffff08000-387ffff0bfff : ioatdma
>> 387ffff0c000-387ffff0ffff : 0000:80:04.4
>> 387ffff0c000-387ffff0ffff : ioatdma
>> 387ffff10000-387ffff13fff : 0000:80:04.3
>> 387ffff10000-387ffff13fff : ioatdma
>> 387ffff14000-387ffff17fff : 0000:80:04.2
>> 387ffff14000-387ffff17fff : ioatdma
>> 387ffff18000-387ffff1bfff : 0000:80:04.1
>> 387ffff18000-387ffff1bfff : ioatdma
>> 387ffff1c000-387ffff1ffff : 0000:80:04.0
>> 387ffff1c000-387ffff1ffff : ioatdma
>
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
