Thank you for the explanation.
Here is the relevant dmesg content:
[ 0.790457] PCI: root bus ff: using default resources
[ 0.790491] PCI host bridge to bus 0000:ff
[ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff]
[ 0.790495] pci_bus 0000:ff: root bus resource [mem
0x00000000-0x3fffffffffff]
[ 0.790497] pci_bus 0000:ff: root bus resource [bus ff]
0.795379] PCI host bridge to bus 0000:7f
[ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff]
[ 0.795383] pci_bus 0000:7f: root bus resource [mem
0x00000000-0x3fffffffffff]
[ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f]
0.809149] PCI host bridge to bus 0000:00
[ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 0.809153] pci_bus 0000:00: root bus resource [mem
0x00000000-0x3fffffffffff]
[ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e]
[ 0.852071] PCI host bridge to bus 0000:80
[ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff]
[ 0.852075] pci_bus 0000:80: root bus resource [mem
0x00000000-0x3fffffffffff]
[ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe]
Following lines are from output of /proc/iomem
c8000000-c8000000 : <BAD>
c8000001-c8000001 : <BAD>
c8000002-c8000002 : <BAD>
c8000003-c8000003 : <BAD>
c8100000-c91fffff : PCI Bus 0000:02
c8100000-c90fffff : PCI Bus 0000:03
c8100000-c90fffff : PCI Bus 0000:04
c8100000-c82fffff : PCI Bus 0000:05
c8100000-c81fffff : PCI Bus 0000:06
c8100000-c8100fff : 0000:06:00.0
c8200000-c82fffff : PCI Bus 0000:0a
c8200000-c8203fff : 0000:0a:00.0
c9100000-c913ffff : 0000:02:00.0
fb000000-fb8fffff : PCI Bus 0000:81
fb000000-fb8fffff : PCI Bus 0000:82
fb000000-fb7fffff : PCI Bus 0000:8c
fb000000-fb7fffff : 0000:8c:00.0
fb000000-fb7fffff : Plx_Mgr
fb800000-fb8fffff : PCI Bus 0000:87
fb800000-fb800fff : 0000:87:00.0
fb900000-fb900fff : 0000:80:05.4
I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This
device is available after reboot, and initial PCIe bus enumeration
time.
The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug
operation is done.
Attached is the complete /proc/iomem and do not find the line
c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for
PCI Bus 0000:80? BAR access works fine through this bus.
On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote:
> Hi Kallol,
>
> Sorry I didn't respond to this earlier; I was on vacation for most of
> last week.
>
> On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote:
>> I am trying to debug an issue that the PCIe memory
>> transactions are not routed through a host bridge.
>>
>> The config transactions work fine. The primary, secondary and
>> subordinate bus numbers look fine for the PCIe tree under the host
>> bridge. The memory windows are programmed fine.
>>
>> Can someone point me to the code that programs the chipset/root port
>> registers those direct processor transactions to a specific host
>> bridge?
>
> It looks like you have an x86 system with ACPI. In that case, we use
> the ACPI pci_root.c host bridge driver. The chipset configuration for
> routing processor memory transactions to host bridges is done by the
> firmware, and the kernel doesn't touch it. There should be
> information in the chipset spec about how to do this, but there's no
> code in Linux itself that does it.
>
>> Cat of /proc/mem is copied below.
>>
>> /root/physmem -p c8100000 -s 4 -r
>> Operation:read to Addr: 0xc8100000 Value: 0
>> ffffffff ffffffff ffffffff ffffffff
>>
>> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and
>> external switches/devices does not capture any mem transactions.
>> However, config transactions are visible in the trace.
>
> 00:03.0 is not a host bridge; it's a Root Port, which logically
> connects two PCI buses. A host bridge logically connects a processor
> bus to a PCI bus.
>
>> root@xlink-target:~# cat /proc/iomem
>> 00000000-00000fff : reserved
>> 00001000-0009abff : System RAM
>> 0009ac00-0009ffff : reserved
>> 000c0000-000c7fff : Video ROM
>> 000c8000-000c8fff : Adapter ROM
>> 000e0000-000fffff : reserved
>> 000f0000-000fffff : System ROM
>> 00100000-78f36fff : System RAM
>> 01000000-01828f14 : Kernel code
>> 01828f15-01f3fb3f : Kernel data
>> 020bb000-021fdfff : Kernel bss
>> 30000000-37ffffff : Crash kernel
>> 78f37000-79839fff : reserved
>> 79116018-79116018 : APEI ERST
>> 7911601c-79116021 : APEI ERST
>> 79116028-79116039 : APEI ERST
>> 79116040-7911604c : APEI ERST
>> 79116050-7911804f : APEI ERST
>> 7983a000-79d4afff : ACPI Non-volatile Storage
>> 79d4b000-8fffffff : reserved
>> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
>> c6000000-c70fffff : PCI Bus 0000:2f
>> c6000000-c70fffff : PCI Bus 0000:30
>> c6000000-c6ffffff : 0000:30:00.0
>> c7000000-c701ffff : 0000:30:00.0
>> c7100000-c71fffff : PCI Bus 0000:31
>> c7100000-c711ffff : 0000:31:00.1
>> c7100000-c711ffff : igb
>> c7120000-c713ffff : 0000:31:00.0
>> c7120000-c713ffff : igb
>> c7140000-c7143fff : 0000:31:00.1
>> c7140000-c7143fff : igb
>> c7144000-c7147fff : 0000:31:00.0
>> c7144000-c7147fff : igb
>> c7300000-c730ffff : 0000:00:14.0
>> c7300000-c730ffff : xhci-hcd
>> c7310000-c73107ff : 0000:00:1f.2
>> c7310000-c73107ff : ahci
>> c7311000-c73113ff : 0000:00:1d.0
>> c7311000-c73113ff : ehci_hcd
>> c7312000-c73123ff : 0000:00:1a.0
>> c7312000-c73123ff : ehci_hcd
>> c7314000-c731400f : 0000:00:16.1
>> c7315000-c731500f : 0000:00:16.0
>> c7316000-c73167ff : 0000:00:11.4
>> c7316000-c73167ff : ahci
>> c7317000-c7317fff : 0000:00:05.4
>> c7ffc000-c7ffcfff : dmar1
>> c8000000-c8000000 : <BAD>
>> c8000001-c8000001 : <BAD>
>> c8000002-c8000002 : <BAD>
>> c8000003-c8000003 : <BAD>
>
> Interesting, what are these from?
>
>> c8100000-c91fffff : PCI Bus 0000:02
>> c8100000-c90fffff : PCI Bus 0000:03
>> c8100000-c90fffff : PCI Bus 0000:04
>> c8100000-c82fffff : PCI Bus 0000:05
>> c8100000-c81fffff : PCI Bus 0000:06
>> c8100000-c8100fff : 0000:06:00.0
>
> You're trying to read the 06:00.0 BAR at 0xc8100000. The windows
> above look fine, but the top-level host bridge window is missing.
>
> The c8100000-c91fffff window is a window through the 00:03.0 Root
> Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards
> memory transactions in the c8100000-c91fffff range from its primary
> bus (bus 00) to its secondary bus (bus 02).
>
> But we don't know how those transactions get to bus 00 in the first
> place. The host bridge is the connection between the processor and
> bus 00, and it should have a window that encloses c8100000-c91fffff.
>
> Your dmesg log should have information about this bridge, e.g.,
> something like this:
>
> ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f])
> PCI host bridge to bus 0000:00
> pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window]
>
> and there should be a line in /proc/iomem something like this:
>
> c8100000-c9ffffff : PCI Bus 0000:00
>
> Bjorn
00000000-00000fff : reserved
00001000-0009abff : System RAM
0009ac00-0009ffff : reserved
000c0000-000c7fff : Video ROM
000c8000-000c8fff : Adapter ROM
000e0000-000fffff : reserved
000f0000-000fffff : System ROM
00100000-78f36fff : System RAM
01000000-01828f14 : Kernel code
01828f15-01f3fb3f : Kernel data
020bb000-021fdfff : Kernel bss
30000000-37ffffff : Crash kernel
78f37000-79839fff : reserved
79116018-79116018 : APEI ERST
7911601c-79116021 : APEI ERST
79116028-79116039 : APEI ERST
79116040-7911604c : APEI ERST
79116050-7911804f : APEI ERST
7983a000-79d4afff : ACPI Non-volatile Storage
79d4b000-8fffffff : reserved
80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
c6000000-c70fffff : PCI Bus 0000:2f
c6000000-c70fffff : PCI Bus 0000:30
c6000000-c6ffffff : 0000:30:00.0
c7000000-c701ffff : 0000:30:00.0
c7100000-c71fffff : PCI Bus 0000:31
c7100000-c711ffff : 0000:31:00.1
c7100000-c711ffff : igb
c7120000-c713ffff : 0000:31:00.0
c7120000-c713ffff : igb
c7140000-c7143fff : 0000:31:00.1
c7140000-c7143fff : igb
c7144000-c7147fff : 0000:31:00.0
c7144000-c7147fff : igb
c7300000-c730ffff : 0000:00:14.0
c7300000-c730ffff : xhci-hcd
c7310000-c73107ff : 0000:00:1f.2
c7310000-c73107ff : ahci
c7311000-c73113ff : 0000:00:1d.0
c7311000-c73113ff : ehci_hcd
c7312000-c73123ff : 0000:00:1a.0
c7312000-c73123ff : ehci_hcd
c7314000-c731400f : 0000:00:16.1
c7315000-c731500f : 0000:00:16.0
c7316000-c73167ff : 0000:00:11.4
c7316000-c73167ff : ahci
c7317000-c7317fff : 0000:00:05.4
c7ffc000-c7ffcfff : dmar1
c8000000-c8000000 : <BAD>
c8000001-c8000001 : <BAD>
c8000002-c8000002 : <BAD>
c8000003-c8000003 : <BAD>
c8100000-c91fffff : PCI Bus 0000:02
c8100000-c90fffff : PCI Bus 0000:03
c8100000-c90fffff : PCI Bus 0000:04
c8100000-c82fffff : PCI Bus 0000:05
c8100000-c81fffff : PCI Bus 0000:06
c8100000-c8100fff : 0000:06:00.0
c8200000-c82fffff : PCI Bus 0000:0a
c8200000-c8203fff : 0000:0a:00.0
c9100000-c913ffff : 0000:02:00.0
fb000000-fb8fffff : PCI Bus 0000:81
fb000000-fb8fffff : PCI Bus 0000:82
fb000000-fb7fffff : PCI Bus 0000:8c
fb000000-fb7fffff : 0000:8c:00.0
fb000000-fb7fffff : Plx_Mgr
fb800000-fb8fffff : PCI Bus 0000:87
fb800000-fb800fff : 0000:87:00.0
fb900000-fb900fff : 0000:80:05.4
fbffc000-fbffcfff : dmar0
fec00000-fecfffff : PNP0003:00
fec00000-fec003ff : IOAPIC 0
fec01000-fec013ff : IOAPIC 1
fec40000-fec403ff : IOAPIC 2
fed00000-fed003ff : HPET 0
fed00000-fed003ff : PNP0103:00
fed12000-fed1200f : pnp 00:01
fed12010-fed1201f : pnp 00:01
fed1b000-fed1bfff : pnp 00:01
fed1c000-fed44fff : reserved
fed1c000-fed3ffff : pnp 00:01
fed1f410-fed1f414 : iTCO_wdt.0.auto
fed45000-fed8bfff : pnp 00:01
fee00000-feefffff : pnp 00:01
fee00000-fee00fff : Local APIC
ff000000-ffffffff : reserved
ff000000-ffffffff : pnp 00:01
100000000-47fffffff : System RAM
480000000-4807fffff : PCI Bus 0000:02
480000000-4807fffff : PCI Bus 0000:03
480000000-4807fffff : PCI Bus 0000:04
480000000-4800fffff : PCI Bus 0000:05
480000000-4800fffff : PCI Bus 0000:06
480000000-480003fff : 0000:06:00.0
383fffe00000-383fffefffff : PCI Bus 0000:31
383fffe00000-383fffe1ffff : 0000:31:00.1
383fffe20000-383fffe3ffff : 0000:31:00.1
383fffe40000-383fffe5ffff : 0000:31:00.0
383fffe60000-383fffe7ffff : 0000:31:00.0
383ffff00000-383ffff03fff : 0000:00:04.7
383ffff00000-383ffff03fff : ioatdma
383ffff04000-383ffff07fff : 0000:00:04.6
383ffff04000-383ffff07fff : ioatdma
383ffff08000-383ffff0bfff : 0000:00:04.5
383ffff08000-383ffff0bfff : ioatdma
383ffff0c000-383ffff0ffff : 0000:00:04.4
383ffff0c000-383ffff0ffff : ioatdma
383ffff10000-383ffff13fff : 0000:00:04.3
383ffff10000-383ffff13fff : ioatdma
383ffff14000-383ffff17fff : 0000:00:04.2
383ffff14000-383ffff17fff : ioatdma
383ffff18000-383ffff1bfff : 0000:00:04.1
383ffff18000-383ffff1bfff : ioatdma
383ffff1c000-383ffff1ffff : 0000:00:04.0
383ffff1c000-383ffff1ffff : ioatdma
383ffff21000-383ffff210ff : 0000:00:1f.3
387fffe00000-387fffefffff : PCI Bus 0000:81
387fffe00000-387fffefffff : PCI Bus 0000:82
387fffe00000-387fffefffff : PCI Bus 0000:87
387fffe00000-387fffe03fff : 0000:87:00.0
387ffff00000-387ffff03fff : 0000:80:04.7
387ffff00000-387ffff03fff : ioatdma
387ffff04000-387ffff07fff : 0000:80:04.6
387ffff04000-387ffff07fff : ioatdma
387ffff08000-387ffff0bfff : 0000:80:04.5
387ffff08000-387ffff0bfff : ioatdma
387ffff0c000-387ffff0ffff : 0000:80:04.4
387ffff0c000-387ffff0ffff : ioatdma
387ffff10000-387ffff13fff : 0000:80:04.3
387ffff10000-387ffff13fff : ioatdma
387ffff14000-387ffff17fff : 0000:80:04.2
387ffff14000-387ffff17fff : ioatdma
387ffff18000-387ffff1bfff : 0000:80:04.1
387ffff18000-387ffff1bfff : ioatdma
387ffff1c000-387ffff1ffff : 0000:80:04.0
387ffff1c000-387ffff1ffff : ioatdma
