> Hello Bjorn, > Now memory access works: The following window has been setup properly: > 90000000-c7ffbfff : PCI Bus 0000:00 > > 9d4b000-8fffffff : reserved > 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] > 90000000-c7ffbfff : PCI Bus 0000:00 > 90000000-910fffff : PCI Bus 0000:02 > 90000000-90ffffff : PCI Bus 0000:03 > 90000000-90ffffff : PCI Bus 0000:04 > 90000000-901fffff : PCI Bus 0000:05 > 90000000-900fffff : PCI Bus 0000:06 > 90000000-90000fff : 0000:06:00.0 > 90000000-90000fff : r8169 > 90100000-901fffff : PCI Bus 0000:0a > 90100000-90103fff : 0000:0a:00.0 > 91000000-9103ffff : 0000:02:00.0 > c6000000-c70fffff : PCI Bus 0000:2f > > Thank you very much. nocrs was creating the problem. > > Just FYI, my setup is a regular supermicro PC, running Ubuntu 16.04 and my modified 4.4 kernel. I am trying to bring up devices via a PCIe fabric path. > > > On Mon, Aug 15, 2016 at 2:07 PM, Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote: >> >> On Mon, Aug 15, 2016 at 01:43:03PM -0700, Kallol Biswas wrote: >> > The kernel source code is 4.4. I have not done any modification except >> > pre-allocated bus numbers and memory window for a DS port to allow >> > successful hot plug operation of a PCIe port. >> > >> > 0.611919] acpi PNP0A08:00: host bridge window [io 0x0cf8-0x0cff] (ignored) >> > [ 0.611921] acpi PNP0A08:00: host bridge window [io 0x0000-0x0cf7 >> > window] (ignored) >> > [ 0.611922] acpi PNP0A08:00: host bridge window [io 0x1000-0x7fff >> > window] (ignored) >> > [ 0.611924] acpi PNP0A08:00: host bridge window [mem >> > 0x000a0000-0x000bffff window] (ignored) >> > [ 0.611926] acpi PNP0A08:00: host bridge window [mem >> > 0xfedb0000-0xfedb000f window] (ignored) >> > [ 0.611928] acpi PNP0A08:00: host bridge window [mem >> > 0xfedc0000-0xfedc000f window] (ignored) >> > [ 0.611929] acpi PNP0A08:00: host bridge window [mem >> > 0x90000000-0xc7ffbfff window] (ignored) >> > [ 0.611931] acpi PNP0A08:00: host bridge window [mem >> > 0x380000000000-0x383fffffffff window] (ignored) >> > [ 0.611932] PCI: root bus 00: using default resources >> > [ 0.612217] PCI host bridge to bus 0000:00 >> > [ 0.612219] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] >> > [ 0.612220] pci_bus 0000:00: root bus resource [mem >> > 0x00000000-0x3fffffffffff] >> > [ 0.612222] pci_bus 0000:00: root bus resource [bus 00-7e] >> > >> > kernel command line: >> > >> > BOOT_IMAGE=/boot/vmlinuz-4.4.8-kb >> > root=UUID=c6f326f6-7cc4-49f5-a3be-c511566165b4 ro >> > pci=realloc,nocrs,assign-busses pcie_ports=native >> > pciehp.pciehp_debug=1 pci=hpmemsize=0M console=ttyS1,115200n8 quiet >> > splash crashkernel=384M-:128M vt.handoff=7 >> >> You're booting with "pci=nocrs", which tells Linux to ignore the ACPI >> information about host bridge windows. You can't expect resource >> allocation and hotplug to work correctly when you're ignoring that. >> >> > On Mon, Aug 15, 2016 at 12:06 PM, Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote: >> > > On Mon, Aug 15, 2016 at 11:39:56AM -0700, Kallol Biswas wrote: >> > >> Thank you for the explanation. >> > >> >> > >> >> > >> Here is the relevant dmesg content: >> > >> >> > >> [ 0.790457] PCI: root bus ff: using default resources >> > >> [ 0.790491] PCI host bridge to bus 0000:ff >> > >> [ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff] >> > >> [ 0.790495] pci_bus 0000:ff: root bus resource [mem >> > >> 0x00000000-0x3fffffffffff] >> > >> [ 0.790497] pci_bus 0000:ff: root bus resource [bus ff] >> > >> >> > >> 0.795379] PCI host bridge to bus 0000:7f >> > >> [ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff] >> > >> [ 0.795383] pci_bus 0000:7f: root bus resource [mem >> > >> 0x00000000-0x3fffffffffff] >> > >> [ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f] >> > >> >> > >> 0.809149] PCI host bridge to bus 0000:00 >> > >> [ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] >> > >> [ 0.809153] pci_bus 0000:00: root bus resource [mem >> > >> 0x00000000-0x3fffffffffff] >> > >> [ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e] >> > > >> > > This is the host bridge that claims to lead to bus 0000:00. >> > > >> > > But the I/O and memory resources of all these bridges are wrong. The >> > > ranges they show are CPU addresses, not PCI bus addresses, so they >> > > cannot overlap. >> > > >> > > It could be that you're using a non-standard host bridge driver that >> > > did not apply the translation between CPU addresses and PCI bus >> > > addresses. In ACPI, this is the _TRA property. >> > > >> > >> [ 0.852071] PCI host bridge to bus 0000:80 >> > >> [ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff] >> > >> [ 0.852075] pci_bus 0000:80: root bus resource [mem >> > >> 0x00000000-0x3fffffffffff] >> > >> [ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe] >> > >> >> > >> Following lines are from output of /proc/iomem >> > >> >> > >> c8000000-c8000000 : <BAD> >> > >> c8000001-c8000001 : <BAD> >> > >> c8000002-c8000002 : <BAD> >> > >> c8000003-c8000003 : <BAD> >> > >> c8100000-c91fffff : PCI Bus 0000:02 >> > >> c8100000-c90fffff : PCI Bus 0000:03 >> > >> c8100000-c90fffff : PCI Bus 0000:04 >> > >> c8100000-c82fffff : PCI Bus 0000:05 >> > >> c8100000-c81fffff : PCI Bus 0000:06 >> > >> c8100000-c8100fff : 0000:06:00.0 >> > >> c8200000-c82fffff : PCI Bus 0000:0a >> > >> c8200000-c8203fff : 0000:0a:00.0 >> > >> c9100000-c913ffff : 0000:02:00.0 >> > >> fb000000-fb8fffff : PCI Bus 0000:81 >> > >> fb000000-fb8fffff : PCI Bus 0000:82 >> > >> fb000000-fb7fffff : PCI Bus 0000:8c >> > >> fb000000-fb7fffff : 0000:8c:00.0 >> > >> fb000000-fb7fffff : Plx_Mgr >> > >> fb800000-fb8fffff : PCI Bus 0000:87 >> > >> fb800000-fb800fff : 0000:87:00.0 >> > >> fb900000-fb900fff : 0000:80:05.4 >> > >> >> > >> >> > >> I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This >> > >> device is available after reboot, and initial PCIe bus enumeration >> > >> time. >> > >> >> > >> The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug >> > >> operation is done. >> > >> >> > >> Attached is the complete /proc/iomem and do not find the line >> > >> c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for >> > >> PCI Bus 0000:80? BAR access works fine through this bus. >> > >> >> > >> >> > >> >> > >> On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote: >> > >> > Hi Kallol, >> > >> > >> > >> > Sorry I didn't respond to this earlier; I was on vacation for most of >> > >> > last week. >> > >> > >> > >> > On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote: >> > >> >> I am trying to debug an issue that the PCIe memory >> > >> >> transactions are not routed through a host bridge. >> > >> >> >> > >> >> The config transactions work fine. The primary, secondary and >> > >> >> subordinate bus numbers look fine for the PCIe tree under the host >> > >> >> bridge. The memory windows are programmed fine. >> > >> >> >> > >> >> Can someone point me to the code that programs the chipset/root port >> > >> >> registers those direct processor transactions to a specific host >> > >> >> bridge? >> > >> > >> > >> > It looks like you have an x86 system with ACPI. In that case, we use >> > >> > the ACPI pci_root.c host bridge driver. The chipset configuration for >> > >> > routing processor memory transactions to host bridges is done by the >> > >> > firmware, and the kernel doesn't touch it. There should be >> > >> > information in the chipset spec about how to do this, but there's no >> > >> > code in Linux itself that does it. >> > >> > >> > >> >> Cat of /proc/mem is copied below. >> > >> >> >> > >> >> /root/physmem -p c8100000 -s 4 -r >> > >> >> Operation:read to Addr: 0xc8100000 Value: 0 >> > >> >> ffffffff ffffffff ffffffff ffffffff >> > >> >> >> > >> >> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and >> > >> >> external switches/devices does not capture any mem transactions. >> > >> >> However, config transactions are visible in the trace. >> > >> > >> > >> > 00:03.0 is not a host bridge; it's a Root Port, which logically >> > >> > connects two PCI buses. A host bridge logically connects a processor >> > >> > bus to a PCI bus. >> > >> > >> > >> >> root@xlink-target:~# cat /proc/iomem >> > >> >> 00000000-00000fff : reserved >> > >> >> 00001000-0009abff : System RAM >> > >> >> 0009ac00-0009ffff : reserved >> > >> >> 000c0000-000c7fff : Video ROM >> > >> >> 000c8000-000c8fff : Adapter ROM >> > >> >> 000e0000-000fffff : reserved >> > >> >> 000f0000-000fffff : System ROM >> > >> >> 00100000-78f36fff : System RAM >> > >> >> 01000000-01828f14 : Kernel code >> > >> >> 01828f15-01f3fb3f : Kernel data >> > >> >> 020bb000-021fdfff : Kernel bss >> > >> >> 30000000-37ffffff : Crash kernel >> > >> >> 78f37000-79839fff : reserved >> > >> >> 79116018-79116018 : APEI ERST >> > >> >> 7911601c-79116021 : APEI ERST >> > >> >> 79116028-79116039 : APEI ERST >> > >> >> 79116040-7911604c : APEI ERST >> > >> >> 79116050-7911804f : APEI ERST >> > >> >> 7983a000-79d4afff : ACPI Non-volatile Storage >> > >> >> 79d4b000-8fffffff : reserved >> > >> >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] >> > >> >> c6000000-c70fffff : PCI Bus 0000:2f >> > >> >> c6000000-c70fffff : PCI Bus 0000:30 >> > >> >> c6000000-c6ffffff : 0000:30:00.0 >> > >> >> c7000000-c701ffff : 0000:30:00.0 >> > >> >> c7100000-c71fffff : PCI Bus 0000:31 >> > >> >> c7100000-c711ffff : 0000:31:00.1 >> > >> >> c7100000-c711ffff : igb >> > >> >> c7120000-c713ffff : 0000:31:00.0 >> > >> >> c7120000-c713ffff : igb >> > >> >> c7140000-c7143fff : 0000:31:00.1 >> > >> >> c7140000-c7143fff : igb >> > >> >> c7144000-c7147fff : 0000:31:00.0 >> > >> >> c7144000-c7147fff : igb >> > >> >> c7300000-c730ffff : 0000:00:14.0 >> > >> >> c7300000-c730ffff : xhci-hcd >> > >> >> c7310000-c73107ff : 0000:00:1f.2 >> > >> >> c7310000-c73107ff : ahci >> > >> >> c7311000-c73113ff : 0000:00:1d.0 >> > >> >> c7311000-c73113ff : ehci_hcd >> > >> >> c7312000-c73123ff : 0000:00:1a.0 >> > >> >> c7312000-c73123ff : ehci_hcd >> > >> >> c7314000-c731400f : 0000:00:16.1 >> > >> >> c7315000-c731500f : 0000:00:16.0 >> > >> >> c7316000-c73167ff : 0000:00:11.4 >> > >> >> c7316000-c73167ff : ahci >> > >> >> c7317000-c7317fff : 0000:00:05.4 >> > >> >> c7ffc000-c7ffcfff : dmar1 >> > >> >> c8000000-c8000000 : <BAD> >> > >> >> c8000001-c8000001 : <BAD> >> > >> >> c8000002-c8000002 : <BAD> >> > >> >> c8000003-c8000003 : <BAD> >> > >> > >> > >> > Interesting, what are these from? >> > >> > >> > >> >> c8100000-c91fffff : PCI Bus 0000:02 >> > >> >> c8100000-c90fffff : PCI Bus 0000:03 >> > >> >> c8100000-c90fffff : PCI Bus 0000:04 >> > >> >> c8100000-c82fffff : PCI Bus 0000:05 >> > >> >> c8100000-c81fffff : PCI Bus 0000:06 >> > >> >> c8100000-c8100fff : 0000:06:00.0 >> > >> > >> > >> > You're trying to read the 06:00.0 BAR at 0xc8100000. The windows >> > >> > above look fine, but the top-level host bridge window is missing. >> > >> > >> > >> > The c8100000-c91fffff window is a window through the 00:03.0 Root >> > >> > Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards >> > >> > memory transactions in the c8100000-c91fffff range from its primary >> > >> > bus (bus 00) to its secondary bus (bus 02). >> > >> > >> > >> > But we don't know how those transactions get to bus 00 in the first >> > >> > place. The host bridge is the connection between the processor and >> > >> > bus 00, and it should have a window that encloses c8100000-c91fffff. >> > >> > >> > >> > Your dmesg log should have information about this bridge, e.g., >> > >> > something like this: >> > >> > >> > >> > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f]) >> > >> > PCI host bridge to bus 0000:00 >> > >> > pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window] >> > >> > >> > >> > and there should be a line in /proc/iomem something like this: >> > >> > >> > >> > c8100000-c9ffffff : PCI Bus 0000:00 >> > >> > >> > >> > Bjorn >> > > >> > >> 00000000-00000fff : reserved >> > >> 00001000-0009abff : System RAM >> > >> 0009ac00-0009ffff : reserved >> > >> 000c0000-000c7fff : Video ROM >> > >> 000c8000-000c8fff : Adapter ROM >> > >> 000e0000-000fffff : reserved >> > >> 000f0000-000fffff : System ROM >> > >> 00100000-78f36fff : System RAM >> > >> 01000000-01828f14 : Kernel code >> > >> 01828f15-01f3fb3f : Kernel data >> > >> 020bb000-021fdfff : Kernel bss >> > >> 30000000-37ffffff : Crash kernel >> > >> 78f37000-79839fff : reserved >> > >> 79116018-79116018 : APEI ERST >> > >> 7911601c-79116021 : APEI ERST >> > >> 79116028-79116039 : APEI ERST >> > >> 79116040-7911604c : APEI ERST >> > >> 79116050-7911804f : APEI ERST >> > >> 7983a000-79d4afff : ACPI Non-volatile Storage >> > >> 79d4b000-8fffffff : reserved >> > >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff] >> > >> c6000000-c70fffff : PCI Bus 0000:2f >> > >> c6000000-c70fffff : PCI Bus 0000:30 >> > >> c6000000-c6ffffff : 0000:30:00.0 >> > >> c7000000-c701ffff : 0000:30:00.0 >> > >> c7100000-c71fffff : PCI Bus 0000:31 >> > >> c7100000-c711ffff : 0000:31:00.1 >> > >> c7100000-c711ffff : igb >> > >> c7120000-c713ffff : 0000:31:00.0 >> > >> c7120000-c713ffff : igb >> > >> c7140000-c7143fff : 0000:31:00.1 >> > >> c7140000-c7143fff : igb >> > >> c7144000-c7147fff : 0000:31:00.0 >> > >> c7144000-c7147fff : igb >> > >> c7300000-c730ffff : 0000:00:14.0 >> > >> c7300000-c730ffff : xhci-hcd >> > >> c7310000-c73107ff : 0000:00:1f.2 >> > >> c7310000-c73107ff : ahci >> > >> c7311000-c73113ff : 0000:00:1d.0 >> > >> c7311000-c73113ff : ehci_hcd >> > >> c7312000-c73123ff : 0000:00:1a.0 >> > >> c7312000-c73123ff : ehci_hcd >> > >> c7314000-c731400f : 0000:00:16.1 >> > >> c7315000-c731500f : 0000:00:16.0 >> > >> c7316000-c73167ff : 0000:00:11.4 >> > >> c7316000-c73167ff : ahci >> > >> c7317000-c7317fff : 0000:00:05.4 >> > >> c7ffc000-c7ffcfff : dmar1 >> > >> c8000000-c8000000 : <BAD> >> > >> c8000001-c8000001 : <BAD> >> > >> c8000002-c8000002 : <BAD> >> > >> c8000003-c8000003 : <BAD> >> > >> c8100000-c91fffff : PCI Bus 0000:02 >> > >> c8100000-c90fffff : PCI Bus 0000:03 >> > >> c8100000-c90fffff : PCI Bus 0000:04 >> > >> c8100000-c82fffff : PCI Bus 0000:05 >> > >> c8100000-c81fffff : PCI Bus 0000:06 >> > >> c8100000-c8100fff : 0000:06:00.0 >> > >> c8200000-c82fffff : PCI Bus 0000:0a >> > >> c8200000-c8203fff : 0000:0a:00.0 >> > >> c9100000-c913ffff : 0000:02:00.0 >> > >> fb000000-fb8fffff : PCI Bus 0000:81 >> > >> fb000000-fb8fffff : PCI Bus 0000:82 >> > >> fb000000-fb7fffff : PCI Bus 0000:8c >> > >> fb000000-fb7fffff : 0000:8c:00.0 >> > >> fb000000-fb7fffff : Plx_Mgr >> > >> fb800000-fb8fffff : PCI Bus 0000:87 >> > >> fb800000-fb800fff : 0000:87:00.0 >> > >> fb900000-fb900fff : 0000:80:05.4 >> > >> fbffc000-fbffcfff : dmar0 >> > >> fec00000-fecfffff : PNP0003:00 >> > >> fec00000-fec003ff : IOAPIC 0 >> > >> fec01000-fec013ff : IOAPIC 1 >> > >> fec40000-fec403ff : IOAPIC 2 >> > >> fed00000-fed003ff : HPET 0 >> > >> fed00000-fed003ff : PNP0103:00 >> > >> fed12000-fed1200f : pnp 00:01 >> > >> fed12010-fed1201f : pnp 00:01 >> > >> fed1b000-fed1bfff : pnp 00:01 >> > >> fed1c000-fed44fff : reserved >> > >> fed1c000-fed3ffff : pnp 00:01 >> > >> fed1f410-fed1f414 : iTCO_wdt.0.auto >> > >> fed45000-fed8bfff : pnp 00:01 >> > >> fee00000-feefffff : pnp 00:01 >> > >> fee00000-fee00fff : Local APIC >> > >> ff000000-ffffffff : reserved >> > >> ff000000-ffffffff : pnp 00:01 >> > >> 100000000-47fffffff : System RAM >> > >> 480000000-4807fffff : PCI Bus 0000:02 >> > >> 480000000-4807fffff : PCI Bus 0000:03 >> > >> 480000000-4807fffff : PCI Bus 0000:04 >> > >> 480000000-4800fffff : PCI Bus 0000:05 >> > >> 480000000-4800fffff : PCI Bus 0000:06 >> > >> 480000000-480003fff : 0000:06:00.0 >> > >> 383fffe00000-383fffefffff : PCI Bus 0000:31 >> > >> 383fffe00000-383fffe1ffff : 0000:31:00.1 >> > >> 383fffe20000-383fffe3ffff : 0000:31:00.1 >> > >> 383fffe40000-383fffe5ffff : 0000:31:00.0 >> > >> 383fffe60000-383fffe7ffff : 0000:31:00.0 >> > >> 383ffff00000-383ffff03fff : 0000:00:04.7 >> > >> 383ffff00000-383ffff03fff : ioatdma >> > >> 383ffff04000-383ffff07fff : 0000:00:04.6 >> > >> 383ffff04000-383ffff07fff : ioatdma >> > >> 383ffff08000-383ffff0bfff : 0000:00:04.5 >> > >> 383ffff08000-383ffff0bfff : ioatdma >> > >> 383ffff0c000-383ffff0ffff : 0000:00:04.4 >> > >> 383ffff0c000-383ffff0ffff : ioatdma >> > >> 383ffff10000-383ffff13fff : 0000:00:04.3 >> > >> 383ffff10000-383ffff13fff : ioatdma >> > >> 383ffff14000-383ffff17fff : 0000:00:04.2 >> > >> 383ffff14000-383ffff17fff : ioatdma >> > >> 383ffff18000-383ffff1bfff : 0000:00:04.1 >> > >> 383ffff18000-383ffff1bfff : ioatdma >> > >> 383ffff1c000-383ffff1ffff : 0000:00:04.0 >> > >> 383ffff1c000-383ffff1ffff : ioatdma >> > >> 383ffff21000-383ffff210ff : 0000:00:1f.3 >> > >> 387fffe00000-387fffefffff : PCI Bus 0000:81 >> > >> 387fffe00000-387fffefffff : PCI Bus 0000:82 >> > >> 387fffe00000-387fffefffff : PCI Bus 0000:87 >> > >> 387fffe00000-387fffe03fff : 0000:87:00.0 >> > >> 387ffff00000-387ffff03fff : 0000:80:04.7 >> > >> 387ffff00000-387ffff03fff : ioatdma >> > >> 387ffff04000-387ffff07fff : 0000:80:04.6 >> > >> 387ffff04000-387ffff07fff : ioatdma >> > >> 387ffff08000-387ffff0bfff : 0000:80:04.5 >> > >> 387ffff08000-387ffff0bfff : ioatdma >> > >> 387ffff0c000-387ffff0ffff : 0000:80:04.4 >> > >> 387ffff0c000-387ffff0ffff : ioatdma >> > >> 387ffff10000-387ffff13fff : 0000:80:04.3 >> > >> 387ffff10000-387ffff13fff : ioatdma >> > >> 387ffff14000-387ffff17fff : 0000:80:04.2 >> > >> 387ffff14000-387ffff17fff : ioatdma >> > >> 387ffff18000-387ffff1bfff : 0000:80:04.1 >> > >> 387ffff18000-387ffff1bfff : ioatdma >> > >> 387ffff1c000-387ffff1ffff : 0000:80:04.0 >> > >> 387ffff1c000-387ffff1ffff : ioatdma >> > > > > -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
