On Tue, May 25, 2021 at 3:43 PM Ard Biesheuvel <ardb@xxxxxxxxxx> wrote:
>
> On Tue, 25 May 2021 at 21:15, Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote:
> >
> > On Tue, May 25, 2021 at 05:54:56PM +0200, Ard Biesheuvel wrote:
> > > On Tue, 25 May 2021 at 17:34, Peter Geis <pgwipeout@xxxxxxxxx> wrote:
> >
> > > > > > >> > On 2021-05-18 10:09, Alexandru Elisei wrote:
> > > > > > >> >> [..]
> > > > > > >> >> [ 0.305183] rockchip-pcie f8000000.pcie: host bridge /pcie@f8000000 ranges:
> > > > > > >> >> [ 0.305248] rockchip-pcie f8000000.pcie: MEM 0x00fa000000..0x00fbdfffff -> 0x00fa000000
> > > > > > >> >> [ 0.305285] rockchip-pcie f8000000.pcie: IO 0x00fbe00000..0x00fbefffff -> 0x00fbe00000
> > > > > > >> >> [ 0.373705] rockchip-pcie f8000000.pcie: PCI host bridge to bus 0000:00
> > > > > > >> >> [ 0.373730] pci_bus 0000:00: root bus resource [bus 00-1f]
> > > > > > >> >> [ 0.373751] pci_bus 0000:00: root bus resource [mem 0xfa000000-0xfbdfffff 64bit]
> > > > > > >> >> [ 0.373777] pci_bus 0000:00: root bus resource [io 0x0000-0xfffff] (bus address [0xfbe00000-0xfbefffff])
> >
> > > ... For some reason, lspci translates the BAR values to CPU
> > > addresses, but the PCI side addresses are within 32-bits.
> >
> > lspci shows BARs as CPU physical addresses by default. These are the
> > same addresses you would see in pdev->resource[n] and the same as BAR
> > values you would see in dmesg.
> >
> > A 64-bit CPU physical address can certainly be translated by the host
> > bridge to a 32-bit PCI address. But that's not happening here because
> > this host bridge applies no translation (CPU physical 0xfa000000 maps
> > to bus address 0xfa000000).
> >
> > "lspci -b" shows the PCI bus addresses.
>
> Ah, thanks.
>
> It does seem, though, that the information overload in this thread is
> causing confusion now. Peter shared some log output where there is
> definitely MMIO translation being applied.
Yes, I've done work on the rk3399 pcie controller which is why this
caught my attention.
The original issue still seems to exist:
For some reason:
commit 9d57e61bf723 ("of/pci: Add IORESOURCE_MEM_64 to resource flags for
64-bit memory addresses")
causes allocation issues now.
The original description of the issue aligned with issues I was having
bringing up the rk356x pcie controller.
>
> > > [ 6.673497] pci_bus 0000:00: root bus resource [io 0x0000-0xfffff]
> > > (bus address [0x3f700000-0x3f7fffff])
> > > [ 6.674642] pci_bus 0000:00: root bus resource [mem
> > > 0x300000000-0x33f6fffff] (bus address [0x00000000-0x3f6fffff])
>
> In this case, the I/O translation definitely looks wrong. On a typical
> ARM DT system, you will see something like
>
> [ 1.500324] Remapped I/O 0x0000000067f00000 to [io 0x0000-0xffff window]
> [ 1.500522] pci_bus 0000:00: root bus resource [io 0x0000-0xffff window]
>
> The MMIO window looks correct, but I suspect that both 0x82000000 and
> 0x83000000 in the DT ranges are describing the resource window as
> prefetchable, preventing the allocation of non-prefetchable BARs in
> this window.
I checked with lspci -vvvbxxxxnn:
Before your changes:
00:00.0 PCI bridge [0604]: Fuzhou Rockchip Electronics Co., Ltd Device
[1d87:3566] (rev 01) (prog-if 00 [Normal decode])
I/O behind bridge: 00001000-00001fff [size=4K]
Memory behind bridge: 50000000-500fffff [size=1M]
Prefetchable memory behind bridge:
0000000040000000-000000004fffffff [size=256M]
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc.
[AMD/ATI] Turks PRO [Radeon HD 7570] [1002:675d] (prog-if 00 [VGA
controller])
Region 0: Memory at 40000000 (64-bit, prefetchable)
Region 2: Memory at 50000000 (64-bit, non-prefetchable)
Region 4: I/O ports at 7f701000
Expansion ROM at 50020000 [disabled]
After your changes:
lspci -vvvbxxxxnn
00:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3566
(rev 01) (prog-if 00 [Normal decode])
I/O behind bridge: 00001000-00001fff [size=4K]
Memory behind bridge: 10000000-100fffff [size=1M]
Prefetchable memory behind bridge:
0000000000000000-000000000fffffff [size=256M]
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc.
[AMD/ATI] Turks PRO [Radeon HD 7570] (prog-if 00 [VGA controller])
Region 0: Memory at <unassigned> (64-bit, prefetchable) [virtual]
Region 2: Memory at 10000000 (64-bit, non-prefetchable) [virtual]
Region 4: I/O ports at 1000 [virtual]
Expansion ROM at 10020000 [disabled]
>
> Peter, for the configuration listed here, could you try something like
>
> ranges = <0x1000000 0x0 0x0 [IO base in the CPU address map] [IO size]>,
> <0x2000000 0x0 0x0 [MMIO base in the CPU address map] [MMIO size]>;
That was similar to what I already had, removing the relocatable flag
and setting both addresses to 0x0 are the changes.
Here is the result:
[ 6.410670] rockchip-dw-pcie 3c0000000.pcie: Looking up
vpcie3v3-supply from device tree
[ 6.413924] rockchip-dw-pcie 3c0000000.pcie: host bridge
/pcie@fe260000 ranges:
[ 6.414653] rockchip-dw-pcie 3c0000000.pcie: Parsing ranges property...
[ 6.415321] rockchip-dw-pcie 3c0000000.pcie: IO
0x033f700000..0x033f7fffff -> 0x0000000000
[ 6.416375] rockchip-dw-pcie 3c0000000.pcie: MEM
0x0300000000..0x033f6fffff -> 0x0000000000
[ 6.417363] rockchip-dw-pcie 3c0000000.pcie: got 49 for legacy interrupt
[ 6.418676] rockchip-dw-pcie 3c0000000.pcie: found 5 interrupts
[ 6.419224] rockchip-dw-pcie 3c0000000.pcie: invalid resource
[ 6.419764] rockchip-dw-pcie 3c0000000.pcie: iATU unroll: enabled
[ 6.420388] rockchip-dw-pcie 3c0000000.pcie: Detected iATU regions:
8 outbound, 8 inbound
[ 6.628630] rockchip-dw-pcie 3c0000000.pcie: Link up
[ 6.631026] rockchip-dw-pcie 3c0000000.pcie: PCI host bridge to bus 0000:00
[ 6.631734] pci_bus 0000:00: root bus resource [bus 00]
[ 6.632548] pci_bus 0000:00: root bus resource [io 0x0000-0xfffff]
[ 6.633175] pci_bus 0000:00: root bus resource [mem
0x300000000-0x33f6fffff] (bus address [0x00000000-0x3f6fffff])
[ 6.634170] pci_bus 0000:00: scanning bus
[ 6.635052] pci 0000:00:00.0: disabling Extended Tags (this device
can't handle them)
[ 6.635813] pci 0000:00:00.0: [1d87:3566] type 01 class 0x060400
[ 6.636782] pci 0000:00:00.0: reg 0x38: [mem 0x300000000-0x30000ffff pref]
[ 6.638302] pci 0000:00:00.0: supports D1 D2
[ 6.638742] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[ 6.639432] pci 0000:00:00.0: PME# disabled
[ 6.650984] pci_bus 0000:00: fixups for bus
[ 6.651465] pci 0000:00:00.0: scanning [bus 01-ff] behind bridge, pass 0
[ 6.653454] pci_bus 0000:01: busn_res: can not insert [bus 01-ff]
under [bus 00] (conflicts with (null) [bus 00])
[ 6.654427] pci_bus 0000:01: scanning bus
[ 6.655134] pci 0000:01:00.0: [1002:675d] type 00 class 0x030000
[ 6.655938] pci 0000:01:00.0: reg 0x10: [mem
0x300000000-0x30fffffff 64bit pref]
[ 6.656934] pci 0000:01:00.0: reg 0x18: [mem 0x300000000-0x30001ffff 64bit]
[ 6.657677] pci 0000:01:00.0: reg 0x20: [io 0x0000-0x00ff]
[ 6.658363] pci 0000:01:00.0: reg 0x30: [mem 0x300000000-0x30001ffff pref]
[ 6.659845] pci 0000:01:00.0: supports D1 D2
[ 6.660715] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth,
limited by 2.5 GT/s PCIe x1 link at 0000:00:00.0 (capable of 32.000
Gb/s with 2.5 GT/s PCIe x16 link)
[ 6.664276] pci 0000:01:00.0: vgaarb: VGA device added:
decodes=io+mem,owns=none,locks=none
[ 6.666120] pci 0000:01:00.1: [1002:aa90] type 00 class 0x040300
[ 6.666922] pci 0000:01:00.1: reg 0x10: [mem 0x300000000-0x300003fff 64bit]
[ 6.668888] pci 0000:01:00.1: supports D1 D2
[ 6.680490] pci_bus 0000:01: fixups for bus
[ 6.680941] pci_bus 0000:01: bus scan returning with max=01
[ 6.681507] pci 0000:00:00.0: scanning [bus 01-ff] behind bridge, pass 1
[ 6.682180] pci_bus 0000:00: bus scan returning with max=ff
[ 6.682780] pci 0000:00:00.0: BAR 15: assigned [mem
0x300000000-0x30fffffff 64bit pref]
[ 6.683535] pci 0000:00:00.0: BAR 14: assigned [mem 0x310000000-0x3100fffff]
[ 6.684322] pci 0000:00:00.0: BAR 6: assigned [mem
0x310100000-0x31010ffff pref]
[ 6.685021] pci 0000:00:00.0: BAR 13: assigned [io 0x1000-0x1fff]
[ 6.685669] pci 0000:01:00.0: BAR 0: assigned [mem
0x300000000-0x30fffffff 64bit pref]
[ 6.686500] pci 0000:01:00.0: BAR 2: assigned [mem
0x310000000-0x31001ffff 64bit]
[ 6.687281] pci 0000:01:00.0: BAR 6: assigned [mem
0x310020000-0x31003ffff pref]
[ 6.687985] pci 0000:01:00.1: BAR 0: assigned [mem
0x310040000-0x310043fff 64bit]
[ 6.688872] pci 0000:01:00.0: BAR 4: assigned [io 0x1000-0x10ff]
[ 6.689492] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[ 6.690000] pci 0000:00:00.0: bridge window [io 0x1000-0x1fff]
[ 6.690593] pci 0000:00:00.0: bridge window [mem 0x310000000-0x3100fffff]
[ 6.691254] pci 0000:00:00.0: bridge window [mem
0x300000000-0x30fffffff 64bit pref]
[ 6.694100] pcieport 0000:00:00.0: assign IRQ: got 95
[ 6.695189] ITS ALLOCATE 2nd level WORKAROUND
[ 6.727845] pcieport 0000:00:00.0: PME: Signaling with IRQ 96
[ 6.729830] pcieport 0000:00:00.0: saving config space at offset
0x0 (reading 0x35661d87)
[ 6.730587] pcieport 0000:00:00.0: saving config space at offset
0x4 (reading 0x100507)
[ 6.731315] pcieport 0000:00:00.0: saving config space at offset
0x8 (reading 0x6040001)
[ 6.732051] pcieport 0000:00:00.0: saving config space at offset
0xc (reading 0x10000)
[ 6.733095] pcieport 0000:00:00.0: saving config space at offset
0x10 (reading 0x0)
[ 6.733797] pcieport 0000:00:00.0: saving config space at offset
0x14 (reading 0x0)
[ 6.734496] pcieport 0000:00:00.0: saving config space at offset
0x18 (reading 0xff0100)
[ 6.735231] pcieport 0000:00:00.0: saving config space at offset
0x1c (reading 0x20001010)
[ 6.735981] pcieport 0000:00:00.0: saving config space at offset
0x20 (reading 0x10001000)
[ 6.736786] pcieport 0000:00:00.0: saving config space at offset
0x24 (reading 0xff10001)
[ 6.737528] pcieport 0000:00:00.0: saving config space at offset
0x28 (reading 0x0)
[ 6.738223] pcieport 0000:00:00.0: saving config space at offset
0x2c (reading 0x0)
[ 6.738918] pcieport 0000:00:00.0: saving config space at offset
0x30 (reading 0x0)
[ 6.739613] pcieport 0000:00:00.0: saving config space at offset
0x34 (reading 0x40)
[ 6.740369] pcieport 0000:00:00.0: saving config space at offset
0x38 (reading 0x0)
[ 6.741067] pcieport 0000:00:00.0: saving config space at offset
0x3c (reading 0x2015f)
[ 6.742525] radeon 0000:01:00.0: assign IRQ: got 95
