On 09/22/2018 07:06 PM, Marek Vasut wrote: > On 09/18/2018 12:51 PM, Wolfram Sang wrote: >> On Tue, Sep 18, 2018 at 11:15:55AM +0100, Lorenzo Pieralisi wrote: >>> On Tue, May 22, 2018 at 12:05:14AM +0200, Marek Vasut wrote: >>>> From: Phil Edworthy <phil.edworthy@xxxxxxxxxxx> >>>> >>>> The PCIe DMA controller on RCar Gen2 and earlier is on 32bit bus, >>>> so limit the DMA range to 32bit. >>>> >>>> Signed-off-by: Phil Edworthy <phil.edworthy@xxxxxxxxxxx> >>>> Signed-off-by: Marek Vasut <marek.vasut+renesas@xxxxxxxxx> >>>> Cc: Arnd Bergmann <arnd@xxxxxxxx> >>>> Cc: Geert Uytterhoeven <geert+renesas@xxxxxxxxx> >>>> Cc: Phil Edworthy <phil.edworthy@xxxxxxxxxxx> >>>> Cc: Simon Horman <horms+renesas@xxxxxxxxxxxx> >>>> Cc: Wolfram Sang <wsa@xxxxxxxxxxxxx> >>>> Cc: linux-renesas-soc@xxxxxxxxxxxxxxx >>>> To: linux-pci@xxxxxxxxxxxxxxx >>>> --- >>>> NOTE: I'm aware of https://patchwork.kernel.org/patch/9495895/ , but the >>>> discussion seems to have gone way off, so I'm sending this as a >>>> RFC. Any feedback on how to do this limiting properly would be nice. >>>> --- >>>> drivers/pci/host/pcie-rcar.c | 28 ++++++++++++++++++++++++++++ >>>> 1 file changed, 28 insertions(+) >>> >>> The issue solved by this patch was solved in a more generic way through >>> this series: >>> >>> https://lists.linuxfoundation.org/pipermail/iommu/2018-July/028792.html >>> >>> I will therefore drop this patch from the PCI patch queue. >> >> Cool. Thanks for this series and thanks for the heads up! >> >> Marek, can you confirm our issue is fixed (if you haven't done already)? > > I assembled a setup with NVMe controller, which should be triggering > this issue according to [1], but I'm not having much success getting it > to work with or without this patchset. Logs are below, nvme smart-log > seems to produce completely bogus results, although the device is at > least recognized. > > Maybe there's something else that's broken ? > > [1] https://patchwork.kernel.org/patch/9495895/ > > # dmesg | egrep '(pci|nvm)' > rcar-pcie fe000000.pcie: host bridge /soc/pcie@fe000000 ranges: > rcar-pcie fe000000.pcie: Parsing ranges property... > rcar-pcie fe000000.pcie: IO 0xfe100000..0xfe1fffff -> 0x00000000 > rcar-pcie fe000000.pcie: MEM 0xfe200000..0xfe3fffff -> 0xfe200000 > rcar-pcie fe000000.pcie: MEM 0x30000000..0x37ffffff -> 0x30000000 > rcar-pcie fe000000.pcie: MEM 0x38000000..0x3fffffff -> 0x38000000 > rcar-pcie fe000000.pcie: PCIe x1: link up > rcar-pcie fe000000.pcie: Current link speed is 5 GT/s > rcar-pcie fe000000.pcie: PCI host bridge to bus 0000:00 > pci_bus 0000:00: root bus resource [bus 00-ff] > pci_bus 0000:00: root bus resource [io 0x0000-0xfffff] > pci_bus 0000:00: root bus resource [mem 0xfe200000-0xfe3fffff] > pci_bus 0000:00: root bus resource [mem 0x30000000-0x37ffffff] > pci_bus 0000:00: root bus resource [mem 0x38000000-0x3fffffff pref] > pci_bus 0000:00: scanning bus > pci 0000:00:00.0: [1912:0025] type 01 class 0x060400 > pci 0000:00:00.0: enabling Extended Tags > pci 0000:00:00.0: PME# supported from D0 D3hot D3cold > pci 0000:00:00.0: PME# disabled > pci_bus 0000:00: fixups for bus > pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 0 > pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 1 > pci_bus 0000:01: scanning bus > pci 0000:01:00.0: [8086:f1a5] type 00 class 0x010802 > pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit] > pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s > x1 link at 0000:00:00.0 (capable of 31.504 Gb/s with 8 GT/s x4 link) > pci_bus 0000:01: fixups for bus > pci_bus 0000:01: bus scan returning with max=01 > pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 > pci_bus 0000:00: bus scan returning with max=01 > pci 0000:00:00.0: BAR 14: assigned [mem 0xfe200000-0xfe2fffff] > pci 0000:01:00.0: BAR 0: assigned [mem 0xfe200000-0xfe203fff 64bit] > pci 0000:00:00.0: PCI bridge to [bus 01] > pci 0000:00:00.0: bridge window [mem 0xfe200000-0xfe2fffff] > pcieport 0000:00:00.0: assign IRQ: got 0 > pcieport 0000:00:00.0: enabling device (0000 -> 0002) > pcieport 0000:00:00.0: enabling bus mastering > pcieport 0000:00:00.0: Signaling PME with IRQ 198 > rcar-pcie ee800000.pcie: host bridge /soc/pcie@ee800000 ranges: > rcar-pcie ee800000.pcie: Parsing ranges property... > rcar-pcie ee800000.pcie: IO 0xee900000..0xee9fffff -> 0x00000000 > rcar-pcie ee800000.pcie: MEM 0xeea00000..0xeebfffff -> 0xeea00000 > rcar-pcie ee800000.pcie: MEM 0xc0000000..0xc7ffffff -> 0xc0000000 > rcar-pcie ee800000.pcie: MEM 0xc8000000..0xcfffffff -> 0xc8000000 > rcar-pcie ee800000.pcie: PCIe link down > nvme 0000:01:00.0: assign IRQ: got 171 > nvme nvme0: pci function 0000:01:00.0 > nvme 0000:01:00.0: enabling device (0000 -> 0002) > nvme 0000:01:00.0: enabling bus mastering > nvme nvme0: Shutdown timeout set to 60 seconds > > # lspci -nvvvs 01:00.0 > 01:00.0 0108: 8086:f1a5 (rev 03) (prog-if 02 [NVM Express]) > Subsystem: 8086:390a > Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- > ParErr- Stepping- SERR- FastB2B- DisINTx- > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0 > Interrupt: pin A routed to IRQ 171 > Region 0: Memory at fe200000 (64-bit, non-prefetchable) [size=16K] > Capabilities: [40] Power Management version 3 > Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA > PME(D0-,D1-,D2-,D3hot-,D3cold-) > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- > Capabilities: [70] Express (v2) Endpoint, MSI 00 > DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s > unlimited, L1 unlimited > ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ > SlotPowerLimit 0.000W > DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ > Unsupported+ > RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- > FLReset- > MaxPayload 128 bytes, MaxReadReq 512 bytes > DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ > TransPend- > LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit > Latency L0s <1us, L1 <8us > ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ > DLActive- BWMgmt- ABWMgmt- > DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, > LTR+, OBFF Via message > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, > LTR-, OBFF Disabled > LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- > SpeedDis- > Transmit Margin: Normal Operating Range, > EnterModifiedCompliance- ComplianceSOS- > Compliance De-emphasis: -6dB > LnkSta2: Current De-emphasis Level: -6dB, > EqualizationComplete-, EqualizationPhase1- > EqualizationPhase2-, EqualizationPhase3-, > LinkEqualizationRequest- > Capabilities: [b0] MSI-X: Enable- Count=16 Masked- > Vector table: BAR=0 offset=00002000 > PBA: BAR=0 offset=00002100 > Capabilities: [100 v2] Advanced Error Reporting > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- > UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > NonFatalErr- > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > NonFatalErr+ > AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ > ChkEn- > Capabilities: [158 v1] #19 > Capabilities: [178 v1] Latency Tolerance Reporting > Max snoop latency: 0ns > Max no snoop latency: 0ns > Capabilities: [180 v1] L1 PM Substates > L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ > ASPM_L1.1+ L1_PM_Substates+ > PortCommonModeRestoreTime=10us > PortTPowerOnTime=10us > L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- > T_CommonMode=0us LTR1.2_Threshold=0ns > L1SubCtl2: T_PwrOn=10us > Kernel driver in use: nvme > lspci: Unable to load libkmod resources: error -12 > > # nvme smart-log /dev/nvme0 > Smart Log for NVME device:nvme0 namespace-id:ffffffff > critical_warning : 0 > temperature : -273 C > available_spare : 0% > available_spare_threshold : 0% > percentage_used : 0% > data_units_read : 0 > data_units_written : 0 > host_read_commands : 0 > host_write_commands : 0 > controller_busy_time : 0 > power_cycles : 0 > power_on_hours : 0 > unsafe_shutdowns : 0 > media_errors : 0 > num_err_log_entries : 0 > Warning Temperature Time : 0 > Critical Composite Temperature Time : 0 > Thermal Management T1 Trans Count : 0 > Thermal Management T2 Trans Count : 0 > Thermal Management T1 Total Time : 0 > Thermal Management T2 Total Time : 0 Any suggestions ? -- Best regards, Marek Vasut
