RE: [EXT] Re: nvme may get timeout from dd when using different non-prefetch mmio outbound/ranges

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi, Bjorn

> -----Original Message-----
> From: Bjorn Helgaas [mailto:helgaas@xxxxxxxxxx]
> Sent: Monday, October 25, 2021 11:48 PM
> To: Li Chen
> Cc: linux-pci@xxxxxxxxxxxxxxx; Lorenzo Pieralisi; Rob Herring; kw@xxxxxxxxx;
> Bjorn Helgaas; linux-kernel@xxxxxxxxxxxxxxx; Tom Joseph; Keith Busch; Jens
> Axboe; Christoph Hellwig; Sagi Grimberg; linux-nvme@xxxxxxxxxxxxxxxxxxx
> Subject: [EXT] Re: nvme may get timeout from dd when using different non-
> prefetch mmio outbound/ranges
> 
> [+cc Tom (Cadence maintainer), NVMe folks]
> 
> On Fri, Oct 22, 2021 at 10:08:20AM +0000, Li Chen wrote:
> > Hi, all
> >
> > I found my nvme may get timeout with simply dd, the pcie controller
> > is cadence, and the pcie-controller platform I use is
> > pcie-cadence-host.c, kenrel version is 5.10.32:
> 
> Given that nvme is pretty widely used, I tend to suspect the cadence
> driver or DTS here, but I added both cadence and nvme folks to the CC:
> list.

Thanks!

> 
> > # dd if=/dev/urandom of=/dev/nvme0n1 bs=4k count=1024000
> > [   41.913484][  T274] urandom_read: 2 callbacks suppressed
> > [   41.913490][  T274] random: dd: uninitialized urandom read (4096 bytes read)
> > [   41.926130][  T274] random: dd: uninitialized urandom read (4096 bytes read)
> > [   41.933348][  T274] random: dd: uninitialized urandom read (4096 bytes read)
> > [   47.651842][    C0] random: crng init done
> > [   47.655963][    C0] random: 2 urandom warning(s) missed due to ratelimiting
> > [   81.448128][   T64] nvme nvme0: controller is down; will reset: CSTS=0x3,
> PCI_STATUS=0x2010
> > [   81.481139][    T7] nvme 0000:05:00.0: enabling bus mastering
> > [   81.486946][    T7] nvme 0000:05:00.0: saving config space at offset 0x0
> (reading 0xa809144d)
> > [   81.495517][    T7] nvme 0000:05:00.0: saving config space at offset 0x4
> (reading 0x20100006)
> > [   81.504091][    T7] nvme 0000:05:00.0: saving config space at offset 0x8
> (reading 0x1080200)
> > [   81.512571][    T7] nvme 0000:05:00.0: saving config space at offset 0xc
> (reading 0x0)
> > [   81.520527][    T7] nvme 0000:05:00.0: saving config space at offset 0x10
> (reading 0x8000004)
> > [   81.529094][    T7] nvme 0000:05:00.0: saving config space at offset 0x14
> (reading 0x0)
> > [   81.537138][    T7] nvme 0000:05:00.0: saving config space at offset 0x18
> (reading 0x0)
> > [   81.545186][    T7] nvme 0000:05:00.0: saving config space at offset 0x1c
> (reading 0x0)
> > [   81.553252][    T7] nvme 0000:05:00.0: saving config space at offset 0x20
> (reading 0x0)
> > [   81.561296][    T7] nvme 0000:05:00.0: saving config space at offset 0x24
> (reading 0x0)
> > [   81.569340][    T7] nvme 0000:05:00.0: saving config space at offset 0x28
> (reading 0x0)
> > [   81.577384][    T7] nvme 0000:05:00.0: saving config space at offset 0x2c
> (reading 0xa801144d)
> > [   81.586038][    T7] nvme 0000:05:00.0: saving config space at offset 0x30
> (reading 0x0)
> > [   81.594081][    T7] nvme 0000:05:00.0: saving config space at offset 0x34
> (reading 0x40)
> > [   81.602217][    T7] nvme 0000:05:00.0: saving config space at offset 0x38
> (reading 0x0)
> > [   81.610266][    T7] nvme 0000:05:00.0: saving config space at offset 0x3c
> (reading 0x12c)
> > [   81.634065][    T7] nvme nvme0: Shutdown timeout set to 8 seconds
> > [   81.674332][    T7] nvme nvme0: 1/0/0 default/read/poll queues
> > [  112.168136][  T256] nvme nvme0: I/O 12 QID 1 timeout, disable controller
> > [  112.193145][  T256] blk_update_request: I/O error, dev nvme0n1, sector
> 600656 op 0x1:(WRITE) flags 0x104000 phys_seg 127 prio class 0
> > [  112.205220][  T256] Buffer I/O error on dev nvme0n1, logical block 75082, lost
> async page write
> > [  112.213978][  T256] Buffer I/O error on dev nvme0n1, logical block 75083, lost
> async page write
> > [  112.222727][  T256] Buffer I/O error on dev nvme0n1, logical block 75084, lost
> async page write
> > [  112.231474][  T256] Buffer I/O error on dev nvme0n1, logical block 75085, lost
> async page write
> > [  112.240220][  T256] Buffer I/O error on dev nvme0n1, logical block 75086, lost
> async page write
> > [  112.248966][  T256] Buffer I/O error on dev nvme0n1, logical block 75087, lost
> async page write
> > [  112.257719][  T256] Buffer I/O error on dev nvme0n1, logical block 75088, lost
> async page write
> > [  112.266467][  T256] Buffer I/O error on dev nvme0n1, logical block 75089, lost
> async page write
> > [  112.275213][  T256] Buffer I/O error on dev nvme0n1, logical block 75090, lost
> async page write
> > [  112.283959][  T256] Buffer I/O error on dev nvme0n1, logical block 75091, lost
> async page write
> > [  112.293554][  T256] blk_update_request: I/O error, dev nvme0n1, sector
> 601672 op 0x1:(WRITE) flags 0x104000 phys_seg 127 prio class 0
> > [  112.306559][  T256] blk_update_request: I/O error, dev nvme0n1, sector
> 602688 op 0x1:(WRITE) flags 0x104000 phys_seg 127 prio class 0
> > [  112.319525][  T256] blk_update_request: I/O error, dev nvme0n1, sector
> 603704 op 0x1:(WRITE) flags 0x104000 phys_seg 127 prio class 0
> > [  112.332501][  T256] blk_update_request: I/O error, dev nvme0n1, sector
> 604720 op 0x1:(WRITE) flags 0x104000 phys_seg 127 prio class 0
> > [  112.345466][  T256] blk_update_request: I/O error, dev nvme0n1, sector
> 605736 op 0x1:(WRITE) flags 0x104000 phys_seg 127 prio class 0
> > [  112.358427][  T256] blk_update_request: I/O error, dev nvme0n1, sector
> 606752 op 0x1:(WRITE) flags 0x104000 phys_seg 127 prio class 0
> > [  112.371386][  T256] blk_update_request: I/O error, dev nvme0n1, sector
> 607768 op 0x1:(WRITE) flags 0x104000 phys_seg 127 prio class 0
> > [  112.384346][  T256] blk_update_request: I/O error, dev nvme0n1, sector
> 608784 op 0x1:(WRITE) flags 0x104000 phys_seg 127 prio class 0
> > [  112.397315][  T256] blk_update_request: I/O error, dev nvme0n1, sector
> 609800 op 0x1:(WRITE) flags 0x104000 phys_seg 127 prio class 0
> > [  112.459313][    T7] nvme nvme0: failed to mark controller live state
> > [  112.465758][    T7] nvme nvme0: Removing after probe failure status: -19
> > dd: error writing '/dev/nvme0n1': No space left on device
> > 112200+0 records in
> > 112199+0 records out
> > 459567104 bytes (438.3MB) copied, 70.573266 seconds, 6.2MB/s
> > # [  112.935768][    T7] nvme nvme0: failed to set APST feature (-19)
> >
> >
> >
> >
> > Here is the dts I used:
> > pciec: pcie-controller@2040000000 {
> >                                 compatible = "cdns,cdns-pcie-host";
> > 		device_type = "pci";
> > 		#address-cells = <3>;
> > 		#size-cells = <2>;
> > 		bus-range = <0 5>;
> > 		linux,pci-domain = <0>;
> > 		cdns,no-bar-match-nbits = <38>;
> > 		vendor-id = <0x17cd>;
> > 		device-id = <0x0100>;
> > 		reg-names = "reg", "cfg";
> > 		reg = <0x20 0x40000000 0x0 0x10000000>,
> > 		      <0x20 0x00000000 0x0 0x00001000>;	/* RC only */
> >
> > 		/*
> > 		 * type: 0x00000000 cfg space
> > 		 * type: 0x01000000 IO
> > 		 * type: 0x02000000 32bit mem space No prefetch
> > 		 * type: 0x03000000 64bit mem space No prefetch
> > 		 * type: 0x43000000 64bit mem space prefetch
> > 		 * The First 16MB from BUS_DEV_FUNC=0:0:0 for cfg space
> > 		 * <0x00000000 0x00 0x00000000 0x20 0x00000000 0x00
> 0x01000000>, CFG_SPACE
> > 		*/
> > 		ranges = <0x01000000 0x00 0x00000000 0x20 0x00100000 0x00
> 0x00100000>,
> > 			 <0x02000000 0x00 0x08000000 0x20 0x08000000 0x00
> 0x08000000>;
> >
> > 		#interrupt-cells = <0x1>;
> > 		interrupt-map-mask = <0x00 0x0 0x0 0x7>;
> > 		interrupt-map = <0x0 0x0 0x0 0x1 &gic 0 229 0x4>,
> > 				<0x0 0x0 0x0 0x2 &gic 0 230 0x4>,
> > 				<0x0 0x0 0x0 0x3 &gic 0 231 0x4>,
> > 				<0x0 0x0 0x0 0x4 &gic 0 232 0x4>;
> > 		phys = <&pcie_phy>;
> > 		phy-names="pcie-phy";
> > 		status = "ok";
> > 	};
> >
> >
> > After some digging, I find if I change the controller's range
> > property from
> >
> > <0x02000000 0x00 0x08000000 0x20 0x08000000 0x00 0x08000000> into
> > <0x02000000 0x00 0x00400000 0x20 0x00400000 0x00 0x08000000>,
> >
> > then dd will success without timeout. IIUC, range here
> > is only for non-prefetch 32bit mmio, but dd will use dma (maybe cpu
> > will send cmd to nvme controller via mmio?).
> 
> I don't know how to interpret "ranges".  Can you supply the dmesg and
> "lspci -vvs 0000:05:00.0" output both ways, e.g.,
> 
>   pci_bus 0000:00: root bus resource [mem 0x7f800000-0xefffffff window]
>   pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff window]
>   pci 0000:05:00.0: [vvvv:dddd] type 00 class 0x...
>   pci 0000:05:00.0: reg 0x10: [mem 0x.....000-0x.....fff ...]
> 

>From the source code:

static int devm_of_pci_get_host_bridge_resources
{
	...
	err = of_pci_range_parser_init(&parser, dev_node);
	...
	for_each_of_pci_range(&parser, &range) {
		/* Read next ranges element */
		if ((range.flags & IORESOURCE_TYPE_BITS) == IORESOURCE_IO)
			range_type = "IO";
		else if ((range.flags & IORESOURCE_TYPE_BITS) == IORESOURCE_MEM)
			range_type = "MEM";
		else
			range_type = "err";
		dev_info(dev, "  %6s %#012llx..%#012llx -> %#012llx\n",
			 range_type, range.cpu_addr,
			 range.cpu_addr + range.size - 1, range.pci_addr);

		/*
		 * If we failed translation or got a zero-sized region
		 * then skip this range
		 */
		if (range.cpu_addr == OF_BAD_ADDR || range.size == 0)
			continue;

		err = of_pci_range_to_resource(&range, dev_node, &tmp_res);
		...
}

So, it simply translates mmio mapping.



Take
ranges = <0x01000000 0x00 0x00000000 0x20 0x00100000 0x00 0x00100000>,
			 <0x02000000 0x00 0x08000000 0x20 0x08000000 0x00 0x04000000>;
for example (this ranges will make nvme timeout):

lspci output:
# lspci -i /usr/share/misc/pci.ids -vvs 0000:05:00.0
05:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 980 (prog-if 02 [NVM Express])
        Subsystem: Samsung Electronics Co Ltd Device a801
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 44
        Region 0: Memory at 2008000000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 25.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (ok), Width x4 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [b0] MSI-X: Enable- Count=13 Masked-
                Vector table: BAR=0 offset=00003000
                PBA: BAR=0 offset=00002000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [158 v1] Power Budgeting <?>
        Capabilities: [168 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [188 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [190 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=0ns
                L1SubCtl2: T_PwrOn=10us
        Kernel driver in use: nvme
        Kernel modules: nvme

	


/proc/iomem:

01200000-ffffffff : System RAM
  01280000-022affff : Kernel code
  022b0000-0295ffff : reserved
  02960000-040cffff : Kernel data
  05280000-0528ffff : reserved
  41cc0000-422c0fff : reserved
  422c1000-4232afff : reserved
  4232d000-667bbfff : reserved
  667bc000-667bcfff : reserved
  667bd000-667c0fff : reserved
  667c1000-ffffffff : reserved
2000000000-2000000fff : cfg
2008000000-200bffffff : pcie-controller@2040000000
  2008000000-20081fffff : PCI Bus 0000:01
    2008000000-20080fffff : PCI Bus 0000:02
      2008000000-20080fffff : PCI Bus 0000:05
        2008000000-2008003fff : 0000:05:00.0
          2008000000-2008003fff : nvme
    2008100000-200817ffff : 0000:01:00.0
...

>From dmesg output:

# dmesg | grep pci
[    0.000000] Kernel command line: console=ttyS0 ubi.mtd=lnx root=ubi0:rootfs rw rootfstype=ubifs init=/linuxrc pci=nomsi pcie_pme=nomsi nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pci=nommconf nvme_core.default_ps_max_latency_us=0
[    2.431732] cdns-pcie 2040000000.pcie-controller: host bridge /pcie-controller@2040000000 ranges:
[    2.442055] cdns-pcie 2040000000.pcie-controller: Parsing ranges property...
[    2.450010] cdns-pcie 2040000000.pcie-controller:       IO 0x2000100000..0x20001fffff -> 0x0000000000
[    2.460215] cdns-pcie 2040000000.pcie-controller:      MEM 0x2008000000..0x200bffffff -> 0x0008000000
[    2.523271] cdns-pcie 2040000000.pcie-controller: Link up
[    2.530647] cdns-pcie 2040000000.pcie-controller: PCI host bridge to bus 0000:00
[    2.538885] pci_bus 0000:00: root bus resource [bus 00-05]
[    2.545194] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff]
[    2.552280] pci_bus 0000:00: root bus resource [mem 0x2008000000-0x200bffffff] (bus address [0x08000000-0x0bffffff])
[    2.563651] pci_bus 0000:00: scanning bus
[    2.568753] pci 0000:00:00.0: [17cd:0100] type 01 class 0x060400
[    2.576196] pci 0000:00:00.0: supports D1
[    2.581006] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    2.587667] pci 0000:00:00.0: PME# disabled
[    2.597799] pci_bus 0000:00: fixups for bus
[    2.602800] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 0
[    2.610308] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.619160] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 1
[    2.627525] pci_bus 0000:01: scanning bus
[    2.632550] pci 0000:01:00.0: [12d8:c016] type 01 class 0x060400
[    2.639425] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0007ffff]
[    2.647250] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[    2.654165] pci 0000:01:00.0: PME# disabled
[    2.664011] pci_bus 0000:01: fixups for bus
[    2.669012] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 0
[    2.676521] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.685370] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 1
[    2.693768] pci_bus 0000:02: scanning bus
[    2.699101] pci 0000:02:02.0: [12d8:c016] type 01 class 0x060400
[    2.706740] pci 0000:02:02.0: PME# supported from D0 D3hot D3cold
[    2.713653] pci 0000:02:02.0: PME# disabled
[    2.720417] pci 0000:02:04.0: [12d8:c016] type 01 class 0x060400
[    2.728032] pci 0000:02:04.0: PME# supported from D0 D3hot D3cold
[    2.734945] pci 0000:02:04.0: PME# disabled
[    2.741684] pci 0000:02:06.0: [12d8:c016] type 01 class 0x060400
[    2.749288] pci 0000:02:06.0: PME# supported from D0 D3hot D3cold
[    2.756201] pci 0000:02:06.0: PME# disabled
[    2.765483] pci_bus 0000:02: fixups for bus
[    2.770482] pci 0000:02:02.0: scanning [bus 00-00] behind bridge, pass 0
[    2.777991] pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.786839] pci 0000:02:04.0: scanning [bus 00-00] behind bridge, pass 0
[    2.794337] pci 0000:02:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.803184] pci 0000:02:06.0: scanning [bus 00-00] behind bridge, pass 0
[    2.810691] pci 0000:02:06.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.819539] pci 0000:02:02.0: scanning [bus 00-00] behind bridge, pass 1
[    2.827948] pci_bus 0000:03: scanning bus
[    2.836100] pci_bus 0000:03: fixups for bus
[    2.841085] pci_bus 0000:03: bus scan returning with max=03
[    2.847467] pci_bus 0000:03: busn_res: [bus 03-05] end is updated to 03
[    2.854911] pci 0000:02:04.0: scanning [bus 00-00] behind bridge, pass 1
[    2.863151] pci_bus 0000:04: scanning bus
[    2.871319] pci_bus 0000:04: fixups for bus
[    2.876303] pci_bus 0000:04: bus scan returning with max=04
[    2.882683] pci_bus 0000:04: busn_res: [bus 04-05] end is updated to 04
[    2.890151] pci 0000:02:06.0: scanning [bus 00-00] behind bridge, pass 1
[    2.898383] pci_bus 0000:05: scanning bus
[    2.903362] pci 0000:05:00.0: [144d:a809] type 00 class 0x010802
[    2.910266] pci 0000:05:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[    2.923188] pci_bus 0000:05: fixups for bus
[    2.928174] pci_bus 0000:05: bus scan returning with max=05
[    2.934553] pci_bus 0000:05: busn_res: [bus 05] end is updated to 05
[    2.941724] pci_bus 0000:02: bus scan returning with max=05
[    2.948101] pci_bus 0000:02: busn_res: [bus 02-05] end is updated to 05
[    2.955557] pci_bus 0000:01: bus scan returning with max=05
[    2.961983] pci_bus 0000:01: busn_res: [bus 01-05] end is updated to 05
[    2.969413] pci_bus 0000:00: bus scan returning with max=05
[    2.975843] pci 0000:00:00.0: BAR 8: assigned [mem 0x2008000000-0x20081fffff]
[    2.983843] pci 0000:01:00.0: BAR 8: assigned [mem 0x2008000000-0x20080fffff]
[    2.991795] pci 0000:01:00.0: BAR 0: assigned [mem 0x2008100000-0x200817ffff]
[    2.999801] pci 0000:02:06.0: BAR 8: assigned [mem 0x2008000000-0x20080fffff]
[    3.007763] pci 0000:02:02.0: PCI bridge to [bus 03]
[    3.013622] pci 0000:02:04.0: PCI bridge to [bus 04]
[    3.019487] pci 0000:05:00.0: BAR 0: assigned [mem 0x2008000000-0x2008003fff 64bit]
[    3.028012] pci 0000:02:06.0: PCI bridge to [bus 05]
[    3.033792] pci 0000:02:06.0:   bridge window [mem 0x2008000000-0x20080fffff]
[    3.041767] pci 0000:01:00.0: PCI bridge to [bus 02-05]
[    3.047808] pci 0000:01:00.0:   bridge window [mem 0x2008000000-0x20080fffff]
[    3.055782] pci 0000:00:00.0: PCI bridge to [bus 01-05]
[    3.061822] pci 0000:00:00.0:   bridge window [mem 0x2008000000-0x20081fffff]
[    4.184289]     pcie_pme=nomsi


Also take
ranges = <0x01000000 0x00 0x00000000 0x20 0x00100000 0x00 0x00100000>,
			 <0x02000000 0x00 0x08000000 0x20 0x08000000 0x00 0x04000000>;
for example (this ranges will not make nvme timeout):

lspci:
# lspci -i /usr/share/misc/pci.ids -vvs 0000:05:00.0
05:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 980 (prog-if 02 [NVM Express])
        Subsystem: Samsung Electronics Co Ltd Device a801
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 44
        Region 0: Memory at 2000400000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 25.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (ok), Width x4 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [b0] MSI-X: Enable- Count=13 Masked-
                Vector table: BAR=0 offset=00003000
                PBA: BAR=0 offset=00002000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap+ MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [158 v1] Power Budgeting <?>
        Capabilities: [168 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [188 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Capabilities: [190 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=0ns
                L1SubCtl2: T_PwrOn=10us
        Kernel driver in use: nvme
        Kernel modules: nvme

/proc/iomem:
01200000-ffffffff : System RAM
  01280000-022affff : Kernel code
  022b0000-0295ffff : reserved
  02960000-040cffff : Kernel data
  05280000-0528ffff : reserved
  41cc0000-422c0fff : reserved
  422c1000-4232afff : reserved
  4232d000-667bbfff : reserved
  667bc000-667bcfff : reserved
  667bd000-667c0fff : reserved
  667c1000-ffffffff : reserved
2000000000-2000000fff : cfg
2000400000-20083fffff : pcie-controller@2040000000
  2000400000-20005fffff : PCI Bus 0000:01
    2000400000-20004fffff : PCI Bus 0000:02
      2000400000-20004fffff : PCI Bus 0000:05
        2000400000-2000403fff : 0000:05:00.0
    2000500000-200057ffff : 0000:01:00.0
...

Dmesg:
# dmesg | grep pci
[    0.000000] Kernel command line: console=ttyS0 ubi.mtd=lnx root=ubi0:rootfs rw rootfstype=ubifs init=/linuxrc pci=nomsi pcie_pme=nomsi nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pci=nommconf nvme_core.default_ps_max_latency_us=0
[    2.436357] cdns-pcie 2040000000.pcie-controller: host bridge /pcie-controller@2040000000 ranges:
[    2.446129] cdns-pcie 2040000000.pcie-controller: Parsing ranges property...
[    2.454141] cdns-pcie 2040000000.pcie-controller:       IO 0x2000100000..0x20001fffff -> 0x0000000000
[    2.464350] cdns-pcie 2040000000.pcie-controller:      MEM 0x2000400000..0x20083fffff -> 0x0000400000
[    2.527395] cdns-pcie 2040000000.pcie-controller: Link up
[    2.534745] cdns-pcie 2040000000.pcie-controller: PCI host bridge to bus 0000:00
[    2.543036] pci_bus 0000:00: root bus resource [bus 00-05]
[    2.549345] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff]
[    2.556431] pci_bus 0000:00: root bus resource [mem 0x2000400000-0x20083fffff] (bus address [0x00400000-0x083fffff])
[    2.567802] pci_bus 0000:00: scanning bus
[    2.572823] pci 0000:00:00.0: [17cd:0100] type 01 class 0x060400
[    2.580290] pci 0000:00:00.0: supports D1
[    2.585099] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    2.591709] pci 0000:00:00.0: PME# disabled
[    2.601802] pci_bus 0000:00: fixups for bus
[    2.606801] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 0
[    2.614310] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.623161] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 1
[    2.631551] pci_bus 0000:01: scanning bus
[    2.636563] pci 0000:01:00.0: [12d8:c016] type 01 class 0x060400
[    2.643460] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0007ffff]
[    2.651289] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[    2.658202] pci 0000:01:00.0: PME# disabled
[    2.667966] pci_bus 0000:01: fixups for bus
[    2.673047] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 0
[    2.680556] pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.689405] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 1
[    2.697804] pci_bus 0000:02: scanning bus
[    2.703079] pci 0000:02:02.0: [12d8:c016] type 01 class 0x060400
[    2.710742] pci 0000:02:02.0: PME# supported from D0 D3hot D3cold
[    2.717657] pci 0000:02:02.0: PME# disabled
[    2.724466] pci 0000:02:04.0: [12d8:c016] type 01 class 0x060400
[    2.732078] pci 0000:02:04.0: PME# supported from D0 D3hot D3cold
[    2.739037] pci 0000:02:04.0: PME# disabled
[    2.745734] pci 0000:02:06.0: [12d8:c016] type 01 class 0x060400
[    2.753344] pci 0000:02:06.0: PME# supported from D0 D3hot D3cold
[    2.760257] pci 0000:02:06.0: PME# disabled
[    2.769510] pci_bus 0000:02: fixups for bus
[    2.774532] pci 0000:02:02.0: scanning [bus 00-00] behind bridge, pass 0
[    2.782043] pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.790892] pci 0000:02:04.0: scanning [bus 00-00] behind bridge, pass 0
[    2.798494] pci 0000:02:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.807347] pci 0000:02:06.0: scanning [bus 00-00] behind bridge, pass 0
[    2.814855] pci 0000:02:06.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.823701] pci 0000:02:02.0: scanning [bus 00-00] behind bridge, pass 1
[    2.832098] pci_bus 0000:03: scanning bus
[    2.840343] pci_bus 0000:03: fixups for bus
[    2.845328] pci_bus 0000:03: bus scan returning with max=03
[    2.851708] pci_bus 0000:03: busn_res: [bus 03-05] end is updated to 03
[    2.859152] pci 0000:02:04.0: scanning [bus 00-00] behind bridge, pass 1
[    2.867404] pci_bus 0000:04: scanning bus
[    2.875551] pci_bus 0000:04: fixups for bus
[    2.880535] pci_bus 0000:04: bus scan returning with max=04
[    2.886914] pci_bus 0000:04: busn_res: [bus 04-05] end is updated to 04
[    2.894359] pci 0000:02:06.0: scanning [bus 00-00] behind bridge, pass 1
[    2.902637] pci_bus 0000:05: scanning bus
[    2.907632] pci 0000:05:00.0: [144d:a809] type 00 class 0x010802
[    2.914536] pci 0000:05:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[    2.927440] pci_bus 0000:05: fixups for bus
[    2.932426] pci_bus 0000:05: bus scan returning with max=05
[    2.938807] pci_bus 0000:05: busn_res: [bus 05] end is updated to 05
[    2.946025] pci_bus 0000:02: bus scan returning with max=05
[    2.952403] pci_bus 0000:02: busn_res: [bus 02-05] end is updated to 05
[    2.959834] pci_bus 0000:01: bus scan returning with max=05
[    2.966233] pci_bus 0000:01: busn_res: [bus 01-05] end is updated to 05
[    2.973666] pci_bus 0000:00: bus scan returning with max=05
[    2.980097] pci 0000:00:00.0: BAR 8: assigned [mem 0x2000400000-0x20005fffff]
[    2.988097] pci 0000:01:00.0: BAR 8: assigned [mem 0x2000400000-0x20004fffff]
[    2.996046] pci 0000:01:00.0: BAR 0: assigned [mem 0x2000500000-0x200057ffff]
[    3.004092] pci 0000:02:06.0: BAR 8: assigned [mem 0x2000400000-0x20004fffff]
[    3.012055] pci 0000:02:02.0: PCI bridge to [bus 03]
[    3.017867] pci 0000:02:04.0: PCI bridge to [bus 04]
[    3.023709] pci 0000:05:00.0: BAR 0: assigned [mem 0x2000400000-0x2000403fff 64bit]
[    3.032253] pci 0000:02:06.0: PCI bridge to [bus 05]
[    3.038071] pci 0000:02:06.0:   bridge window [mem 0x2000400000-0x20004fffff]
[    3.046086] pci 0000:01:00.0: PCI bridge to [bus 02-05]
[    3.052127] pci 0000:01:00.0:   bridge window [mem 0x2000400000-0x20004fffff]
[    3.060100] pci 0000:00:00.0: PCI bridge to [bus 01-05]
[    3.066140] pci 0000:00:00.0:   bridge window [mem 0x2000400000-0x20005fffff]
[    4.181033]     pcie_pme=nomsi
[   44.419264] nvme nvme0: pci function 0000:05:00.0
[   44.425541] pci 0000:00:00.0: enabling device (0000 -> 0002)
[   44.432217] pci 0000:00:00.0: enabling bus mastering
[   44.438183] pci 0000:01:00.0: enabling device (0000 -> 0002)
[   44.444720] pci 0000:01:00.0: enabling bus mastering
[   44.453373] pci 0000:02:06.0: enabling device (0000 -> 0002)
[   44.459951] pci 0000:02:06.0: enabling bus mastering


[firstlove@sh-lchen tmp]$ diff no-timeout.lspci timeout.lspci 
8c8
<         Region 0: Memory at 2000400000 (64-bit, non-prefetchable) [size=16K]
---
>         Region 0: Memory at 2008000000 (64-bit, non-prefetchable) [size=16K]

> > Question:
> > 1.  Why dd can cause nvme timeout? Is there more debug ways?
> > 2. How can this mmio range affect nvme timeout?
> >
> > Regards,
> > Li
> >
> >
> **************************************************************
> ********
> > This email and attachments contain Ambarella Proprietary and/or Confidential
> Information and is intended solely for the use of the individual(s) to whom it is
> addressed. Any unauthorized review, use, disclosure, distribute, copy, or print is
> prohibited. If you are not an intended recipient, please contact the sender by
> reply email and destroy all copies of the original message. Thank you.
> 
> ##############################################################
> ########
> This EXTERNAL email has been scanned by Proofpoint Email Protect service.

Regards,
Li




[Index of Archives]     [DMA Engine]     [Linux Coverity]     [Linux USB]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [Greybus]

  Powered by Linux