Hi Keith Many thanks for coming back on this > -----Original Message----- > From: Keith Busch [mailto:keith.busch@xxxxxxxxx] > Sent: 02 February 2017 23:23 > To: Gabriele Paoloni > Cc: linux-pci@xxxxxxxxxxxxxxx; Linuxarm; liudongdong (C); zhangjukuo; > Wangzhou (B) > Subject: Re: Question: PCIe DPC not allowing for link retraining and > bus re-scan > > On Tue, Jan 31, 2017 at 04:59:31PM +0000, Gabriele Paoloni wrote: > > Many thanks for this, I'll try to get such switch too so maybe > > I can help with development and/or validation. > > If you know of specific switches that support for sure such SW > > trigger feature please let me know and I'll try to buy them. > > I'm not trying to advertise for any particular product, but this is > what > I'm currently using for testing my NVMe drives: > > http://www.serialcables.com/products.asp?cat=350&tier=264 > > It uses these PCIe switches: > > https://www.broadcom.com/products/pcie-switches- > bridges/expressfabric/pex9781 Many thanks for the details, eventually I'll try to buys these... > > I just use this for PCIe SSDs, but there are other vendors that have > capable switches too. > > These do support DPC sofware triggering, and the method I mentioned > before is correct: > > # setpci -s <B:D.f> ECAP_DPC+6.w=40:40 > > On all my testing, the DPC event necesssarily triggers a PCIe HP event > that automatically does the rescan when the containment is released. > Here is a sequence from a single software trigger and nothing else, > using 4.10-rc6: Yes, I have actually read the specs twice... in 6.2.10 we have: "Software can use Data Link Layer State Changed interrupts, DL_Active ERR_COR signaling, or both, to signal when the Link reaches the DL_Active state again" and again in 6.2.10.5 Implementation note: "It is recommended that operating systems use Data Link Layer State Changed interrupts for signaling when DL_ACTIVE changes state." So you are right "Data Link Layer State Changed" is a hot-plug event that will trigger the re-enumeration > > root@localhost: ~# setpci -s 89:03.0 ECAP_DPC+6.w=40:40 > > dpc 0000:89:03.0:pcie210: DPC containment event, status:0x002f > source:0x0000 > dpc 0000:89:03.0:pcie210: DPC extended error triggered, remove > downstream devices > pciehp 0000:89:03.0:pcie204: Slot(3): Link Down > dpc 0000:89:03.0:pcie210: DPC containment event, status:0x0006 > source:0x0000 > pciehp 0000:89:03.0:pcie204: Slot(3): Card not present > dpc 0000:89:03.0:pcie210: DPC containment event, status:0x0006 > source:0x0000 > pciehp 0000:89:03.0:pcie204: Slot(3): Card present > dpc 0000:89:03.0:pcie210: DPC containment event, status:0x0006 > source:0x0000 > pciehp 0000:89:03.0:pcie204: Slot(3): Link Up > pciehp 0000:89:03.0:pcie204: Slot(3): Link Up event ignored; already > powering on > pci 0000:8d:00.0: [8086:0953] type 00 class 0x010802 > pci 0000:8d:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit] > pci 0000:8d:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref] > pci 0000:8d:00.0: Max Payload Size set to 256 (was 128, max 256) > pci 0000:8d:00.0: BAR 6: assigned [mem 0xec000000-0xec00ffff pref] > pci 0000:8d:00.0: BAR 0: assigned [mem 0xec010000-0xec013fff 64bit] > pcieport 0000:89:03.0: PCI bridge to [bus 8d] > pcieport 0000:89:03.0: bridge window [mem 0xec000000-0xec0fffff] > pcieport 0000:89:03.0: bridge window [mem 0x840c00000-0x840dfffff > 64bit pre > > So I think we're okay to not have DPC initiate the rescan since it > would > just compete with pciehp. The only thing we need to do differently is > wait for RP busy if it implements that capability, and no such device > for general availabilty currently exists, as far as I know. I will send out the patch once I have a root port that supports DPC Many Thanks again Gab
