On Thu, 2 Mar 2023 09:40:35 +0200 Tasos Sahanidis <tasos@xxxxxxxxxxxx> wrote: > On 2023-03-01 16:10, Alex Williamson wrote: > > 0000:02:00.0 is the upstream port of that switch and 0000:03:02.0 is > > the downstream port for the 7790. 0000:03:02.0 is the port that should > > also now enter D3hot. > > > >> If so, I tested in 5.18, both before and while running the VM, with 6.2 > >> both with and without disable_idle_d3, and in all cases they stayed at D0. > > > > In this case the upstream port should always stay in D0, it has quite a > > lot more devices under it than just the GPU. It's interesting that the > > MosChip that assigns ok is also under a downstream port of this switch. > > That means the downstream port 0000:03:06.0 should also be entering > > D3hot when all of the MosChip devices are attached to vfio-pci and > > unused. > > > > I'm not convinced thought that the MosChip assignment is a good > > comparison device though, as a "multi-i/o" controller, it's possible > > that it doesn't actually make use of DMA that would trigger the IOMMU > > like the GPU does. Do you have a NIC card you could replace one of > > these with? > > Unfortunately I do not have any other PCIe devices available. > I did, however, pass through the onboard Realtek NIC and it worked > perfectly fine (without any errors in dmesg). > > 0000:03:05.0 should be the root port for the NIC. > > ==> 6_2_before_vm_rtl <== > + cat /sys/bus/pci/devices/0000:03:05.0/power_state > D3hot > + cat /sys/bus/pci/devices/0000:03:05.0/power/runtime_status > suspended > + cat /sys/bus/pci/devices/0000:03:05.0/power/control > auto > + cat /sys/bus/pci/devices/0000:09:00.0/power_state > D3hot > + cat /sys/bus/pci/devices/0000:09:00.0/power/runtime_status > suspended > + cat /sys/bus/pci/devices/0000:09:00.0/power/control > auto > + lspci -vvv -s 0000:03:05.0 > 03:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge (prog-if 00 [Normal decode]) > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 64 bytes > Interrupt: pin ? routed to IRQ 37 > Bus: primary=03, secondary=09, subordinate=09, sec-latency=0 > I/O behind bridge: 0000b000-0000bfff [size=4K] > Memory behind bridge: d0300000-d03fffff [size=1M] > Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] > Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- > BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B- > PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- > Capabilities: <access denied> > Kernel driver in use: pcieport > > > ==> 6_2_running_vm_rtl <== > + cat /sys/bus/pci/devices/0000:03:05.0/power_state > D0 > + cat /sys/bus/pci/devices/0000:03:05.0/power/runtime_status > active > + cat /sys/bus/pci/devices/0000:03:05.0/power/control > auto > + cat /sys/bus/pci/devices/0000:09:00.0/power_state > D0 > + cat /sys/bus/pci/devices/0000:09:00.0/power/runtime_status > active > + cat /sys/bus/pci/devices/0000:09:00.0/power/control > auto > + lspci -vvv -s 0000:03:05.0 > 03:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge (prog-if 00 [Normal decode]) > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 64 bytes > Interrupt: pin ? routed to IRQ 37 > Bus: primary=03, secondary=09, subordinate=09, sec-latency=0 > I/O behind bridge: 0000b000-0000bfff [size=4K] > Memory behind bridge: d0300000-d03fffff [size=1M] > Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] > Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- > BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B- > PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- > Capabilities: <access denied> > Kernel driver in use: pcieport > > > > It's possible the switch has a problem with D3hot support and it may > > need to be disabled or augmented with a PCI quirk. In addition to > > investigating what power state the downstream port is achieving and > > reporting lspci -vvv with and without disable_idle_d3, would you mind > > reporting "lspci -nns 2:00.0" and "lspci -nns 3:" to report all the > > vendor and device IDs of the switch. Thanks, > > > > It seems that way, especially after manually preventing the root port > for the graphics card from entering D3hot, however the one for the NIC > seems to be doing that just fine, which makes things more confusing. Yes, the fact that the NIC works suggests there's not simply a blatant chip defect where we should blindly disable D3 power state support for this downstream port. I'm also not seeing any difference in the downstream port configuration between the VM running after the port has resumed from D3hot and the case where the port never entered D3hot. But it suddenly dawns on me that you're assigning a Radeon HD 7790, which is one of the many AMD GPUs which is plagued by reset problems. I wonder if that's a factor there. This particular GPU even has special handling in QEMU to try to manually reset the device, and which likely has never been tested since adding runtime power management support. In fact, I'm surprised anyone is doing regular device assignment with an HD 7790 and considers it a normal, acceptable experience even with the QEMU workarounds. I certainly wouldn't feel comfortable proposing a quirk for the downstream port to disable D3hot for an issue only seen when assigning a device with such a nefarious background relative to device assignment. It does however seem like there are sufficient options in place to work around the issue, either disabling power management at the vfio-pci driver, or specifically for the downstream port via sysfs. I don't really have any better suggestions given our limited ability to test and highly suspect target device. Any other ideas, Abhishek? Thanks, Alex > $ lspci -nns 2:00.0 > 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad] > > $ lspci -nns 3: > 03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3] > 03:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3] > 03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3] > 03:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3] > 03:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3] > 03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4] > 03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4] > 03:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4] > > And the information for the PCIe port: > > ==> 6_2_before_vm <== > + cat /sys/bus/pci/devices/0000:03:02.0/power_state > D3hot > + cat /sys/bus/pci/devices/0000:03:02.0/power/runtime_status > suspended > + cat /sys/bus/pci/devices/0000:03:02.0/power/control > auto > + lspci -vvv -s 0000:03:02.0 > 03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge (prog-if 00 [Normal decode]) > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 64 bytes > Interrupt: pin ? routed to IRQ 35 > Bus: primary=03, secondary=06, subordinate=06, sec-latency=0 > I/O behind bridge: 0000d000-0000dfff [size=4K] > Memory behind bridge: d0100000-d01fffff [size=1M] > Prefetchable memory behind bridge: 0000001030000000-0000001047ffffff [size=384M] > Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- > BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B- > PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- > Capabilities: [50] Power Management version 3 > Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) > Status: D3 NoSoftRst- PME-Enable+ DSel=0 DScale=0 PME- > Capabilities: [58] Express (v2) Downstream Port (Slot+), MSI 00 > DevCap: MaxPayload 512 bytes, PhantFunc 0 > ExtTag+ RBE+ > DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ > RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ > MaxPayload 128 bytes, MaxReadReq 512 bytes > DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- > LnkCap: Port #2, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <32us > ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+ > LnkCtl: ASPM Disabled; Disabled- CommClk+ > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 8GT/s (downgraded), Width x4 (ok) > TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- > SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- > Slot #0, PowerLimit 0.000W; Interlock- NoCompl+ > SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- > Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- > SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- > Changed: MRL- PresDet- LinkState+ > DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR+ > 10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix- > EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- > FRS-, ARIFwd- > AtomicOpsCap: Routing- > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- > AtomicOpsCtl: EgressBlck- > LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB > Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- > Compliance De-emphasis: -6dB > LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ > EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- > Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ > Address: 00000000fee00000 Data: 0000 > Capabilities: [c0] Subsystem: ASUSTeK Computer Inc. Matisse PCIe GPP Bridge > Capabilities: [c8] HyperTransport: MSI Mapping Enable+ Fixed+ > Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> > Capabilities: [150 v2] Advanced Error Reporting > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ > AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- > MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- > HeaderLog: 00000000 00000000 00000000 00000000 > Capabilities: [270 v1] Secondary PCI Express > LnkCtl3: LnkEquIntrruptEn-, PerformEqu- > LaneErrStat: LaneErr at lane: 0 1 2 3 > Capabilities: [2a0 v1] Access Control Services > ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ > ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- > Capabilities: [370 v1] L1 PM Substates > L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+ > L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- > L1SubCtl2: > Capabilities: [400 v1] Data Link Feature <?> > Capabilities: [410 v1] Physical Layer 16.0 GT/s <?> > Capabilities: [440 v1] Lane Margining at the Receiver <?> > Kernel driver in use: pcieport > > > ==> 6_2_running_vm <== > + cat /sys/bus/pci/devices/0000:03:02.0/power_state > D0 > + cat /sys/bus/pci/devices/0000:03:02.0/power/runtime_status > active > + cat /sys/bus/pci/devices/0000:03:02.0/power/control > auto > + lspci -vvv -s 0000:03:02.0 > 03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge (prog-if 00 [Normal decode]) > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 64 bytes > Interrupt: pin ? routed to IRQ 35 > Bus: primary=03, secondary=06, subordinate=06, sec-latency=0 > I/O behind bridge: 0000d000-0000dfff [size=4K] > Memory behind bridge: d0100000-d01fffff [size=1M] > Prefetchable memory behind bridge: 0000001030000000-0000001047ffffff [size=384M] > Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- > BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B- > PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- > Capabilities: [50] Power Management version 3 > Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- > Capabilities: [58] Express (v2) Downstream Port (Slot+), MSI 00 > DevCap: MaxPayload 512 bytes, PhantFunc 0 > ExtTag+ RBE+ > DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ > RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ > MaxPayload 128 bytes, MaxReadReq 512 bytes > DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- > LnkCap: Port #2, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <32us > ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+ > LnkCtl: ASPM Disabled; Disabled- CommClk+ > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 8GT/s (downgraded), Width x4 (ok) > TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+ > SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- > Slot #0, PowerLimit 0.000W; Interlock- NoCompl+ > SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- > Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- > SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- > Changed: MRL- PresDet- LinkState+ > DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR+ > 10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix- > EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- > FRS-, ARIFwd- > AtomicOpsCap: Routing- > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- > AtomicOpsCtl: EgressBlck- > LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB > Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- > Compliance De-emphasis: -6dB > LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ > EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- > Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ > Address: 00000000fee00000 Data: 0000 > Capabilities: [c0] Subsystem: ASUSTeK Computer Inc. Matisse PCIe GPP Bridge > Capabilities: [c8] HyperTransport: MSI Mapping Enable+ Fixed+ > Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> > Capabilities: [150 v2] Advanced Error Reporting > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ > AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- > MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- > HeaderLog: 00000000 00000000 00000000 00000000 > Capabilities: [270 v1] Secondary PCI Express > LnkCtl3: LnkEquIntrruptEn-, PerformEqu- > LaneErrStat: LaneErr at lane: 0 1 2 3 > Capabilities: [2a0 v1] Access Control Services > ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ > ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- > Capabilities: [370 v1] L1 PM Substates > L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+ > L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- > L1SubCtl2: > Capabilities: [400 v1] Data Link Feature <?> > Capabilities: [410 v1] Physical Layer 16.0 GT/s <?> > Capabilities: [440 v1] Lane Margining at the Receiver <?> > Kernel driver in use: pcieport > > > ==> 6_2_before_vm_disable_d3_idle <== > + cat /sys/bus/pci/devices/0000:03:02.0/power_state > D0 > + cat /sys/bus/pci/devices/0000:03:02.0/power/runtime_status > active > + cat /sys/bus/pci/devices/0000:03:02.0/power/control > auto > + lspci -vvv -s 0000:03:02.0 > 03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge (prog-if 00 [Normal decode]) > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 64 bytes > Interrupt: pin ? routed to IRQ 35 > Bus: primary=03, secondary=06, subordinate=06, sec-latency=0 > I/O behind bridge: 0000d000-0000dfff [size=4K] > Memory behind bridge: d0100000-d01fffff [size=1M] > Prefetchable memory behind bridge: 0000001030000000-0000001047ffffff [size=384M] > Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- > BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B- > PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- > Capabilities: [50] Power Management version 3 > Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- > Capabilities: [58] Express (v2) Downstream Port (Slot+), MSI 00 > DevCap: MaxPayload 512 bytes, PhantFunc 0 > ExtTag+ RBE+ > DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ > RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ > MaxPayload 128 bytes, MaxReadReq 512 bytes > DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- > LnkCap: Port #2, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <32us > ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+ > LnkCtl: ASPM Disabled; Disabled- CommClk+ > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 8GT/s (downgraded), Width x4 (ok) > TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- > SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- > Slot #0, PowerLimit 0.000W; Interlock- NoCompl+ > SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- > Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- > SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- > Changed: MRL- PresDet- LinkState+ > DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR+ > 10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix- > EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- > FRS-, ARIFwd- > AtomicOpsCap: Routing- > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- > AtomicOpsCtl: EgressBlck- > LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB > Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- > Compliance De-emphasis: -6dB > LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ > EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- > Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ > Address: 00000000fee00000 Data: 0000 > Capabilities: [c0] Subsystem: ASUSTeK Computer Inc. Matisse PCIe GPP Bridge > Capabilities: [c8] HyperTransport: MSI Mapping Enable+ Fixed+ > Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> > Capabilities: [150 v2] Advanced Error Reporting > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ > AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- > MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- > HeaderLog: 00000000 00000000 00000000 00000000 > Capabilities: [270 v1] Secondary PCI Express > LnkCtl3: LnkEquIntrruptEn-, PerformEqu- > LaneErrStat: LaneErr at lane: 0 1 2 3 > Capabilities: [2a0 v1] Access Control Services > ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ > ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- > Capabilities: [370 v1] L1 PM Substates > L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+ > L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- > L1SubCtl2: > Capabilities: [400 v1] Data Link Feature <?> > Capabilities: [410 v1] Physical Layer 16.0 GT/s <?> > Capabilities: [440 v1] Lane Margining at the Receiver <?> > Kernel driver in use: pcieport > > > ==> 6_2_running_vm_disable_d3_idle <== > + cat /sys/bus/pci/devices/0000:03:02.0/power_state > D0 > + cat /sys/bus/pci/devices/0000:03:02.0/power/runtime_status > active > + cat /sys/bus/pci/devices/0000:03:02.0/power/control > auto > + lspci -vvv -s 0000:03:02.0 > 03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge (prog-if 00 [Normal decode]) > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 64 bytes > Interrupt: pin ? routed to IRQ 35 > Bus: primary=03, secondary=06, subordinate=06, sec-latency=0 > I/O behind bridge: 0000d000-0000dfff [size=4K] > Memory behind bridge: d0100000-d01fffff [size=1M] > Prefetchable memory behind bridge: 0000001030000000-0000001047ffffff [size=384M] > Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- > BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B- > PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- > Capabilities: [50] Power Management version 3 > Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- > Capabilities: [58] Express (v2) Downstream Port (Slot+), MSI 00 > DevCap: MaxPayload 512 bytes, PhantFunc 0 > ExtTag+ RBE+ > DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ > RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ > MaxPayload 128 bytes, MaxReadReq 512 bytes > DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- > LnkCap: Port #2, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <32us > ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+ > LnkCtl: ASPM Disabled; Disabled- CommClk+ > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 8GT/s (downgraded), Width x4 (ok) > TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+ > SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- > Slot #0, PowerLimit 0.000W; Interlock- NoCompl+ > SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- > Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- > SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- > Changed: MRL- PresDet- LinkState+ > DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR+ > 10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix- > EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- > FRS-, ARIFwd- > AtomicOpsCap: Routing- > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- > AtomicOpsCtl: EgressBlck- > LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB > Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- > Compliance De-emphasis: -6dB > LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ > EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- > Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ > Address: 00000000fee00000 Data: 0000 > Capabilities: [c0] Subsystem: ASUSTeK Computer Inc. Matisse PCIe GPP Bridge > Capabilities: [c8] HyperTransport: MSI Mapping Enable+ Fixed+ > Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> > Capabilities: [150 v2] Advanced Error Reporting > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ > AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn- > MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- > HeaderLog: 00000000 00000000 00000000 00000000 > Capabilities: [270 v1] Secondary PCI Express > LnkCtl3: LnkEquIntrruptEn-, PerformEqu- > LaneErrStat: LaneErr at lane: 0 1 2 3 > Capabilities: [2a0 v1] Access Control Services > ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ > ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- > Capabilities: [370 v1] L1 PM Substates > L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+ L1_PM_Substates+ > L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- > L1SubCtl2: > Capabilities: [400 v1] Data Link Feature <?> > Capabilities: [410 v1] Physical Layer 16.0 GT/s <?> > Capabilities: [440 v1] Lane Margining at the Receiver <?> > Kernel driver in use: pcieport > > > -- > Tasos >
