Re: [PATCH] PCI: update NumVFs register properly during enable/disable sriov

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 2013/7/29 22:31, Bjorn Helgaas wrote:
> [+cc Don]
> 
> On Wed, Jul 24, 2013 at 3:26 AM, Yijing Wang <wangyijing@xxxxxxxxxx> wrote:
>> Currently, we only update NumVFs register during sriov_enable().
>> This register should also be updated properly during sriov_disable.
>> Otherwise, we will get the stale "Number of VFs" info from lspci.
> 
> Has this always been broken, or was it broken by a recent change?
> 
> Can you post a transcript demonstrating what's going wrong?  Doesn't
> have to be in the changelog unless it's short, but just something to
> help reproduce the problem.


Hi Bjorn,
   It seems to be not a regression. This is not a serious problem, only
make users get stale "NumVFs" info from lspci.

Following is my demonstrating info:

euler-linux:~ # lspci -tv
-+-[0000:ff]-+-00.0  Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers
...............[snip]...................
 \-[0000:00]-+-00.0  Intel Corporation 5500 I/O Hub to ESI Port
             +-01.0-[0000:01]--+-00.0  Intel Corporation 82576 Gigabit Network Connection
             |                 \-00.1  Intel Corporation 82576 Gigabit Network Connection
             +-03.0-[0000:02]----00.0  LSI Logic / Symbios Logic SAS1064ET PCI-Express Fusion-MPT SAS
             +-07.0-[0000:03]--+-00.0  Intel Corporation 82576 Gigabit Network Connection         ----------------->support sriov
             |                 \-00.1  Intel Corporation 82576 Gigabit Network Connection
             +-09.0-[0000:04]--+-00.0  Intel Corporation 82576 Gigabit Network Connection
             |                 \-00.1  Intel Corporation 82576 Gigabit Network Connection
             +-13.0  Intel Corporation 5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller
             +-14.0  Intel Corporation 5520/5500/X58 I/O Hub System Management Registers
             +-14.1  Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers
             +-14.2  Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers
             +-14.3  Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers
             +-1a.0  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
             +-1a.7  Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
             +-1d.0  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
             +-1d.1  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
             +-1d.2  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
             +-1d.7  Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
             +-1e.0-[0000:05]----00.0  XGI Technology Inc. (eXtreme Graphics Innovation) Z9s/Z9m (XG21 core)
             +-1f.0  Intel Corporation 82801JIB (ICH10) LPC Interface Controller
             +-1f.2  Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
             +-1f.3  Intel Corporation 82801JI (ICH10 Family) SMBus Controller
             \-1f.5  Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
euler-linux:~ # lspci -vvv -s 03:00.0
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
...........[snip]................
	Capabilities: [140] Device Serial Number 0c-da-21-ff-ff-ba-e2-90
	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00  --------------------->total is 8, current number is 0
		VF offset: 128, stride: 2, Device ID: 10ca
		Supported Page Size: 00000553, System Page Size: 00000001
		VF Migration: offset: 00000000, BIR: 1
	Kernel driver in use: igb
	Kernel modules: igb

euler-linux:~ # cd /sys/bus/pci/devices/0000\:03\:00.0/
euler-linux:/sys/bus/pci/devices/0000:03:00.0 # cat sriov_numvfs
0
euler-linux:/sys/bus/pci/devices/0000:03:00.0 # cat sriov_totalvfs
7
euler-linux:/sys/bus/pci/devices/0000:03:00.0 # echo 7 > sriov_numvfs             -------------------------->enable the sriov
euler-linux:/sys/bus/pci/devices/0000:03:00.0 # lspci -tv
-+-[0000:ff]-+-00.0  Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers
................[snip].....................
 \-[0000:00]-+-00.0  Intel Corporation 5500 I/O Hub to ESI Port
             +-01.0-[0000:01]--+-00.0  Intel Corporation 82576 Gigabit Network Connection
             |                 \-00.1  Intel Corporation 82576 Gigabit Network Connection
             +-03.0-[0000:02]----00.0  LSI Logic / Symbios Logic SAS1064ET PCI-Express Fusion-MPT SAS
             +-07.0-[0000:03]--+-00.0  Intel Corporation 82576 Gigabit Network Connection
             |                 +-00.1  Intel Corporation 82576 Gigabit Network Connection
             |                 +-10.0  Intel Corporation 82576 Virtual Function    -------------------------->the new VFs
             |                 +-10.2  Intel Corporation 82576 Virtual Function
             |                 +-10.4  Intel Corporation 82576 Virtual Function
             |                 +-10.6  Intel Corporation 82576 Virtual Function
             |                 +-11.0  Intel Corporation 82576 Virtual Function
             |                 +-11.2  Intel Corporation 82576 Virtual Function
             |                 \-11.4  Intel Corporation 82576 Virtual Function
             +-09.0-[0000:04]--+-00.0  Intel Corporation 82576 Gigabit Network Connection
             |                 \-00.1  Intel Corporation 82576 Gigabit Network Connection
             +-13.0  Intel Corporation 5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller
             +-14.0  Intel Corporation 5520/5500/X58 I/O Hub System Management Registers
             +-14.1  Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers
             +-14.2  Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers
             +-14.3  Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers
             +-1a.0  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
             +-1a.7  Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
             +-1d.0  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
             +-1d.1  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
             +-1d.2  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
             +-1d.7  Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
             +-1e.0-[0000:05]----00.0  XGI Technology Inc. (eXtreme Graphics Innovation) Z9s/Z9m (XG21 core)
             +-1f.0  Intel Corporation 82801JIB (ICH10) LPC Interface Controller
             +-1f.2  Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
             +-1f.3  Intel Corporation 82801JI (ICH10 Family) SMBus Controller
             \-1f.5  Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
euler-linux:/sys/bus/pci/devices/0000:03:00.0 # lspci -vvv -s 03:00.0
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
................[snip]......................
	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 7, Function Dependency Link: 00   ---------------------> Number of VFs is ok, updated properly
		VF offset: 128, stride: 2, Device ID: 10ca
		Supported Page Size: 00000553, System Page Size: 00000001
		VF Migration: offset: 00000000, BIR: 1
	Kernel driver in use: igb
	Kernel modules: igb

euler-linux:/sys/bus/pci/devices/0000:03:00.0 # echo 0 > sriov_numvfs             ---------------------> disable sriov
euler-linux:/sys/bus/pci/devices/0000:03:00.0 # lspci -tv
-+-[0000:ff]-+-00.0  Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers
................[snip]....................
 \-[0000:00]-+-00.0  Intel Corporation 5500 I/O Hub to ESI Port
             +-01.0-[0000:01]--+-00.0  Intel Corporation 82576 Gigabit Network Connection
             |                 \-00.1  Intel Corporation 82576 Gigabit Network Connection
             +-03.0-[0000:02]----00.0  LSI Logic / Symbios Logic SAS1064ET PCI-Express Fusion-MPT SAS
             +-07.0-[0000:03]--+-00.0  Intel Corporation 82576 Gigabit Network Connection
             |                 \-00.1  Intel Corporation 82576 Gigabit Network Connection
             +-09.0-[0000:04]--+-00.0  Intel Corporation 82576 Gigabit Network Connection
             |                 \-00.1  Intel Corporation 82576 Gigabit Network Connection
             +-13.0  Intel Corporation 5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller
             +-14.0  Intel Corporation 5520/5500/X58 I/O Hub System Management Registers
             +-14.1  Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers
             +-14.2  Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers
             +-14.3  Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers
             +-1a.0  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
             +-1a.7  Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
             +-1d.0  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
             +-1d.1  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
             +-1d.2  Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
             +-1d.7  Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
             +-1e.0-[0000:05]----00.0  XGI Technology Inc. (eXtreme Graphics Innovation) Z9s/Z9m (XG21 core)
             +-1f.0  Intel Corporation 82801JIB (ICH10) LPC Interface Controller
             +-1f.2  Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
             +-1f.3  Intel Corporation 82801JI (ICH10 Family) SMBus Controller
             \-1f.5  Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
euler-linux:/sys/bus/pci/devices/0000:03:00.0 # cat sriov_numvfs
0
euler-linux:/sys/bus/pci/devices/0000:03:00.0 # lspci -vvv -s 03:00.0
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
....................[snip].......................
	Capabilities: [140] Device Serial Number 0c-da-21-ff-ff-ba-e2-90
	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 7, Function Dependency Link: 00   --------------------->problem: still the old info, should be updated to 0
		VF offset: 128, stride: 2, Device ID: 10ca
		Supported Page Size: 00000553, System Page Size: 00000001
		VF Migration: offset: 00000000, BIR: 1
	Kernel driver in use: igb
	Kernel modules: igb

euler-linux:/sys/bus/pci/devices/0000:03:00.0 #

Changes in sriov_enable() just avoid device write nr_virtfn to NumVFs register while enable failed.
Changes in sriov_disable() just make sriov update NumVFs properly.


Thanks!
Yijing.


> 
>> Signed-off-by: Yijing Wang <wangyijing@xxxxxxxxxx>
>> ---
>>  drivers/pci/iov.c |    4 +++-
>>  1 files changed, 3 insertions(+), 1 deletions(-)
>>
>> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
>> index de8ffac..6e4844b 100644
>> --- a/drivers/pci/iov.c
>> +++ b/drivers/pci/iov.c
>> @@ -286,7 +286,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
>>             (!(iov->cap & PCI_SRIOV_CAP_VFM) && (nr_virtfn > initial)))
>>                 return -EINVAL;
>>
>> -       pci_write_config_word(dev, iov->pos + PCI_SRIOV_NUM_VF, nr_virtfn);
>>         pci_read_config_word(dev, iov->pos + PCI_SRIOV_VF_OFFSET, &offset);
>>         pci_read_config_word(dev, iov->pos + PCI_SRIOV_VF_STRIDE, &stride);
>>         if (!offset || (nr_virtfn > 1 && !stride))
>> @@ -334,6 +333,7 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
>>                         return rc;
>>         }
>>
>> +       pci_write_config_word(dev, iov->pos + PCI_SRIOV_NUM_VF, nr_virtfn);
>>         iov->ctrl |= PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE;
>>         pci_cfg_access_lock(dev);
>>         pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
>> @@ -368,6 +368,7 @@ failed:
>>         iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
>>         pci_cfg_access_lock(dev);
>>         pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
>> +       pci_write_config_word(dev, iov->pos + PCI_SRIOV_NUM_VF, 0);
>>         ssleep(1);
>>         pci_cfg_access_unlock(dev);
>>
>> @@ -401,6 +402,7 @@ static void sriov_disable(struct pci_dev *dev)
>>                 sysfs_remove_link(&dev->dev.kobj, "dep_link");
>>
>>         iov->num_VFs = 0;
>> +       pci_write_config_word(dev, iov->pos + PCI_SRIOV_NUM_VF, 0);
>>  }
>>
>>  static int sriov_init(struct pci_dev *dev, int pos)
>> --
>> 1.7.1
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 


-- 
Thanks!
Yijing

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [DMA Engine]     [Linux Coverity]     [Linux USB]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [Greybus]

  Powered by Linux