On Wed, Aug 14, 2013 at 12:44 PM, Bjorn Helgaas <bhelgaas@xxxxxxxxxx> wrote:
> [+cc Don]
>
> On Fri, Aug 9, 2013 at 5:44 PM, Yinghai Lu <yinghai@xxxxxxxxxx> wrote:
>> On Fri, Aug 9, 2013 at 10:04 AM, Bjorn Helgaas <bhelgaas@xxxxxxxxxx> wrote:
>
>>> What would happen if we just reverted dc087f2f6a2 for now? Jiang?
>
> I tried to reproduce the problem you reported ("Cannot add device at
> 0000:02:00"), but I haven't been able to. I have an ixgbe SR-IOV
> device in an external hotplug-capable chassis, and I'm using v3.11-rc4
> with a patch to add a little instrumentation. Here's what I see:
>
> # echo -n 8 > /sys/bus/pci/devices/0000:08:00.0/sriov_numvfs
> ixgbe 0000:08:00.0 eth1: SR-IOV enabled with 8 VFs
> pci 0000:08:10.0: [8086:1515] type 00 class 0x020000 # VF1
> pci 0000:08:10.2: [8086:1515] type 00 class 0x020000 # VF2
> ...
> pci 0000:08:11.6: [8086:1515] type 00 class 0x020000 # VF8
>
> Created 8 VFs; no problem there.
>
> # echo -n 0 > /sys/bus/pci/slots/4-1/power
> pciehp 0000:04:03.0:pcie24: disable_slot: physical_slot = 4-1
> pciehp 0000:04:03.0:pcie24: pciehp_unconfigure_device:
> domain:bus:dev = 0000:08:00
> ixgbevf 0000:08:11.6: pci_stop_and_remove_bus_device
> pci 0000:08:11.6: pci_release_dev # release VF8
> ixgbevf 0000:08:11.4: pci_stop_and_remove_bus_device
> pci 0000:08:11.4: pci_release_dev # release VF7
> ...
> ixgbevf 0000:08:10.0: pci_stop_and_remove_bus_device
> pci 0000:08:10.0: pci_release_dev # release VF1
> ixgbe 0000:08:00.1: pci_stop_and_remove_bus_device
> pci 0000:08:00.1: pci_release_dev # release PF1
> ixgbe 0000:08:00.0: pci_stop_and_remove_bus_device # stop/remove PF0
> # PF0 not released
>
> When I turn off power to the slot, pciehp stops and removes all the
> devices, starting with the VFs. The PF 08:00.0 is stopped but not
> released, because we didn't call virtfn_remove(), so the reference
> held by the VFs was never released. This is a problem.
>
> # echo -n 1 > /sys/bus/pci/slots/4-1/power
> pci 0000:08:00.0: [8086:1528] type 00 class 0x020000
> pci 0000:08:00.1: [8086:1528] type 00 class 0x020000
>
> When I turn the slot back on, we find both PFs; no apparent problem there.
>
> On v3.10 (which does not contain dc087f2f6a), I see essentially the
> same thing. The PFs are stopped before the VFs because v3.10 doesn't
> contain 29ed1f29b68, which removes devices in reverse order. But even
> in v3.10, PF0 is stopped but not released because the VFs still hold
> references to it. And there's no complaint when turning the slot back
> on.
>
> So we definitely have a problem: when we remove the PF, we don't
> release it correctly because we don't release the references held by
> VFs. But this is not a regression, and doesn't seem urgent enough to
> rush something for v3.11.
>
> I think we're headed toward tearing down all VFs when removing a PF.
> Obviously, we *have* to do that when powering off the device, and I
> think we also need to do it when removing the PF driver just to keep
> all the bookkeeping sane. I don't know how much heartburn this will
> cause Don.
>
> So tell me if you still think we need to do something for v3.11.
in your test: the PF is not get freed but it is removed for bus's device list.
so pciehp_configure_device will create new one even the old is not
really released.
Please check with attached patch, that only release that from link-list in
pci_release device. I posted it a while back.
After it get applied, you will not have chance to create new device in
pciehp_configure_device.
Thanks
Yinghai
Attachment:
fix_racing_removing_6_2_before.patch
Description: Binary data
