(CC+: linux-pci)
On Jun 11, 2022 / 16:34, Yi Zhang wrote:
> On Fri, Jun 10, 2022 at 10:49 PM Keith Busch <kbusch@xxxxxxxxxx> wrote:
> >
> > On Fri, Jun 10, 2022 at 12:25:17PM +0000, Shinichiro Kawasaki wrote:
> > > On Jun 10, 2022 / 09:32, Chaitanya Kulkarni wrote:
> > > > >> #6: nvme/032: Failed at the first run after system reboot.
> > > > >> Used QEMU NVME device as TEST_DEV.
> > > > >>
> > > >
> > > > ofcourse we need to fix this issue but can you also
> > > > try it with the real H/W ?
> > >
> > > Hi Chaitanya, thank you for looking into the failures. I have just run the test
> > > case nvme/032 with real NVME device and observed the exactly same symptom as
> > > QEMU NVME device.
> >
> > QEMU is perfectly fine for this test. There's no need to bring in "real"
> > hardware for this.
> >
> > And I am not even sure this is real. I don't know yet why this is showing up
> > only now, but this should fix it:
>
> Hi Keith
>
> Confirmed the WARNING issue was fixed with the change, here is the log:
Thanks. I also confirmed that Keith's change to add __ATTR_IGNORE_LOCKDEP to
dev_attr_dev_rescan avoids the fix, on v5.19-rc2.
I took a closer look into this issue and found The deadlock WARN can be
recreated with following two commands:
# echo 1 > /sys/bus/pci/devices/0000\:00\:09.0/rescan
# echo 1 > /sys/bus/pci/devices/0000\:00\:09.0/remove
And it can be recreated with PCI devices other than NVME controller, such as
SCSI controller or VGA controller. Then this is not a storage sub-system issue.
I checked function call stacks of the two commands above. As shown below, it
looks like ABBA deadlock possibility is detected and warned.
echo 1 > /sys/bus/pci/devices/*/rescan
kernfs_fop_write_iter
kernfs_get_active
atomic_inc_unless_nagative
rwsem_acquire_read(&kn->dep_map, 0, 1, _RET_IP) :lock L1 for "rescan" file
dev_rescan_store
pci_lock_rescan_remove
mutex_lock(&pci_rescan_remove_lock) :lock L2
echo 1 > /sys/bus/pci/devices/*/remove
kernfs_fop_write_iter
remove_store
pci_stop_and_remove_bus_device_locked
pci_lock_rescan_remove
mutex_lock(&pci_rescan_remove_lock) :lock L2
pci_stop_and_remove_bus_device
pci_remove_bus_device
device_del
device_remove_attrs
sysfs_remove_attrs
sysfs_remove_groups
sysfs_remove_group
remove_files .... iterates for pci device sysfs files including "rescan" file?
kernfs_remove_by_name_ns
__kernfs_remove
kernfs_drain
rwsem_acquire(&kn->dep_map, 0, 0, _RET_IP): lock L1
It looks for me that the deadlock possibility exists for real, even though the
race between rescan operation and remove operation is really rare use case.
I would like to hear comments on the guess above. I take the liberty to CC this
to linux-pci list. Is this an issue to fix?
--
Shin'ichiro Kawasaki
