On 6/24/24 21:10, Yihang Li wrote: >> Thank you for the explanation, but as Niklas said, it would be a lot easier for >> me to recreate the issue if you send the exact commands you execute to trigger >> the issue. E.g. "suspend all disks" in step a can have a lot of different >> meaning depending on which type os suspend you are using... So please send the >> exact commands you use. >> is what exactly ? autosuspend ? or something else ? I am failing to recreate the exact same issue. I do see a lot of bad things happening though, but that is not looking like what you sent. I do endup with the 4 drives connected on my HBA being disabled by libata as revalidate/IDENTIFY fails. And even worse: I hit a deadlock on dev->mutex when I try to do "rmmod pm80xx" after running your test. I am using a pm80xx adapter as that is the only libsas adapter I have. I think your test just kicked a big can of worms... There seem to be a lot of wrong things going on, but I now need to sort out if the problems are with the pm80xx driver, libsas, libata or sd. Probably a combination of all. ATA device suspend/resume has been a constant source of issues since scsi layer switched to doing PM operations asynchronouly. Your issue is latest one. This will take a while to debug. > In step a, I suspend all disks by issuing the following command to all disks > attached to the SAS controller 0000:b4:02.0: > [root@localhost ~]# echo auto > /sys/devices/pci0000:b4/0000:b4:02.0/host6/port-6:0/end_device-6:0/target6:0:0/6:0:0:0/power/control > [root@localhost ~]# echo 5000 > /sys/devices/pci0000:b4/0000:b4:02.0/host6/port-6:0/end_device-6:0/target6:0:0/6:0:0:0/power/autosuspend_delay_ms > ... > [root@localhost ~]# echo auto > /sys/devices/pci0000:b4/0000:b4:02.0/host6/port-6:6/end_device-6:6/target6:0:6/6:0:6:0/power/control > [root@localhost ~]# echo 5000 > /sys/devices/pci0000:b4/0000:b4:02.0/host6/port-6:6/end_device-6:6/target6:0:6/6:0:6:0/power/autosuspend_delay_ms This works as expected on my system and I see my drives going to sleep after 5s. > Step b, Suspend the SAS controller: > [root@localhost ~]# echo auto > /sys/devices/pci0000:b4/0000:b4:02.0/power/control This has no effect for me. Can you confirm that your controller is actually sleeping ? I.e., what do the following show ? cat /sys/devices/pci0000:b4/0000:b4:02.0/power/runtime_active_kids cat /sys/devices/pci0000:b4/0000:b4:02.0/power/runtime_status ? > At this point, the SAS controller is suspended. Next step c is trigger PCI FLR. > [root@localhost ~]# echo 1 > /sys/bus/pci/devices/0000:b4:02.0/reset What does cat /sys/bus/pci/devices/0000:b4:02.0/reset_method is on your system ? Mine is "bus" only. >>> The issue 2: >>> a. Suspend all disks on controller B. >>> b. Suspend controller B. >>> c. Resuming all disks on controller B. >>> d. Run the "lsmod" command to check the driver reference counting. What is the reference count before you do step (a), after you run step (b) and at step (d) ? For my system using the pm80xx driver, I get: pm80xx 352256 0 libsas 155648 1 pm80xx before and after, and that is all normal. But there is the difference that suspending the pm80xx controller does not seem to be supported and does nothing. -- Damien Le Moal Western Digital Research
