Hi linux-pm/linux-scsi,
We have included fix [1] but continuing to observe supplier loosing
track of consumer.
Below is trace snippet with additional logging added.
Here consumer is 0:0:0:0 and supplier is 0:0:0:49488. In Last three
lines consumer resume is completed but supplier is put down.
kworker/u16:0-7 0.880014: rpm_idle: 0:0:0:0 flags-4
cnt-0 dep-0 auto-1 p-0 irq-0 child-0
kworker/u16:0-7 0.880017: bprint:
pm_runtime_mark_last_busy.46700: :#205 dev_name:0:0:0:0
ktime_get_mono_fast_ns():852365364
kworker/u16:0-7 0.880019: rpm_suspend: 0:0:0:0 flags-8
cnt-0 dep-0 auto-1 p-0 irq-0 child-0
kworker/u16:0-7 0.880022: bprint: pm_runtime_put_noidle.44083:
pm_runtime_put_noidle: #112 dev_name:0:0:0:49488 dev usage_count:5
decremented usage count
kworker/u16:0-7 0.880023: bprint: pm_runtime_put_noidle.44083:
pm_runtime_put_noidle: #112 dev_name:0:0:0:49488 dev usage_count:4
decremented usage count
kworker/u16:2-142 0.880024: rpm_resume: 0:0:0:0 flags-4
cnt-1 dep-0 auto-1 p-0 irq-0 child-0
kworker/u16:0-7 0.880025: bprint: __rpm_put_suppliers:
__rpm_put_suppliers: #348 consumer:0:0:0:0 supplier:0:0:0:49488
usage_count:4
kworker/u16:0-7 0.880061: rpm_idle: 0:0:0:49488
flags-1 cnt-4 dep-0 auto-1 p-0 irq-0 child-0
kworker/u16:0-7 0.880062: rpm_return_int:
rpm_idle+0x16c:0:0:0:49488 ret=-11
kworker/u16:2-142 0.880062: bprint: __pm_runtime_resume:
__pm_runtime_resume: #1147 dev_name:0:0:0:49488 dev usage_count:5
incremented usage count
kworker/u16:2-142 0.880063: rpm_resume: 0:0:0:49488
flags-4 cnt-5 dep-0 auto-1 p-0 irq-0 child-0
kworker/u16:2-142 0.880063: rpm_return_int:
rpm_resume+0x690:0:0:0:49488 ret=1
kworker/u16:0-7 0.880063: rpm_return_int:
rpm_suspend+0x68:0:0:0:0 ret=0
kworker/u16:2-142 0.880063: bprint: rpm_get_suppliers:
rpm_get_suppliers: #300 consumer:0:0:0:0 supplier:0:0:0:49488 usage_count:5
kworker/u16:0-7 0.880065: bprint: pm_runtime_put_noidle.44083:
pm_runtime_put_noidle: #112 dev_name:0:0:0:49488 dev usage_count:4
decremented usage count
kworker/u16:2-142 0.880065: bprint:
pm_runtime_mark_last_busy.44088: :#205 dev_name:0:0:0:0
ktime_get_mono_fast_ns():852413749
kworker/u16:2-142 0.880065: rpm_idle: 0:0:0:0 flags-1
cnt-1 dep-0 auto-1 p-0 irq-0 child-0
kworker/u16:2-142 0.880065: rpm_return_int: rpm_idle+0x16c:0:0:0:0
ret=-11
kworker/u16:0-7 0.880066: bprint: __rpm_put_suppliers:
__rpm_put_suppliers: #348 consumer:0:0:0:0 supplier:0:0:0:49488
usage_count:4
kworker/u16:0-7 0.880067: rpm_return_int: rpm_idle+0x16c:0:0:0:0
ret=-16
kworker/u16:2-142 0.880067: rpm_return_int:
rpm_resume+0x690:0:0:0:0 ret=0
Upon looking into this further the race looks to be in below two
processes running in parallel and process-1 is putting down supplier at
[C] because process-2 is setting runtime_status as resuming at [D].
Also as per runtime PM documentation
In order to use autosuspend, subsystems or drivers must call
pm_runtime_use_autosuspend(), and thereafter they should use the various
`*_autosuspend()` helper functions...
It was also observed that *_autosuspend() API at point [A] was invoked
without first invoking pm_runtime_use_autosuspend() which return
expiration as zero at point [B] and proceeds ahead for immediate runtime
suspend of device which seems lead to this race condition.
Process -1
ufshcd_async_scan context (process 1)
scsi_autopm_put_device() //0:0:0:0
pm_runtime_put_sync()
__pm_runtime_idle()
rpm_idle() -- RPM_GET_PUT(4)
__rpm_callback
scsi_runtime_idle()
pm_runtime_mark_last_busy()
pm_runtime_autosuspend() --[A]
rpm_suspend() -- RPM_AUTO(8)
pm_runtime_autosuspend_expiration()
use_autosuspend is false return 0 --- [B]
__update_runtime_status to RPM_SUSPENDING
__rpm_callback()
__rpm_put_suppliers(dev, false)
__update_runtime_status to RPM_SUSPENDED
rpm_suspend_suppliers()
rpm_idle() for supplier -- RPM_ASYNC(1) return
(-EAGAIN) [ Other consumer active for supplier]
rpm_suspend() – END with return=0
scsi_runtime_idle() END return (-EBUSY) always.
/* Do that if resume fails too.*/
(dev->power.runtime_status == RPM_RESUMING && retval))) return -EBUSY
__rpm_put_suppliers(dev, false) -- [C]
rpm_idle() END return -EBUSY
Process -2
sd_probe context (Process 2)
scsi_autopm_get_device() //0:0:0:0
__pm_runtime_resume(RPM_GET_PUT)
rpm_resume() -- RPM_GET_PUT(4)
__update_runtime_status to RPM_RESUMING --[D]
__rpm_callback()
rpm_get_suppliers()
__pm_runtime_resume() - RPM_GET_PUT(4) – supplier
rpm_resume() for supplier.
__update_runtime_status to RPM_ACTIVE
pm_runtime_mark_last_busy ()
rpm_resume() END return 0
Can you please provide your suggestions on addressing above race condition?
This is also reported at [2].
[1]: https://lore.kernel.org/lkml/4748074.GXAFRqVoOG@kreacher/T/
[2]: https://lkml.org/lkml/2022/10/12/259
Thanks,
Tushar Nimkar
