On 23/01/2019 04:10, jianchao.wang wrote: > Looking through the log > https://pastebin.ubuntu.com/p/YSm82GxhNW/ > > rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: > rcu: 6-...0: (13995 ticks this GP) idle=e16/1/0x4000000000000000 softirq=155/155 fqs=655 > rcu: (detected by 4, t=576151 jiffies, g=-391, q=18) > Task dump for CPU 6: > dd R running task 0 677 671 0x00000002 > Call trace: > __switch_to+0x174/0x1e0 > ufshcd_queuecommand+0x84c/0x9a8 > > The task is in RUNNING state when it was scheduled out. > So it should be a preempt (the path is under preemptible rcu). > > And I wonder why it is not scheduled back for so long time that rcu stall was triggered. > And who was occupying the cpu all the time ? > > Would you please try to show all running tasks on all cpu ? > > echo l > /proc/sysrq-trigger > > In addition, since the rcu grace period didn't pass, a lot of things could not be moved > forward. # echo w > /proc/sysrq-trigger [ 517.619818] sysrq: Show Blocked State [ 517.620542] task PC stack pid father # echo l > /proc/sysrq-trigger [ 530.692149] sysrq: Show backtrace of all active CPUs # ps -o pid,user,stat,args PID USER STAT COMMAND 1 root S init 2 root SW [kthreadd] 3 root IW< [rcu_gp] 4 root IW< [rcu_par_gp] 5 root IW [kworker/0:0-eve] 6 root IW< [kworker/0:0H] 7 root IW [kworker/u16:0-u] 8 root IW< [mm_percpu_wq] 9 root SW [ksoftirqd/0] 10 root IW [rcu_preempt] 11 root SW [migration/0] 12 root IW [kworker/0:1-rcu] 13 root SW [cpuhp/0] 14 root SW [cpuhp/1] 15 root SW [migration/1] 16 root SW [ksoftirqd/1] 17 root IW [kworker/1:0-eve] 18 root IW< [kworker/1:0H] 19 root SW [cpuhp/2] 20 root SW [migration/2] 21 root SW [ksoftirqd/2] 22 root IW [kworker/2:0-eve] 23 root IW< [kworker/2:0H] 24 root SW [cpuhp/3] 25 root SW [migration/3] 26 root SW [ksoftirqd/3] 27 root IW [kworker/3:0-eve] 28 root IW< [kworker/3:0H] 29 root SW [cpuhp/4] 30 root SW [migration/4] 31 root SW [ksoftirqd/4] 32 root IW [kworker/4:0-eve] 33 root IW< [kworker/4:0H-kb] 34 root SW [cpuhp/5] 35 root SW [migration/5] 36 root SW [ksoftirqd/5] 37 root IW [kworker/5:0-eve] 38 root IW< [kworker/5:0H] 39 root SW [cpuhp/6] 40 root SW [migration/6] 41 root SW [ksoftirqd/6] 42 root IW [kworker/6:0-eve] 43 root IW< [kworker/6:0H] 44 root SW [cpuhp/7] 45 root SW [migration/7] 46 root SW [ksoftirqd/7] 47 root IW [kworker/7:0-eve] 48 root IW< [kworker/7:0H] 49 root SW [kdevtmpfs] 50 root IW [kworker/u16:1-e] 52 root SW [rcu_tasks_kthre] 61 root IW [kworker/3:1-mm_] 68 root IW [kworker/5:1-eve] 76 root IW [kworker/6:1-eve] 77 root IW [kworker/7:1-mm_] 85 root IW [kworker/1:1-eve] 93 root IW [kworker/2:1-eve] 102 root IW [kworker/4:1-mm_] 202 root SW [khungtaskd] 203 root SW [oom_reaper] 204 root IW< [writeback] 206 root SW [kcompactd0] 208 root IW< [kblockd] 209 root IW< [devfreq_wq] 211 root IW [kworker/u16:2-e] 214 root IW [kworker/u16:3-e] 224 root IW [kworker/u16:4-e] 235 root IW [kworker/u16:5-e] 259 root IW [kworker/u16:6-e] 380 root SW [kswapd0] 509 root SW [scsi_eh_0] 510 root IW< [scsi_tmf_0] 511 root IW< [ufs_clk_gating_] 512 root IW< [ufs_clkscaling_] 551 root IW [kworker/u16:7-e] 635 root IW [kworker/u16:8-u] 656 root S /sbin/klogd -n 674 root S /bin/sh 675 root IW< [kworker/4:1H] 677 root R ps -o pid,user,stat,args # dd if=/dev/sde of=/dev/null bs=1M & while true; do echo l > /proc/sysrq-trigger; sleep 1; done https://pastebin.ubuntu.com/p/Sx4hXRbHby/ What information can I provide to help debug this issue? Regards.
