one checkpoint: I saw they use serial port for console at kernel
parameter: console=ttyS0,115200n8
*
Booting Linux using a console connection that is too slow to keep up
with the boot-time console-message rate. For example, a 115Kbaud
serial console can be/way/too slow to keep up with boot-time message
rates, and will frequently result in RCU CPU stall warning messages.
Especially if you have added debug|printk()|
<https://www.kernel.org/doc/html/latest/core-api/printk-basics.html#c.printk>s.
On 8/11/2023 4:31 PM, Felix Kuehling wrote:
If you have a complete kernel log, it may be worth looking at
backtraces from other threads, to better understand the interactions.
I'd expect that there is a thread there that's in an RCU read critical
section. It may not be in our driver, though. If it's a customer
system, it may also help to see the kernel config. Maybe the kernel
was configured without preemption:
- For !CONFIG_PREEMPTION kernels, a CPU looping anywhere in the
kernel
without invoking schedule(). If the looping in the kernel is
really expected and desirable behavior, you might need to add
some calls to cond_resched().
But then I would expect cond_resched() to fix the problem, according
to this document.
Regards,
Felix
On 2023-08-11 17:27, Chen, Xiaogang wrote:
On 8/11/2023 4:22 PM, Felix Kuehling wrote:
On 2023-08-11 17:12, Chen, Xiaogang wrote:
I know the original jira ticket. The system got RCU cpu stall, then
kernel enter panic, then no response or ssh. This patch let prange
list update task yield cpu after each range update. It can prevent
task holding mm lock too long.
Calling schedule does not drop the lock. If anything, it causes the
lock to be held longer, because the function takes longer to complete.
Regards,
Felix
Right. I do not see either how this patch target the root cause. It
is on customer system that can have many RCU operations(not necessary
from our code). Any read critical section can cause write stall.
I think we can use some RCU parameters first to see if thing can
change: like config_rcu_cpu_stall_timeout to increase grace period,
or rcuupdate.rcu_cpu_stall_suppress to surppress RCU stall.
Regards
Xiaogang
mm lock is rw_semophore, not RCU mechanism. Can you explain how
that can prevent RCU cpu stall in this case?
Regards
Xiaogang
On 8/11/2023 2:11 PM, James Zhu wrote:
Caution: This message originated from an External Source. Use
proper caution when opening attachments, clicking links, or
responding.
update_list could be big in list_for_each_entry(prange,
&update_list, update_list),
mmap_read_lock(mm) is kept hold all the time, adding schedule()
can remove
RCU stall on CPU for this case.
RIP: 0010:svm_range_cpu_invalidate_pagetables+0x317/0x610 [amdgpu]
Code: 00 00 00 bf 00 02 00 00 48 81 c2 90 00 00 00 e8 1f 6a b9 e0
65 48 8b 14 25 00 bd 01 00 8b 42 2c 48 8b 3c 24 80 e4 f7 0b 43 d8
<89> 42 2c e8 51 dd 2d e1 48 8b 7b 38 e8 98 29 b7 e0 48 83 c4 30 b8
RSP: 0018:ffffc9000ffd7b10 EFLAGS: 00000206
RAX: 0000000000000100 RBX: ffff88c493968d80 RCX: ffff88d1a6469b18
RDX: ffff88e18ef1ec80 RSI: ffffc9000ffd7be0 RDI: ffff88c493968d38
RBP: 000000000003062e R08: 000000003042f000 R09: 000000003062efff
R10: 0000000000001000 R11: ffff88c1ad255000 R12: 000000000003042f
R13: ffff88c493968c00 R14: ffffc9000ffd7be0 R15: ffff88c493968c00
__mmu_notifier_invalidate_range_start+0x132/0x1d0
? amdgpu_vm_bo_update+0x3fd/0x520 [amdgpu]
migrate_vma_setup+0x6c7/0x8f0
? kfd_smi_event_migration_start+0x5f/0x80 [amdgpu]
svm_migrate_ram_to_vram+0x14e/0x580 [amdgpu]
svm_range_set_attr+0xe34/0x11a0 [amdgpu]
kfd_ioctl+0x271/0x4e0 [amdgpu]
? kfd_ioctl_set_xnack_mode+0xd0/0xd0 [amdgpu]
__x64_sys_ioctl+0x92/0xd0
Signed-off-by: James Zhu <James.Zhu@xxxxxxx>
---
drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
index 113fd11aa96e..9f2d48ade7fa 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c
@@ -3573,6 +3573,7 @@ svm_range_set_attr(struct kfd_process *p,
struct mm_struct *mm,
r = svm_range_trigger_migration(mm, prange,
&migrated);
if (r)
goto out_unlock_range;
+ schedule();
if (migrated && (!p->xnack_enabled ||
(prange->flags &
KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED)) &&
--
2.34.1
