On Tue, Jan 21, 2025 at 12:19:37PM +0000, Jon Hunter wrote: > > > > [ 59.918177] tegra-xudc 3550000.usb: ep 3 disabled > > > > [ 59.923097] tegra-xudc 3550000.usb: ep 2 disabled > > > > [ 59.927955] tegra-xudc 3550000.usb: ep 5 disabled > > > > [ 80.911432] rcu: INFO: rcu_preempt self-detected stall on CPU > > > > [ 80.917354] rcu: 6-....: (5248 ticks this GP) > > > > idle=ec24/1/0x4000000000000000 softirq=1213/1213 fqs=2623 > > > > [ 80.927146] rcu: (t=5253 jiffies g=3781 q=1490 ncpus=12) > > > > [ 80.932704] Sending NMI from CPU 6 to CPUs 2: > > > > [ 90.981555] CPU: 6 UID: 0 PID: 18 Comm: rcu_exp_gp_kthr Not > > > > tainted 6.13.0-rc7-00043-g619f0b6fad52 #1 > > > > [ 90.981558] Hardware name: NVIDIA NVIDIA Jetson AGX Orin > > > > Developer Kit/Jetson, BIOS 00.0.0-dev-main_92e5ae_88fd1_296de > > > > 12/16/2024 > > > > [ 90.981559] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT > > > > -SSBS BTYPE=--) > > > > [ 90.981562] pc : smp_call_function_single+0xdc/0x1a0 > > > > [ 90.981574] lr : __sync_rcu_exp_select_node_cpus+0x228/0x3c0 > > > > [ 90.981578] sp : ffff800082eb3cd0 > > > > [ 90.981579] x29: ffff800082eb3cd0 x28: 0000000000000010 x27: > > > > ffff0000802933c0 > > > > [ 90.981582] x26: ffff0007a8a1d700 x25: ffff800082895500 x24: > > > > ffff800080132018 > > > > [ 90.981584] x23: 0000000000000014 x22: ffff800081fb7700 x21: > > > > ffff80008280d970 > > > > [ 90.981586] x20: 0000000000000feb x19: ffff800082eb3d00 x18: > > > > 0000000000000000 > > > > [ 90.981588] x17: 0000000000000000 x16: 0000000000000000 x15: > > > > 0000000000000000 > > > > [ 90.981590] x14: ffff000080293440 x13: 0000000000000001 x12: > > > > 0000000000000000 > > > > [ 90.981591] x11: ffff800081fb2388 x10: ffff0000802933c0 x9 : > > > > 0000000000000001 > > > > [ 90.981593] x8 : 0000000000000040 x7 : 0000000000017068 x6 : > > > > ffff800080132018 > > > > [ 90.981595] x5 : 0000000000000000 x4 : ffff0007a8a4f9c8 x3 : > > > > 0000000000000001 > > > > [ 90.981597] x2 : 0000000000000000 x1 : ffff0007a8a4f9c0 x0 : > > > > 0000000000000004 > > > > [ 90.981599] Call trace: > > > > [ 90.981601] smp_call_function_single+0xdc/0x1a0 (P) > > > > [ 90.981605] __sync_rcu_exp_select_node_cpus+0x228/0x3c0 > > > > [ 90.981607] sync_rcu_exp_select_cpus+0x13c/0x2a0 > > > > [ 90.981609] wait_rcu_exp_gp+0x18/0x30 > > > > [ 90.981611] kthread_worker_fn+0xd0/0x188 > > > > [ 90.981614] kthread+0x118/0x11c > > > > [ 90.981619] ret_from_fork+0x10/0x20 > > > > [ 101.416347] sched: DL replenish lagged too much > > > > > > > > > > Odd, you have a usb-serial gadget device in this system that is > > > disconnecting somehow? That oops doesn't point to anything in the usb > > > gadget codebase, "all" we have done is move the call to shutdown the > > > endpoints to earlier in the disconnect function. > > > > Yes the board starts usb-serial and usb-ethernet gadget and on reboot > > when tearing it down I am seeing the above. As soon as it disables the > > tegra-xudc endpoints (as seen above) the board appears to stall. > > > > > I'm glad to revert this, but it feels really odd that this is causing > > > you an rcu stall issue. > > > > Thanks. I can't say I understand it either, but I am certain it is > > caused by this change. > > > > Happy to run any tests to narrow this down a bit. > > > I did a bit more looking at this and I see that we setup a USB gadget device > via the configfs as described in this doc [0]. The RCU stall occurs when we > attempt to disable the gadget on shutdown by ... > > $ echo "" > /path/to/UDC > > Jon > > [0] https://docs.kernel.org/usb/gadget_configfs.html It's an RCU stall so it's probably a locking bug. The original code dropped the port->port_lock, called usb_ep_disable() then took the lock again. There must have been a reason to drop the lock although I haven't found it yet. The new code moved the usb_ep_disable() under the lock. 1431 port->suspended = false; 1432 spin_unlock(&port->port_lock); 1433 spin_unlock_irqrestore(&serial_port_lock, flags); 1434 1435 /* disable endpoints, aborting down any active I/O */ 1436 usb_ep_disable(gser->out); 1437 usb_ep_disable(gser->in); 1438 1439 /* finally, free any unused/unusable I/O buffers */ 1440 spin_lock_irqsave(&port->port_lock, flags); 1441 if (port->port.count == 0) 1442 kfifo_free(&port->port_write_buf); It might help to try testing again with lockdep enabled. regards, dan carpenter
