Hi all, While attempting to debug a separate issue, I've been observing tty crashes in v4.15. A suspiciously similar bug was spotted last year [1] on v4.10 but the fix was disputed and it does not look like anything was committed upstream. Interestingly, the bad address dereferenced in both cases was the same in that case as in the crash I see on v4.15 (0x2260), even though the arch is different (powerpc versus arm64 respectively). n_tty_receive_buf_common() is the common ancestor in the backtraces I've seen so far. For me, the crash occurs when booting the system while piping the output of base64 </dev/urandom to the serial console from another machine. The machine is an ARM Juno r0 (arm64). The underlying serial port is a pl011, driven as an SBSA Generic UART by the amba-pl011 driver (because this is what ACPI tells us we have). This is _probably_ not a pl011 bug, given the evidence from other hardware in [1], but I am seeing other strange behaviours including massive character loss that I can't currently explain, so I can't completely rule out driver problems. The crash occurs at least back to v4.13 for me, but is a bit tricky to reproduce deterministically enough for a bisect. In my particular setup I can't boot prior to v4.9, so I have not identified any kernel version that I'm convinced doesn't exhibit this behaviour. I may not have time to debug this much further, but here are a couple of backtraces [2] [3] in case it gives somebody some ideas. I have not so far managed to reproduce this with any meaningful kernel debugging options turned on... Cheers ---Dave [1] [PATCH] tty: Fix crash with flush_to_ldisc() https://lkml.org/lkml/2017/4/6/1004 [2] v4.15 Crashes at n_tty.c:n_tty_receive_buf_common():1688 size_t tail = smp_load_acquire(&ldata->read_tail); [ 0.000000] Booting Linux on physical CPU 0x0000000100 [0x410fd030] [ 0.000000] Linux version 4.15.0 (davem@e103592) (gcc version 5.3.0 (GCC)) #218 SMP PREEMPT Thu Feb 1 14:38:15 GMT 2018 [...] [ 8.765506] Unable to handle kernel paging request at virtual address 00002260 [ 8.772922] Mem abort info: [ 8.775829] ESR = 0x96000004 [ 8.779339] Exception class = DABT (current EL), IL = 32 bits [ 8.785377] SET = 0, FnV = 0 [ 8.788713] EA = 0, S1PTW = 0 [ 8.792011] Data abort info: [ 8.795121] ISV = 0, ISS = 0x00000004 [ 8.798978] CM = 0, WnR = 0 [ 8.801962] user pgtable: 4k pages, 48-bit VAs, pgd = 00000000939f1d6d [ 8.808512] [0000000000002260] *pgd=0000000000000000 [ 8.814180] Internal error: Oops: 96000004 [#1] PREEMPT SMP [ 8.819760] Modules linked in: [ 8.822821] CPU: 4 PID: 5 Comm: kworker/u12:0 Not tainted 4.15.0 #218 [ 8.829270] Hardware name: ARM LTD ARM Juno Development Platform/ARM Juno Development Platform, BIOS EDK II Dec 15 2016 [ 8.840080] Workqueue: events_unbound flush_to_ldisc [ 8.845057] pstate: 80000005 (Nzcv daif -PAN -UAO) [ 8.849855] pc : n_tty_receive_buf_common+0x58/0xa08 [ 8.854827] lr : n_tty_receive_buf_common+0x44/0xa08 [ 8.859798] sp : ffff000009cfbc90 [ 8.863115] x29: ffff000009cfbc90 x28: 0000000000000000 [ 8.868440] x27: ffff800975f8e000 x26: ffff000008b7f000 [ 8.873763] x25: ffff000008f6fbf0 x24: ffff8009768d2008 [ 8.879086] x23: 0000000000000000 x22: 0000000000000000 [ 8.884408] x21: ffff800975c26912 x20: 000000000000010e [ 8.889730] x19: 0000000000000000 x18: 0000000000000000 [ 8.895050] x17: 0000ffffaaa54158 x16: ffff000008213fd0 [ 8.900370] x15: 0000000000000400 x14: 0000000000000000 [ 8.905691] x13: 071c71c71c71c71c x12: 0000000000000ab8 [ 8.911015] x11: 0000000000000000 x10: 0000000000000a00 [ 8.916339] x9 : ffff000009cfbd60 x8 : ffff8009768bc260 [ 8.921662] x7 : 0000000000000004 x6 : ffff800975c26820 [ 8.926982] x5 : ffff00000901b400 x4 : 0000000000000001 [ 8.932303] x3 : 000000000000010e x2 : 0000000000000000 [ 8.937625] x1 : 0000000000000001 x0 : 0000000000002260 [ 8.942950] Process kworker/u12:0 (pid: 5, stack limit = 0x00000000de295945) [ 8.950006] Call trace: [ 8.952456] n_tty_receive_buf_common+0x58/0xa08 [ 8.957078] n_tty_receive_buf2+0x10/0x18 [ 8.961092] tty_ldisc_receive_buf+0x20/0x70 [ 8.965369] tty_port_default_receive_buf+0x40/0x80 [ 8.970253] flush_to_ldisc+0xb4/0xc8 [ 8.973920] process_one_work+0x138/0x338 [ 8.977933] worker_thread+0x130/0x468 [ 8.981684] kthread+0xf8/0x128 [ 8.984827] ret_from_fork+0x10/0x18 [ 8.988408] Code: b9009fbf f90047a0 d2844c00 8b160000 (c8dffc03) [ 8.994511] ---[ end trace 6be55dd9289e1824 ]--- [ 36.089289] random: crng init done [3] v4.13 Crashes at serial_core.c:uart_write_room():594 addr2line says it's in a call to uart_port_ref(), so the above might be off by one line, with the crash actually happening on uart_port_lock(). [ 0.000000] Booting Linux on physical CPU 0x100 [ 0.000000] Linux version 4.13.0 (davem@e103592) (gcc version 5.3.0 (GCC)) #216 SMP PREEMPT Thu Feb 1 13:41:36 GMT 2018 [ 7.917766] Unable to handle kernel NULL pointer dereference at virtual address 00000178 [ 7.925899] user pgtable: 4k pages, 48-bit VAs, pgd = ffff800975ecb000 [ 7.932448] [0000000000000178] *pgd=00000009f613c003, *pud=00000009f5d06003, *pmd=0000000000000000 [ 7.941460] Internal error: Oops: 96000006 [#1] PREEMPT SMP [ 7.947047] Modules linked in: [ 7.950115] CPU: 2 PID: 45 Comm: kworker/u12:2 Not tainted 4.13.0 #216 [ 7.956658] Hardware name: ARM LTD ARM Juno Development Platform/ARM Juno Development Platform, BIOS EDK II Dec 15 2016 [ 7.967479] Workqueue: events_unbound flush_to_ldisc [ 7.972461] task: ffff8009769aaa00 task.stack: ffff800976a4c000 [ 7.978395] PC is at uart_write_room+0x10/0x130 [ 7.982934] LR is at tty_write_room+0x18/0x28 [ 7.987299] pc : [<ffff0000085373b8>] lr : [<ffff00000851ea48>] pstate: 80000145 [ 7.994706] sp : ffff800976a4fc00 [ 7.998021] x29: ffff800976a4fc00 x28: ffff000009f042a0 [ 8.003345] x27: ffff800975e5a000 x26: 0000000000000011 [ 8.008668] x25: ffff000009f02000 x24: ffff000009f02000 [ 8.013995] x23: ffff8009764265e0 x22: ffff000009f02000 [ 8.019318] x21: 0000000000000008 x20: ffff800975e5a000 [ 8.024640] x19: 0000000000000000 x18: 0000000000000000 [ 8.029962] x17: 0000000000000000 x16: 0000000000000000 [ 8.035282] x15: 0000000000000000 x14: 00000001d7ee1690 [ 8.040603] x13: 0000000000000000 x12: ffff8009764264e0 [ 8.045923] x11: 0000000000000000 x10: 0000000000000a00 [ 8.051244] x9 : 000000000000006c x8 : ffff8009769ab460 [ 8.056565] x7 : ffff8009764264e0 x6 : ffff000009f02000 [ 8.061885] x5 : ffff000008f74770 x4 : 0000000000000000 [ 8.067206] x3 : 0000000000000000 x2 : 0000000000000000 [ 8.072526] x1 : ffff0000085373a8 x0 : ffff800975e5a000 [ 8.077850] Process kworker/u12:2 (pid: 45, stack limit = 0xffff800976a4c000) [ 8.084995] Stack: (0xffff800976a4fc00 to 0xffff800976a50000) [ 8.090751] fc00: ffff800976a4fc20 ffff00000851ea48 ffff8009764265cf ffff800975e5a000 [ 8.098595] fc20: ffff800976a4fc30 ffff00000851b600 ffff800976a4fc80 ffff00000851d984 [ 8.106439] fc40: ffff8009764265cf 0000000000000011 ffff8009764264cf ffff000009f02000 [ 8.114282] fc60: ffff8009764265e0 ffff000009f02000 ffff000009f02000 ffff000009f02000 [ 8.122127] fc80: ffff800976a4fd20 ffff00000851e140 ffff8009764264cf ffff8009764265cf [ 8.129970] fca0: 0000000000000011 ffff8009769bdc80 0000000000000000 ffff8009765e8008 [ 8.137814] fcc0: ffff800977808078 ffff8009769aaa00 ffff800976a1b000 00000000fffffef7 [ 8.145658] fce0: ffff800975e5a0c0 ffff0000089709f4 ffff000009f02000 ffff8009764264e0 [ 8.153501] fd00: ffff80090000006c ffff000008af53b8 ffff000009f02030 0000000000000001 [ 8.161345] fd20: ffff800976a4fd30 ffff000008520c10 ffff800976a4fd50 ffff000008521728 [ 8.169188] fd40: ffff8009764264cf ffff8009764265cf ffff800976a4fd80 ffff000008520e24 [ 8.177032] fd60: ffff800976426400 ffff8009765e8000 ffff8009765e8008 ffff8009765e8028 [ 8.184876] fd80: ffff800976a4fdb0 ffff0000080d9ea8 0000000000000000 ffff800976a1b000 [ 8.192719] fda0: ffff800977808000 ffff800977807000 ffff800976a4fdf0 ffff0000080da108 [ 8.200563] fdc0: ffff800977808000 ffff800977808000 ffff800976a1b030 ffff800977808020 [ 8.208407] fde0: ffff000008eb7000 ffff800977808020 ffff800976a4fe60 ffff0000080e0164 [ 8.216250] fe00: ffff800976a9d000 ffff800976a1a100 ffff000008fd4ef0 ffff8009769aaa00 [ 8.224094] fe20: ffff000008c65de8 ffff800976a1b000 ffff0000080da0b8 ffff800976a9d038 [ 8.231938] fe40: ffff800976a2bd10 0000000000000000 ffff8009778082a8 ffff8009769aaa00 [ 8.239781] fe60: 0000000000000000 ffff0000080836c0 ffff0000080e0068 ffff800976a1a100 [ 8.247624] fe80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.255467] fea0: 0000000000000000 0000000000000000 0000000000000000 7240018bb0e46014 [ 8.263310] fec0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.271153] fee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.278995] ff00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.286838] ff20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.294680] ff40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.302523] ff60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.310365] ff80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.318208] ffa0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 8.326050] ffc0: 0000000000000000 0000000000000005 0000000000000000 0000000000000000 [ 8.333893] ffe0: 0000000000000000 0000000000000000 401e930a160314e0 02074823020d8802 [ 8.341732] Call trace: [ 8.344179] Exception stack(0xffff800976a4fa30 to 0xffff800976a4fb60) [ 8.350629] fa20: 0000000000000000 0001000000000000 [ 8.358472] fa40: ffff800976a4fc00 ffff0000085373b8 ffff000008eb9000 0000000000000001 [ 8.366315] fa60: ffff800976894b00 0000000000000000 ffff000008eb9000 ffff80097ff8f400 [ 8.374159] fa80: 0000000000000000 000000000000001e ffff800976a4fba0 000000000000000b [ 8.382001] faa0: 0000000000000016 0000000000000016 0000000000000000 0000000000000800 [ 8.389845] fac0: 00000000000001d9 0000000200000000 ffff800975e5a000 ffff0000085373a8 [ 8.397688] fae0: 0000000000000000 0000000000000000 0000000000000000 ffff000008f74770 [ 8.405531] fb00: ffff000009f02000 ffff8009764264e0 ffff8009769ab460 000000000000006c [ 8.413375] fb20: 0000000000000a00 0000000000000000 ffff8009764264e0 0000000000000000 [ 8.421217] fb40: 00000001d7ee1690 0000000000000000 0000000000000000 0000000000000000 [ 8.429063] [<ffff0000085373b8>] uart_write_room+0x10/0x130 [ 8.434643] [<ffff00000851ea48>] tty_write_room+0x18/0x28 [ 8.440049] [<ffff00000851b600>] __process_echoes+0x28/0x298 [ 8.445717] [<ffff00000851d984>] n_tty_receive_buf_common+0x25c/0xa08 [ 8.452167] [<ffff00000851e140>] n_tty_receive_buf2+0x10/0x18 [ 8.457922] [<ffff000008520c10>] tty_ldisc_receive_buf+0x20/0x70 [ 8.463938] [<ffff000008521728>] tty_port_default_receive_buf+0x40/0x80 [ 8.470562] [<ffff000008520e24>] flush_to_ldisc+0xb4/0xc8 [ 8.475970] [<ffff0000080d9ea8>] process_one_work+0x118/0x328 [ 8.481724] [<ffff0000080da108>] worker_thread+0x50/0x428 [ 8.487132] [<ffff0000080e0164>] kthread+0xfc/0x128 [ 8.492017] [<ffff0000080836c0>] ret_from_fork+0x10/0x50 [ 8.497338] Code: a9be7bfd 910003fd a90153f3 f9413013 (b9417a63) [ 8.503442] ---[ end trace a8be11ef13b6a5b5 ]--- -- To unsubscribe from this list: send the line "unsubscribe linux-serial" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
