Hello,
I found these bugs in 4.9.228-rt147:
[ 665.048394] BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
[ 665.048404] IP: [<ffffffff81197daf>] kmem_cache_alloc+0x6f/0x1d0
[ 665.048406] PGD 0
[ 665.048406]
[ 665.048409] Oops: 0000 [#1] PREEMPT SMP
[ 665.048435] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_nat nfsd
iptable_nat nf_nat_ipv4 nf_nat xt_hl nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter
ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables rtc_pcf8563
coretemp rtc_cmos lm75 max6650 i2c_i801 i2c_smbus uio_cif uvcvideo videobuf2_vmalloc
videobuf2_memops videobuf2_v4l2 videobuf2_core videodev fuse ipv6 autofs4 whiteheat
ti_usb_3410_5052 spcp8x5 safe_serial pl2303 oti6858 option usb_wwan mos7840 mos7720
mct_u232 keyspan_pda keyspan ezusb iuu_phoenix io_ti io_edgeport ftdi_sio f81232
empeg digi_acceleport cypress_m8 ch341 belkin_sa ark3116 aircable radeon ttm i915
[ 665.048439] CPU: 0 PID: 4794 Comm: nmbd Not tainted 4.9.228-rt147cpx+ #172
[ 665.048440] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop
Reference Platform, BIOS 6.00 07/02/2015
[ 665.048441] task: ffff88003a1063c0 task.stack: ffffc900038a8000
[ 665.048445] RIP: 0010:[<ffffffff81197daf>] [<ffffffff81197daf>] kmem_cache_alloc+0x6f/0x1d0
[ 665.048447] RSP: 0018:ffffc900038abb40 EFLAGS: 00010202
[ 665.048448] RAX: 0000000000000000 RBX: ffff880037810e00 RCX: 000000000026ebb0
[ 665.048449] RDX: 000000000026eba8 RSI: 000000000026eba8 RDI: ffff880000090700
[ 665.048450] RBP: ffffc900038abb70 R08: 0000000000041fe0 R09: ffffffff816a21f2
[ 665.048450] R10: ffffc900038abd18 R11: 0000000000000000 R12: 0000000002080020
[ 665.048451] R13: 0000000000000001 R14: ffff880000090700 R15: ffff880000090700
[ 665.048453] FS: 00007fd37700f700(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
[ 665.048500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 665.048514] CR2: 0000000000000001 CR3: 000000003dae6000 CR4: 00000000003606f0
[ 665.048550] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 665.048551] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 665.048552] Stack:
[ 665.048555] ffffc900038abb68 ffff880037810e00 0000000002080020 ffffffff81d11240
[ 665.048557] ffff880039cea300 0000000000004b00 ffffc900038abb90 ffffffff816a21f2
[ 665.048559] ffff880037810e00 ffff880037810f00 ffffc900038abc10 ffffffff817031c5
[ 665.048560] Call Trace:
[ 665.048571] [<ffffffff816a21f2>] skb_clone+0x52/0xa0
[ 665.048578] [<ffffffff817031c5>] ip_mc_output+0x155/0x2a0
[ 665.048581] [<ffffffff81702a3c>] ? __ip_local_out+0xdc/0x140
[ 665.048583] [<ffffffff81700a10>] ? ip_forward_options+0x1b0/0x1b0
[ 665.048586] [<ffffffff81702ad5>] ip_local_out+0x35/0x40
[ 665.048589] [<ffffffff81703d39>] ip_send_skb+0x19/0x40
[ 665.048592] [<ffffffff8172bb54>] udp_send_skb+0x114/0x2b0
[ 665.048594] [<ffffffff8172de17>] udp_sendmsg+0x4f7/0x9b0
[ 665.048597] [<ffffffff81701a20>] ? ip_reply_glue_bits+0x50/0x50
[ 665.048601] [<ffffffff8173883f>] inet_sendmsg+0x7f/0xb0
[ 665.048604] [<ffffffff81699288>] sock_sendmsg+0x38/0x50
[ 665.048606] [<ffffffff8169a793>] SyS_sendto+0xf3/0x140
[ 665.048612] [<ffffffff810d3449>] ? ktime_get_ts64+0x49/0xf0
[ 665.048615] [<ffffffff81002bab>] do_syscall_64+0x5b/0xd0
[ 665.048621] [<ffffffff817aec7e>] entry_SYSCALL_64_after_swapgs+0x58/0xc6
[ 665.048645] Code: 48 8b 51 08 48 89 c8 65 48 03 05 b5 53 e7 7e 48 8b 70 08 48
39 f2 75 e7 4c 8b 28 4d 85 ed 74 6d 49 63 46 20 4d 8b 06 48 8d 4a 08 <49> 8b 5c 05
00 4c 89 e8 65 49 0f c7 08 0f 94 c0 84 c0 74 bd 49
[ 665.048647] RIP [<ffffffff81197daf>] kmem_cache_alloc+0x6f/0x1d0
[ 665.048648] RSP <ffffc900038abb40>
[ 665.048649] CR2: 0000000000000001
[ 665.243152] ---[ end trace 0000000000000002 ]---
and
[ 576.690934] general protection fault: 0000 [#1] PREEMPT SMP
[ 576.690960] Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_nat nfsd
iptable_nat nf_nat_ipv4 nf_nat xt_hl nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter
ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables rtc_pcf8563
coretemp rtc_cmos lm75 max6650 i2c_i801 i2c_smbus uio_cif uvcvideo videobuf2_vmalloc
videobuf2_memops videobuf2_v4l2 videobuf2_core videodev fuse ipv6 autofs4 whiteheat
ti_usb_3410_5052 spcp8x5 safe_serial pl2303 oti6858 option usb_wwan mos7840 mos7720
mct_u232 keyspan_pda keyspan ezusb iuu_phoenix io_ti io_edgeport ftdi_sio f81232
empeg digi_acceleport cypress_m8 ch341 belkin_sa ark3116 aircable radeon ttm i915
[ 576.690963] CPU: 0 PID: 11963 Comm: kworker/0:0 Not tainted 4.9.228-rt147cpx+
#170
[ 576.690965] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop
Reference Platform, BIOS 6.00 07/02/2015
[ 576.690975] Workqueue: events key_garbage_collector
[ 576.690977] task: ffff8800397663c0 task.stack: ffffc90004000000
[ 576.690985] RIP: 0010:[<ffffffff8139ccfa>] [<ffffffff8139ccfa>] rb_next+0x1a/0x50
[ 576.690986] RSP: 0018:ffffc90004003dd8 EFLAGS: 00010286
[ 576.690987] RAX: ff880039484308ff RBX: 7fffffffffffffff RCX: 0000000000000000
[ 576.690988] RDX: ff88003781050914 RSI: ffff8800397663c0 RDI: ffff880037810f08
[ 576.690989] RBP: ffffc90004003dd8 R08: ffff88003ca61100 R09: 0000000000000000
[ 576.690990] R10: ffff8800370ee080 R11: 0000000000000000 R12: ffff8800397663c0
[ 576.690991] R13: ffff8800397663c0 R14: 000000005efc6657 R15: ffff880037810f08
[ 576.690993] FS: 0000000000000000(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
[ 576.691036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 576.691050] CR2: 00007f2113b2c000 CR3: 000000003a5bd000 CR4: 00000000003606f0
[ 576.691088] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 576.691089] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 576.691090] Stack:
[ 576.691093] ffffc90004003e28 ffffffff8133bef3 ffff880039766970 ffff8800370e0000
[ 576.691095] ffffc90004003e40 ffff88003d5236c0 ffff88003fc3e340 0000000000000000
[ 576.691097] ffff88003fc42b00 ffffffff81cc3e60 ffffc90004003e68 ffffffff8107b035
[ 576.691098] Call Trace:
[ 576.691104] [<ffffffff8133bef3>] key_garbage_collector+0xa3/0x3d0
[ 576.691112] [<ffffffff8107b035>] process_one_work+0x145/0x450
[ 576.691114] [<ffffffff8107b3a9>] worker_thread+0x69/0x4f0
[ 576.691117] [<ffffffff8107b340>] ? process_one_work+0x450/0x450
[ 576.691120] [<ffffffff810806ac>] kthread+0xec/0x110
[ 576.691122] [<ffffffff810805c0>] ? kthread_park+0x60/0x60
[ 576.691128] [<ffffffff817aed8e>] ret_from_fork+0x3e/0x50
[ 576.691151] Code: 52 08 48 85 d2 75 eb 5d c3 31 c0 5d c3 0f 1f 40 00 55 48 8b
17 48 89 e5 48 39 d7 74 35 48 8b 47 08 48 85 c0 75 05 eb 1a 48 89 d0 <48> 8b 50 10
48 85 d2 75 f4 5d c3 48 3b 79 08 75 11 48 8b 11 48
[ 576.691154] RIP [<ffffffff8139ccfa>] rb_next+0x1a/0x50
[ 576.691155] RSP <ffffc90004003dd8>
[ 576.832119] ---[ end trace 0000000000000002 ]---
Digging into this showed that this happens when sending rt signals to rt processes.
When sending rt signals to a rt process and at the same time connecting multiple
times via ssh triggered the bug within minutes.
When searching for a reason I stumbled over a this commit by Linus:
4306259ff6b8b682322d9aeb0c12b27c61c4a548
signal: avoid double atomic counter increments for user accounting
[ Upstream commit fda31c50292a5062332fa0343c084bd9f46604d9 ]
These changes were not back ported to the following rt patch by Thomas which
resulted in a possible use after free:
bbe92cd3d383dafe071b0f1c45b3603b8666fb88
signals: Allow rt tasks to cache one sigqueue struct
To avoid allocation allow rt tasks to cache one sigqueue struct in
task struct.
So with the following changes I could not reproduce the bugs:
diff --git a/kernel/signal.c b/kernel/signal.c
index aca4b4cd11a2..1f335772615c 100644
--- a/kernel/signal.c
+++ b/kernel/signal.c
@@ -456,8 +456,8 @@ static void sigqueue_free_current(struct sigqueue *q)
up = q->user;
if (rt_prio(current->normal_prio) && !put_task_cache(current, q)) {
- atomic_dec(&up->sigpending);
- free_uid(up);
+ if (atomic_dec_and_test(&q->user->sigpending))
+ free_uid(up);
} else
__sigqueue_free(q);
}
When answering to this mail could you please cc me for I am not subscribed to
linux-rt-users list.
Thanks and kind regards,
Matthias
