Hi *,
Under a very high load of io traffic, we got the below BUG trace.
We can see that:
plist_for_each_entry_safe(this, next, &hb1->chain, list) {
if (match_futex (&this->key, &key1))
were called with hb1 = NULL at futex_wake_up function.
And there is no protection on the code regarding such a scenario.
The NULL can be geting from:
hb1 = hash_futex(&key1);
How can we protect against such a situation?
Deatailes below:
un 10 20:49:40 c-node04 kernel: [97562.144447] STAR4BLKS0_WORK[12672]: segfault at 1aa7cba6 ip 00007f924a6bc68c sp 00007f6ca92e1b70 error 15
Jun 10 20:49:40 c-node04 kram: rpoll(0x23396f70, 24, 50) returning 0 times: 0, 0, 0, 2180, 0 ccount 104
Jun 10 20:49:40 c-node04 kernel: [97562.144463] BUG: unable to handle kernel NULL pointer dereference at 0000000000000246
Jun 10 20:49:40 c-node04 kernel: [97562.145450] PGD 2012ee4067 P4D 2012ee4067 PUD 20135a0067 PMD 0
Jun 10 20:49:40 c-node04 kernel: [97562.145450] Oops: 0000 [#1] SMP
Jun 10 20:49:40 c-node04 kernel: [97562.145450] CPU: 36 PID: 12668 Comm: STAR4BLKS0_WORK Kdump: loaded Tainted: G W OE 4.19.149-KM6 #1
Jun 10 20:49:40 c-node04 kram: rpoll(0x7fe624135b90, 85, 50) returning 0 times: 0, 0, 0, 2203, 0 ccount 42
Jun 10 20:49:40 c-node04 kernel: [97562.145450] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
Jun 10 20:49:40 c-node04 kernel: [97562.145450] RIP: 0010:do_futex+0xdf/0xa90
Jun 10 20:49:40 c-node04 kernel: [97562.145450] Code: 08 4c 8d 6d 08 48 8b 3a 48 8d 72 e8 49 39 d5 4c 8d 67 e8 0f 84 89 00 00 00 31 c0 44 89 3c 24 41 89 df 44 89 f3 41 89 c6 eb 16 <49> 8b 7c
24 18 49 8d 54 24 18 4c 89 e6 4c 39 ea 4c 8d 67 e8 74 58
Jun 10 20:49:40 c-node04 kernel: [97562.145450] RSP: 0018:ffff97f6ea8bbdf0 EFLAGS: 00010283
Jun 10 20:49:40 c-node04 kernel: [97562.145450] RAX: 00007f6db1a5d000 RBX: 0000000000000001 RCX: ffffa5530c5f0140
Jun 10 20:49:40 c-node04 kernel: [97562.145450] RDX: ffff97f6e4287d58 RSI: ffff97f6e4287d40 RDI: 0000000000000246
Jun 10 20:49:40 c-node04 kram: rpoll(0x7fe62414a860, 2, 50) returning 0 times: 0, 0, 0, 2191, 0 ccount 277
Jun 10 20:49:40 c-node04 kernel: [97562.145450] RBP: ffffa5530c5bd580 R08: 00007f6db1a5d9c0 R09: 0000000000000001
Jun 10 20:49:40 c-node04 kernel: [97562.145450] R10: 0000000004000001 R11: 0000000000000000 R12: 000000000000022e
Jun 10 20:49:40 c-node04 kernel: [97562.145450] R13: ffffa5530c5bd588 R14: 0000000000000000 R15: 0000000000000000
Jun 10 20:49:40 c-node04 kernel: [97562.145450] FS: 00007f6cab2e6700(0000) GS:ffff98339f500000(0000) knlGS:0000000000000000
Jun 10 20:49:40 c-node04 kernel: [97562.145450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 10 20:49:40 c-node04 kernel: [97562.145450] CR2: 0000000000000246 CR3: 00000020168d3003 CR4: 00000000003606e0
Jun 10 20:49:40 c-node04 kernel: [97562.145450] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 10 20:49:40 c-node04 kernel: [97562.145450] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jun 10 20:49:40 c-node04 kernel: [97562.145450] Call Trace:
Jun 10 20:49:40 c-node04 kernel: [97562.145450] ? plist_add+0xc1/0xf0
Jun 10 20:49:40 c-node04 kernel: [97562.145450] __x64_sys_futex+0x143/0x1
Jun 10 20:49:40 c-node04 kernel: [97562.145450] do_syscall_64+0x59/0x1b0
Jun 10 20:49:40 c-node04 kernel: [97562.145450] ? prepare_exit_to_usermode+0x70/0x90
Jun 10 20:49:40 c-node04 kram: rpoll(0x233a2b50, 1, 50) returning 0 times: 0, 0, 0, 2177, 0 ccount 277
Jun 10 20:49:40 c-node04 kernel: [97562.145450] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jun 10 20:49:40 c-node04 kram: rpoll(0x7f914813d480, 90, 50) returning 0 times: 0, 0, 0, 2166, 0 ccount 35
Jun 10 20:49:40 c-node04 kernel: [97562.145450] RIP: 0033:0x7f924a6bcf28
Jun 10 20:49:40 c-node04 kernel: [97562.145450] Code: 74 1a 49 8b 48 20 44 8b 59 10 41 83 e3 30 41 83 fb 20 74 21 64 0b 34 25 48 00 00 00 41 ba 01 00 00 00 41 b9 01 00 00 04 0f 05 <48> 3d 01 f0 ff ff 73 1f 31 c0 c3 be 8c 00 00 00 49 89 c8 4d 31 d2
RIP: 0010:do_futex+0xdf/0xa90
0xffffffff81138eff is in do_futex (kernel/futex.c:1748).
1743 put_futex_key(&key1);
1744 cond_resched();
1745 goto retry;
1746 }
1747
1748 plist_for_each_entry_safe(this, next, &hb1->chain, list) {
1749 if (match_futex (&this->key, &key1)) {
1750 if (this->pi_state || this->rt_waiter) {
1751 ret = -EINVAL;
1752 goto out_unlock;
(gdb)
plist_for_each_entry_safe(this, next, &hb1->chain, list) {
if (match_futex (&this->key, &key1)) {
This happened in kernel 4.19.149 running on Azure vm
Thx
David
Reply
Forward
MO
