On Thu, 16 Nov 2023, José Pekkarinen wrote:
On 2023-11-16 07:23, Hugh Dickins wrote:
> On Wed, 15 Nov 2023, Matthew Wilcox wrote:
>> On Wed, Nov 15, 2023 at 06:05:30PM +0200, José Pekkarinen wrote:
>>
>> > > I don't think we should be changing ptlock_ptr().
>> >
>> > This is where the null ptr dereference originates, so the only
>> > alternative I can think of is to protect the life cycle of the ptdesc
>> > to prevent it to die between the pte check and the spin_unlock of
>> > __pte_offset_map_lock. Would that work for you?
>
> Thanks for pursuing this, José, but I agree with Matthew: I don't
> think your patch is right at all. The change in ptlock_ptr() did not
> make sense to me, and the change in __pte_offset_map_lock() leaves us
> still wondering what has gone wrong (and misses an rcu_read_unlock()).
>
> You mentioned "I tested the syzbot reproducer in x86 and it doesn't
> produce this kasan report anymore": are you saying that you were able
> to reproduce the issue on x86 (without your patch)? That would be very
> interesting (and I think would disprove my hypothesis below). I ought
> to try on x86 if you managed to reproduce on it, but it's not worth
> the effort if you did not. If you have an x86 stack and registers,
> please show (though I'm uncertain how much KASAN spoils the output).
Hi,
Yes, I have a local setup based in [1], where I can spin a small
vm, build the reproducer and run it in. The only thing I took from
the webpage is the kernel config file, and the image I made it locally
by debootstrapping and running the modifications in create-image.sh
manually, the kasan report follows:
[ 111.408746][ T8885] general protection fault, probably for
non-canonical
address 0xdffffc0000000005: 0000 [#1] PREEMPT SMP KASAN NOPTI
[ 111.413181][ T8885] KASAN: null-ptr-deref in range
[0x0000000000000028-0x000000000000002f]
[ 111.413181][ T8885] CPU: 1 PID: 8885 Comm: handle_kernel_p Not
tainted
6.7.0-rc1-00007-ge612cb00e200 #6
[ 111.413181][ T8885] Hardware name: QEMU Standard PC (i440FX + PIIX,
1996),
BIOS 1.16.2-debian-1.16.2-1 04/01/2014
[ 111.413181][ T8885] RIP: 0010:__pte_offset_map_lock+0xfa/0x310
[ 111.423642][ T8885] Code: 48 c1 e8 03 80 3c 10 00 0f 85 12 02 00 00
4c 03
3d db 92 cf 0b 48 b8 00 00 00 00 00 fc ff df 49 8d 7f 28 48 89 fa 48
c1 ea 03
<80> 3c 02 00 0f 85 e2 01 00 00 4d 8b 7f 28 4c 89 ff e8 f0 a1 3a 09
[ 111.423642][ T8885] RSP: 0018:ffffc90005baf738 EFLAGS: 00010216
[ 111.423642][ T8885] RAX: dffffc0000000000 RBX: 0005800000000067
RCX:
ffffffff81ada02e
[ 111.423642][ T8885] RDX: 0000000000000005 RSI: ffffffff81ad9f0f
RDI:
0000000000000028
[ 111.423642][ T8885] RBP: ffff8880224c4800 R08: 0000000000000007
R09:
0000000000000000
[ 111.423642][ T8885] R10: 0000000000000000 R11: 0000000000000000
R12:
0005088000000a80
[ 111.423642][ T8885] R13: 1ffff92000b75ee9 R14: ffffc90005bafa88
R15:
0000000000000000
[ 111.423642][ T8885] FS: 00007f8d3972c6c0(0000)
GS:ffff888069700000(0000)
knlGS:0000000000000000
[ 111.423642][ T8885] CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
[ 111.423642][ T8885] CR2: 00007f8d3970af78 CR3: 00000000224d6000
CR4:
00000000000006f0
[ 111.423642][ T8885] Call Trace:
[ 111.423642][ T8885] <TASK>
[ 111.423642][ T8885] ? show_regs+0x8f/0xa0
[ 111.423642][ T8885] ? die_addr+0x4f/0xd0
[ 111.423642][ T8885] ? exc_general_protection+0x150/0x220
[ 111.423642][ T8885] ? asm_exc_general_protection+0x26/0x30
[ 111.423642][ T8885] ? __pte_offset_map_lock+0x1de/0x310
[ 111.423642][ T8885] ? __pte_offset_map_lock+0xbf/0x310
[ 111.423642][ T8885] ? __pte_offset_map_lock+0xfa/0x310
[ 111.423642][ T8885] ? __pte_offset_map_lock+0xbf/0x310
[ 111.423642][ T8885] ? __pfx___pte_offset_map_lock+0x10/0x10
[ 111.423642][ T8885] filemap_map_pages+0x336/0x13b0
[ 111.423642][ T8885] ? __pfx_filemap_map_pages+0x10/0x10
[ 111.423642][ T8885] ? rcu_read_unlock+0x33/0xb0
[ 111.423642][ T8885] do_fault+0x86a/0x1350
[ 111.423642][ T8885] __handle_mm_fault+0xe53/0x23a0
[ 111.423642][ T8885] ? __pfx___handle_mm_fault+0x10/0x10
[ 111.483413][ T8885] handle_mm_fault+0x369/0x890
[ 111.483413][ T8885] __get_user_pages+0x46d/0x15d0
[ 111.483413][ T8885] ? __pfx___get_user_pages+0x10/0x10
[ 111.483413][ T8885] populate_vma_page_range+0x2de/0x420
[ 111.483413][ T8885] ? __pfx_populate_vma_page_range+0x10/0x10
[ 111.483413][ T8885] ? __pfx_find_vma_intersection+0x10/0x10
[ 111.483413][ T8885] ? vm_mmap_pgoff+0x299/0x3c0
[ 111.483413][ T8885] __mm_populate+0x1da/0x380
[ 111.483413][ T8885] ? __pfx___mm_populate+0x10/0x10
[ 111.483413][ T8885] ? up_write+0x1b3/0x520
[ 111.483413][ T8885] vm_mmap_pgoff+0x2d1/0x3c0
[ 111.483413][ T8885] ? __pfx_vm_mmap_pgoff+0x10/0x10
[ 111.483413][ T8885] ksys_mmap_pgoff+0x7d/0x5b0
[ 111.483413][ T8885] __x64_sys_mmap+0x125/0x190
[ 111.483413][ T8885] do_syscall_64+0x45/0xf0
[ 111.483413][ T8885] entry_SYSCALL_64_after_hwframe+0x6e/0x76
[ 111.483413][ T8885] RIP: 0033:0x7f8d39831559
[ 111.483413][ T8885] Code: 08 89 e8 5b 5d c3 66 2e 0f 1f 84 00 00 00
00 00
90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24
08 0f 05
<48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 77 08 0d 00 f7 d8 64 89 01 48
[ 111.483413][ T8885] RSP: 002b:00007f8d3972be78 EFLAGS: 00000216
ORIG_RAX:
0000000000000009
[ 111.483413][ T8885] RAX: ffffffffffffffda RBX: 00007f8d3972c6c0
RCX:
00007f8d39831559
[ 111.483413][ T8885] RDX: b635773f07ebbeea RSI: 0000000000b36000
RDI:
0000000020000000
[ 111.483413][ T8885] RBP: 00007f8d3972bea0 R08: 00000000ffffffff
R09:
0000000000000000
[ 111.483413][ T8885] R10: 0000000000008031 R11: 0000000000000216
R12:
ffffffffffffff80
[ 111.483413][ T8885] R13: 0000000000000000 R14: 00007fffcef921d0
R15:
00007f8d3970c000
[ 111.483413][ T8885] </TASK>
[ 111.483413][ T8885] Modules linked in:
[ 111.763549][ T8885] ---[ end trace 0000000000000000 ]---
[ 111.773557][ T8885] RIP: 0010:__pte_offset_map_lock+0xfa/0x310
[ 111.776045][ T8885] Code: 48 c1 e8 03 80 3c 10 00 0f 85 12 02 00 00
4c 03
3d db 92 cf 0b 48 b8 00 00 00 00 00 fc ff df 49 8d 7f 28 48 89 fa 48
c1 ea 03
<80> 3c 02 00 0f 85 e2 01 00 00 4d 8b 7f 28 4c 89 ff e8 f0 a1 3a 09
[ 111.805040][ T8885] RSP: 0018:ffffc90005baf738 EFLAGS: 00010216
[ 111.820041][ T8885] RAX: dffffc0000000000 RBX: 0005800000000067
RCX:
ffffffff81ada02e
[ 111.837884][ T8885] RDX: 0000000000000005 RSI: ffffffff81ad9f0f
RDI:
0000000000000028
[ 111.855313][ T8885] RBP: ffff8880224c4800 R08: 0000000000000007
R09:
0000000000000000
[ 111.878314][ T8885] R10: 0000000000000000 R11: 0000000000000000
R12:
0005088000000a80
[ 111.910624][ T8885] R13: 1ffff92000b75ee9 R14: ffffc90005bafa88
R15:
0000000000000000
[ 111.923627][ T8885] FS: 00007f8d3972c6c0(0000)
GS:ffff888069700000(0000)
knlGS:0000000000000000
[ 111.932017][ T8885] CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
[ 111.941166][ T8885] CR2: 00007fa26ac38178 CR3: 00000000224d6000
CR4:
00000000000006f0
[ 111.950619][ T8885] Kernel panic - not syncing: Fatal exception
[ 111.953981][ T8885] Kernel Offset: disabled
[ 111.953981][ T8885] Rebooting in 86400 seconds..
I can test some patches for you if it helps finding out
the issue.
Thanks a lot, and you'll see that I've just asked syzbot to try what
I now believe is the correct fix: over in the other thread, since it
didn't recognize yesterday's when I sent from this thread. Please
give that a try yourself, if you have time - thanks.
It turned out that all that I needed was your assurance that you had
the repro working on x86 - I guess I'm simply too x86-centric, and
had assumed that syzbot's arm64 report implied something special on
arm, such as the subtler barriers there.
I gave repro a try on bare metal x86, and it reproduced within a
minute:
though in my case not quite the stack trace you and syzbot reported,
but a more obvious oops in pmd_install(). Depending on one's
"memory model", the macro pfn_to_page() can be more or less strict:
in my case it was strict, and pmd_install() oopsed right there in
pmd_populate(); whereas in your case pmd_populate() uncomplainingly
puts something silly into the pmd entry, leaving
__pte_offset_map_lock()
to stumble on that immediately afterwards. (Neither KASAN nor lockdep
required - though lockdep's spinlock pointer probably helps to make the
badness more obvious, if pmd_install() did not crash already.)
The problem is simply that filemap_map_pmd() assumed that prealloc_pte
is supplied with a preallocated page table whenever pmd_none(); but if
it has racily become pmd_none() since the preallocation decision, then
the oops. My changes have certainly provided an easy way to get that
race, but if I'm not mistaken, there was already another such race,
with the possible bug going back to 5.12.
I'll work on the commit message while waiting to hear from syzbot.
