On Sat, May 26, 2018 at 11:24:01AM +0200, Dmitry Vyukov wrote: > On Sun, May 13, 2018 at 8:21 AM, Eric Biggers <ebiggers3@xxxxxxxxx> wrote: > > On Thu, Apr 05, 2018 at 08:15:24PM -0700, Eric Biggers wrote: > >> On Mon, Jan 29, 2018 at 01:29:48PM +0800, Tianyu Lan wrote: > >> > > >> > > >> > On 1/27/2018 7:27 AM, Eric Biggers wrote: > >> > > On Sat, Dec 16, 2017 at 04:37:02PM +0800, Lan, Tianyu wrote: > >> > > > The root cause is that kvm_irqfd_assign() and kvm_irqfd_deassign() can't > >> > > > be run in parallel. Some data structure(e.g, irqfd->consumer) will be > >> > > > crashed because irqfd may be freed in deassign path before they are used > >> > > > in assign path. The other data maybe used in deassign path before > >> > > > initialization. Syzbot test hit such case. Add mutx between > >> > > > kvm_irqfd_assign() and kvm_irqfd_deassign() can fix such issue. Will > >> > > > send patch to fix it. > >> > > > > >> > > > On 12/16/2017 12:53 PM, Tianyu Lan wrote: > >> > > > > I reproduced the issue. Will have a look. > >> > > > > > >> > > > > -- Best regards Tianyu Lan 2017-12-15 18:14 GMT+08:00 syzbot > >> > > > > <bot+7ad82526b98c64c90062275162dacfa48c20dbd3@xxxxxxxxxxxxxxxxxxxxxxxxx>: > >> > > > > > syzkaller has found reproducer for the following crash on > >> > > > > > 82bcf1def3b5f1251177ad47c44f7e17af039b4b > >> > > > > > git://git.cmpxchg.org/linux-mmots.git/master > >> > > > > > compiler: gcc (GCC) 7.1.1 20170620 > >> > > > > > .config is attached > >> > > > > > Raw console output is attached. > >> > > > > > C reproducer is attached > >> > > > > > syzkaller reproducer is attached. Seehttps://goo.gl/kgGztJ > >> > > > > > for information about syzkaller reproducers > >> > > > > > > >> > > > > > > >> > > > > > ================================================================== > >> > > > > > BUG: KASAN: use-after-free in __list_add include/linux/list.h:64 [inline] > >> > > > > > BUG: KASAN: use-after-free in list_add include/linux/list.h:79 [inline] > >> > > > > > BUG: KASAN: use-after-free in irq_bypass_register_consumer+0x4b4/0x500 > >> > > > > > virt/lib/irqbypass.c:217 > >> > > > > > Write of size 8 at addr ffff8801cdf51180 by task syzkaller436086/15031 > >> > > > > > > >> > > > > > CPU: 1 PID: 15031 Comm: syzkaller436086 Not tainted 4.15.0-rc2-mm1+ #39 > >> > > > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS > >> > > > > > Google 01/01/2011 > >> > > > > > Call Trace: > >> > > > > > __dump_stack lib/dump_stack.c:17 [inline] > >> > > > > > dump_stack+0x194/0x257 lib/dump_stack.c:53 > >> > > > > > print_address_description+0x73/0x250 mm/kasan/report.c:252 > >> > > > > > kasan_report_error mm/kasan/report.c:351 [inline] > >> > > > > > kasan_report+0x25b/0x340 mm/kasan/report.c:409 > >> > > > > > __asan_report_store8_noabort+0x17/0x20 mm/kasan/report.c:435 > >> > > > > > __list_add include/linux/list.h:64 [inline] > >> > > > > > list_add include/linux/list.h:79 [inline] > >> > > > > > irq_bypass_register_consumer+0x4b4/0x500 virt/lib/irqbypass.c:217 > >> > > > > > kvm_irqfd_assign arch/x86/kvm/../../../virt/kvm/eventfd.c:417 [inline] > >> > > > > > kvm_irqfd+0x137f/0x1d50 arch/x86/kvm/../../../virt/kvm/eventfd.c:572 > >> > > > > > kvm_vm_ioctl+0x1079/0x1c40 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2992 > >> > > > > > vfs_ioctl fs/ioctl.c:46 [inline] > >> > > > > > do_vfs_ioctl+0x1b1/0x1530 fs/ioctl.c:686 > >> > > > > > SYSC_ioctl fs/ioctl.c:701 [inline] > >> > > > > > SyS_ioctl+0x8f/0xc0 fs/ioctl.c:692 > >> > > > > > entry_SYSCALL_64_fastpath+0x1f/0x96 > >> > > > > > RIP: 0033:0x44d379 > >> > > > > > RSP: 002b:00007fc5ff9a9d08 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 > >> > > > > > RAX: ffffffffffffffda RBX: 00007fc5ff9aa700 RCX: 000000000044d379 > >> > > > > > RDX: 0000000020080fe0 RSI: 000000004020ae76 RDI: 0000000000000005 > >> > > > > > RBP: 00000000007ff900 R08: 00007fc5ff9aa700 R09: 00007fc5ff9aa700 > >> > > > > > R10: 00007fc5ff9aa700 R11: 0000000000000246 R12: 0000000000000000 > >> > > > > > R13: 00000000007ff8ff R14: 00007fc5ff9aa9c0 R15: 0000000000000000 > >> > > > > > > >> > > > > > Allocated by task 15031: > >> > > > > > save_stack+0x43/0xd0 mm/kasan/kasan.c:447 > >> > > > > > set_track mm/kasan/kasan.c:459 [inline] > >> > > > > > kasan_kmalloc+0xad/0xe0 mm/kasan/kasan.c:551 > >> > > > > > kmem_cache_alloc_trace+0x136/0x750 mm/slab.c:3614 > >> > > > > > kmalloc include/linux/slab.h:516 [inline] > >> > > > > > kzalloc include/linux/slab.h:705 [inline] > >> > > > > > kvm_irqfd_assign arch/x86/kvm/../../../virt/kvm/eventfd.c:296 [inline] > >> > > > > > kvm_irqfd+0x16c/0x1d50 arch/x86/kvm/../../../virt/kvm/eventfd.c:572 > >> > > > > > kvm_vm_ioctl+0x1079/0x1c40 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2992 > >> > > > > > vfs_ioctl fs/ioctl.c:46 [inline] > >> > > > > > do_vfs_ioctl+0x1b1/0x1530 fs/ioctl.c:686 > >> > > > > > SYSC_ioctl fs/ioctl.c:701 [inline] > >> > > > > > SyS_ioctl+0x8f/0xc0 fs/ioctl.c:692 > >> > > > > > entry_SYSCALL_64_fastpath+0x1f/0x96 > >> > > > > > > >> > > > > > Freed by task 1402: > >> > > > > > save_stack+0x43/0xd0 mm/kasan/kasan.c:447 > >> > > > > > set_track mm/kasan/kasan.c:459 [inline] > >> > > > > > kasan_slab_free+0x71/0xc0 mm/kasan/kasan.c:524 > >> > > > > > __cache_free mm/slab.c:3492 [inline] > >> > > > > > kfree+0xca/0x250 mm/slab.c:3807 > >> > > > > > irqfd_shutdown+0x13c/0x1a0 arch/x86/kvm/../../../virt/kvm/eventfd.c:148 > >> > > > > > process_one_work+0xbfd/0x1bc0 kernel/workqueue.c:2113 > >> > > > > > worker_thread+0x223/0x1990 kernel/workqueue.c:2247 > >> > > > > > kthread+0x37a/0x440 kernel/kthread.c:238 > >> > > > > > ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:524 > >> > > > > > > >> > > > > > The buggy address belongs to the object at ffff8801cdf51000 > >> > > > > > which belongs to the cache kmalloc-512 of size 512 > >> > > > > > The buggy address is located 384 bytes inside of > >> > > > > > 512-byte region [ffff8801cdf51000, ffff8801cdf51200) > >> > > > > > The buggy address belongs to the page: > >> > > > > > page:00000000d08a0d63 count:1 mapcount:0 mapping:00000000d54c7be6 index:0x0 > >> > > > > > flags: 0x2fffc0000000100(slab) > >> > > > > > raw: 02fffc0000000100 ffff8801cdf51000 0000000000000000 0000000100000006 > >> > > > > > raw: ffffea00073a7660 ffffea000737f3a0 ffff8801dac00940 0000000000000000 > >> > > > > > page dumped because: kasan: bad access detected > >> > > > > > > >> > > > > > Memory state around the buggy address: > >> > > > > > ffff8801cdf51080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > >> > > > > > ffff8801cdf51100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > >> > > > > > > ffff8801cdf51180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > >> > > > > > ^ > >> > > > > > ffff8801cdf51200: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc > >> > > > > > ffff8801cdf51280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > >> > > > > > ================================================================== > >> > > > > > > >> > > Tianyu, are you still planning to send a patch for this? This is still > >> > > happening. > >> > > > >> > > - Eric > >> > Yes, I sent out fix patch and wait for review. > >> > https://patchwork.kernel.org/patch/10128925/ > >> > > >> > >> This is still happening. Can you please resend your patch? > >> > > > > Still happening; it's pretty easily reproducible, using the program below. > > (Sometimes it crashes in different ways too. It also spams the log, ignore > > that.) Tianyu's patch at https://patchwork.kernel.org/patch/10128925/ fixes it, > > though the locking is difficult to understand so I'm not sure it's correct yet. > > Probably the comment that says "do not drop the file until the irqfd is fully > > initialized, otherwise we might race against the EPOLLHUP" would be obsoleted > > and should be removed? Can the KVM maintainers please review the patch too? > > > > #include <fcntl.h> > > #include <linux/kvm.h> > > #include <pthread.h> > > #include <sys/eventfd.h> > > #include <sys/ioctl.h> > > > > static int dev, vm, event; > > > > static void *irqfd_thrproc(void *_flags) > > { > > struct kvm_irqfd irqfd = { .fd = event, .flags = (long)_flags }; > > > > for (;;) > > ioctl(vm, KVM_IRQFD, &irqfd); > > } > > > > int main() > > { > > pthread_t t; > > > > event = eventfd(0, 0); > > dev = open("/dev/kvm", 0); > > vm = ioctl(dev, KVM_CREATE_VM, 0); > > ioctl(vm, KVM_CREATE_IRQCHIP, 0); > > pthread_create(&t, NULL, irqfd_thrproc, (void *)(long)0); > > irqfd_thrproc((void *)(long)KVM_IRQFD_FLAG_DEASSIGN); > > } > > > Tentative duplicates of this bug: > > BUG: unable to handle kernel NULL pointer dereference in > irq_bypass_register_consumer > KASAN: use-after-free Read in irq_bypass_unregister_consumer > general protection fault in eventfd_ctx_remove_wait_queue > KASAN: use-after-free Read in irq_bypass_register_consumer > > https://syzkaller.appspot.com/bug?id=45591ae3053c59fb50169401fb61cb596735f9d1 Okay, this is no longer occurring and my reproducer no longer works. It seems it was finally fixed by commit b5020a8e6b54d2e: #syz fix: KVM/Eventfd: Avoid crash when assign and deassign specific eventfd in parallel. - Eric
