Re: [BUG] xfs/104 triggered NULL pointer dereference in iomap based dio

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Wednesday, September 13, 2017 4:28:23 PM IST Eryu Guan wrote:
> Hi all,
> 
> Recently I noticed multiple crashes triggered by xfs/104 on ppc64 hosts
> in my upstream 4.13 kernel testings. The block layer is involved in the
> call trace so I add linux-block to cc list too. I append the full
> console log to the end of this mail.
> 
> Now I can reproduce the crash on x86_64 hosts too by running xfs/104
> many times (usually within 100 iterations). A git-bisect run (I ran it
> for 500 iterations before calling it good in bisect run to be safe)
> pointed the first bad commit to commit acdda3aae146 ("xfs: use
> iomap_dio_rw").
> 
> I confirmed the bisect result by checking out a new branch with commit
> acdda3aae146 as HEAD, xfs/104 would crash kernel within 100 iterations,
> then reverting HEAD, xfs/104 passed 1500 iterations.

I am able to recreate the issue on my ppc64 guest. I added some printk()
statements and got this,

xfs_fs_fill_super:1670: Filled up sb c0000006344db800.
iomap_dio_bio_end_io:784: sb = c0000006344db800; inode->i_sb->s_dio_done_wq =           (null), &dio->aio.work = c0000006344bb5b0.


In iomap_dio_rw(), I had added the following printk() statement,

               ret = sb_init_dio_done_wq(inode->i_sb);
                if (ret < 0)
                        iomap_dio_set_error(dio, ret);
                printk("%s:%d: sb = %p; Created s_dio_done_wq.\n",
                        __func__, __LINE__, inode->i_sb);

In the case of crash, I don't see the above message being printed. 

Btw, I am unable to recreate this issue on today's linux-next though. Maybe
it is because the race condition is accidently masked out.

I will continue debugging this and provide an update.

> 
> On one of my test vms, the crash happened as
> 
> [  340.419429] BUG: unable to handle kernel NULL pointer dereference at 0000000000000102
> [  340.420408] IP: __queue_work+0x32/0x420
> 
> and that IP points to
> 
> (gdb) l *(__queue_work+0x32)
> 0x9cf32 is in __queue_work (kernel/workqueue.c:1383).
> 1378            WARN_ON_ONCE(!irqs_disabled());
> 1379
> 1380            debug_work_activate(work);
> 1381
> 1382            /* if draining, only works from the same workqueue are allowed */
> 1383            if (unlikely(wq->flags & __WQ_DRAINING) &&
> 1384                WARN_ON_ONCE(!is_chained_work(wq)))
> 1385                    return;
> 1386    retry:
> 1387            if (req_cpu == WORK_CPU_UNBOUND)
> 
> So looks like "wq" is null. The test vm is a kvm guest running 4.13
> kernel with 4 vcpus and 8G memory.
> 
> If more information is needed please let me know.
> 
> Thanks,
> Eryu
> 
> P.S. console log when crashing
> 
> [  339.746983] run fstests xfs/104 at 2017-09-13 17:38:26
> [  340.027352] XFS (vda6): Unmounting Filesystem
> [  340.207107] XFS (vda6): Mounting V5 Filesystem
> [  340.217553] XFS (vda6): Ending clean mount
> [  340.419429] BUG: unable to handle kernel NULL pointer dereference at 0000000000000102
> [  340.420408] IP: __queue_work+0x32/0x420
> [  340.420408] PGD 215373067
> [  340.420408] P4D 215373067
> [  340.420408] PUD 21210d067
> [  340.420408] PMD 0
> [  340.420408]
> [  340.420408] Oops: 0000 [#1] SMP
> [  340.420408] Modules linked in: xfs ip6t_rpfilter ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack libcrc32c ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter btrfs xor raid6_pq ppdev i2c_piix4 parport_pc i2c_core parport virtio_balloon pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi virtio_net virtio_blk ata_piix libata virtio_pci virtio_ring serio_raw virtio floppy
> [  340.420408] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.13.0 #64
> [  340.420408] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
> [  340.420408] task: ffff8b1d96222500 task.stack: ffffb06bc0cb8000
> [  340.420408] RIP: 0010:__queue_work+0x32/0x420
> [  340.420408] RSP: 0018:ffff8b1d9fd83d18 EFLAGS: 00010046
> [  340.420408] RAX: 0000000000000096 RBX: 0000000000000002 RCX: ffff8b1d9489e6d8
> [  340.420408] RDX: ffff8b1d903c2090 RSI: 0000000000000000 RDI: 0000000000002000
> [  340.420408] RBP: ffff8b1d9fd83d58 R08: 0000000000000400 R09: 0000000000000009
> [  340.420408] R10: ffff8b1d9532b400 R11: 0000000000000000 R12: 0000000000002000
> [  340.420408] R13: 0000000000000000 R14: ffff8b1d903c2090 R15: 0000000000007800
> [  340.420408] FS:  0000000000000000(0000) GS:ffff8b1d9fd80000(0000) knlGS:0000000000000000
> [  340.420408] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  340.420408] CR2: 0000000000000102 CR3: 00000002152ce000 CR4: 00000000000006e0
> [  340.420408] Call Trace:
> [  340.420408]  <IRQ>
> [  340.420408]  ? __slab_free+0x8e/0x260
> [  340.420408]  queue_work_on+0x38/0x40
> [  340.420408]  iomap_dio_bio_end_io+0x86/0x120
> [  340.420408]  bio_endio+0x9f/0x120
> [  340.420408]  blk_update_request+0xa8/0x2f0
> [  340.420408]  blk_mq_end_request+0x1e/0x70
> [  340.420408]  virtblk_request_done+0x22/0x30 [virtio_blk]
> [  340.420408]  __blk_mq_complete_request+0x8f/0x140
> [  340.420408]  blk_mq_complete_request+0x2a/0x30
> [  340.420408]  virtblk_done+0x71/0x100 [virtio_blk]
> [  340.420408]  vring_interrupt+0x34/0x80 [virtio_ring]
> [  340.420408]  __handle_irq_event_percpu+0x7e/0x190
> [  340.420408]  handle_irq_event_percpu+0x32/0x80
> [  340.420408]  handle_irq_event+0x3b/0x60
> [  340.420408]  handle_edge_irq+0x72/0x180
> [  340.420408]  handle_irq+0x6f/0x110
> [  340.420408]  do_IRQ+0x46/0xd0
> [  340.420408]  common_interrupt+0x93/0x93
> [  340.420408] RIP: 0010:native_safe_halt+0x6/0x10
> [  340.420408] RSP: 0018:ffffb06bc0cbbe70 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffae
> [  340.420408] RAX: 0000000000000000 RBX: ffff8b1d96222500 RCX: 0000000000000000
> [  340.420408] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> [  340.420408] RBP: ffffb06bc0cbbe70 R08: 000000000679cadf R09: 0000000000000001
> [  340.420408] R10: 000000000001fdfd R11: 0000000000000000 R12: 0000000000000003
> [  340.420408] R13: ffff8b1d96222500 R14: 0000000000000000 R15: 0000000000000000
> [  340.420408]  </IRQ>
> [  340.420408]  default_idle+0x20/0x100
> [  340.420408]  arch_cpu_idle+0xf/0x20
> [  340.420408]  default_idle_call+0x23/0x30
> [  340.420408]  do_idle+0x174/0x1e0
> [  340.420408]  cpu_startup_entry+0x73/0x80
> [  340.420408]  start_secondary+0x156/0x190
> [  340.420408]  secondary_startup_64+0x9f/0x9f
> [  340.420408] Code: 89 e5 41 57 41 56 41 55 41 54 49 89 f5 53 49 89 d6 41 89 fc 48 83 ec 18 89 7d d4 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 0a 03 00 00 <41> f6 85 02 01 00 00 01 0f 85 0c 03 00 00 48 b8 eb 83 b5 80 46
> [  340.420408] RIP: __queue_work+0x32/0x420 RSP: ffff8b1d9fd83d18
> [  340.420408] CR2: 0000000000000102
> [  340.420408] ---[ end trace 4ae4f080188b0b36 ]---
> [  340.420408] Kernel panic - not syncing: Fatal exception in interrupt
> [  340.420408] Shutting down cpus with NMI
> [  340.420408] Kernel Offset: 0x6000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> [  340.420408] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 


-- 
chandan




[Index of Archives]     [Linux RAID]     [Linux SCSI]     [Linux ATA RAID]     [IDE]     [Linux Wireless]     [Linux Kernel]     [ATH6KL]     [Linux Bluetooth]     [Linux Netdev]     [Kernel Newbies]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Device Mapper]

  Powered by Linux