[BUG] xfs/104 triggered NULL pointer dereference in iomap based dio

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi all,

Recently I noticed multiple crashes triggered by xfs/104 on ppc64 hosts
in my upstream 4.13 kernel testings. The block layer is involved in the
call trace so I add linux-block to cc list too. I append the full
console log to the end of this mail.

Now I can reproduce the crash on x86_64 hosts too by running xfs/104
many times (usually within 100 iterations). A git-bisect run (I ran it
for 500 iterations before calling it good in bisect run to be safe)
pointed the first bad commit to commit acdda3aae146 ("xfs: use
iomap_dio_rw").

I confirmed the bisect result by checking out a new branch with commit
acdda3aae146 as HEAD, xfs/104 would crash kernel within 100 iterations,
then reverting HEAD, xfs/104 passed 1500 iterations.

On one of my test vms, the crash happened as

[  340.419429] BUG: unable to handle kernel NULL pointer dereference at 0000000000000102
[  340.420408] IP: __queue_work+0x32/0x420

and that IP points to

(gdb) l *(__queue_work+0x32)
0x9cf32 is in __queue_work (kernel/workqueue.c:1383).
1378            WARN_ON_ONCE(!irqs_disabled());
1379
1380            debug_work_activate(work);
1381
1382            /* if draining, only works from the same workqueue are allowed */
1383            if (unlikely(wq->flags & __WQ_DRAINING) &&
1384                WARN_ON_ONCE(!is_chained_work(wq)))
1385                    return;
1386    retry:
1387            if (req_cpu == WORK_CPU_UNBOUND)

So looks like "wq" is null. The test vm is a kvm guest running 4.13
kernel with 4 vcpus and 8G memory.

If more information is needed please let me know.

Thanks,
Eryu

P.S. console log when crashing

[  339.746983] run fstests xfs/104 at 2017-09-13 17:38:26
[  340.027352] XFS (vda6): Unmounting Filesystem
[  340.207107] XFS (vda6): Mounting V5 Filesystem
[  340.217553] XFS (vda6): Ending clean mount
[  340.419429] BUG: unable to handle kernel NULL pointer dereference at 0000000000000102
[  340.420408] IP: __queue_work+0x32/0x420
[  340.420408] PGD 215373067
[  340.420408] P4D 215373067
[  340.420408] PUD 21210d067
[  340.420408] PMD 0
[  340.420408]
[  340.420408] Oops: 0000 [#1] SMP
[  340.420408] Modules linked in: xfs ip6t_rpfilter ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack libcrc32c ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter btrfs xor raid6_pq ppdev i2c_piix4 parport_pc i2c_core parport virtio_balloon pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi virtio_net virtio_blk ata_piix libata virtio_pci virtio_ring serio_raw virtio floppy
[  340.420408] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.13.0 #64
[  340.420408] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[  340.420408] task: ffff8b1d96222500 task.stack: ffffb06bc0cb8000
[  340.420408] RIP: 0010:__queue_work+0x32/0x420
[  340.420408] RSP: 0018:ffff8b1d9fd83d18 EFLAGS: 00010046
[  340.420408] RAX: 0000000000000096 RBX: 0000000000000002 RCX: ffff8b1d9489e6d8
[  340.420408] RDX: ffff8b1d903c2090 RSI: 0000000000000000 RDI: 0000000000002000
[  340.420408] RBP: ffff8b1d9fd83d58 R08: 0000000000000400 R09: 0000000000000009
[  340.420408] R10: ffff8b1d9532b400 R11: 0000000000000000 R12: 0000000000002000
[  340.420408] R13: 0000000000000000 R14: ffff8b1d903c2090 R15: 0000000000007800
[  340.420408] FS:  0000000000000000(0000) GS:ffff8b1d9fd80000(0000) knlGS:0000000000000000
[  340.420408] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  340.420408] CR2: 0000000000000102 CR3: 00000002152ce000 CR4: 00000000000006e0
[  340.420408] Call Trace:
[  340.420408]  <IRQ>
[  340.420408]  ? __slab_free+0x8e/0x260
[  340.420408]  queue_work_on+0x38/0x40
[  340.420408]  iomap_dio_bio_end_io+0x86/0x120
[  340.420408]  bio_endio+0x9f/0x120
[  340.420408]  blk_update_request+0xa8/0x2f0
[  340.420408]  blk_mq_end_request+0x1e/0x70
[  340.420408]  virtblk_request_done+0x22/0x30 [virtio_blk]
[  340.420408]  __blk_mq_complete_request+0x8f/0x140
[  340.420408]  blk_mq_complete_request+0x2a/0x30
[  340.420408]  virtblk_done+0x71/0x100 [virtio_blk]
[  340.420408]  vring_interrupt+0x34/0x80 [virtio_ring]
[  340.420408]  __handle_irq_event_percpu+0x7e/0x190
[  340.420408]  handle_irq_event_percpu+0x32/0x80
[  340.420408]  handle_irq_event+0x3b/0x60
[  340.420408]  handle_edge_irq+0x72/0x180
[  340.420408]  handle_irq+0x6f/0x110
[  340.420408]  do_IRQ+0x46/0xd0
[  340.420408]  common_interrupt+0x93/0x93
[  340.420408] RIP: 0010:native_safe_halt+0x6/0x10
[  340.420408] RSP: 0018:ffffb06bc0cbbe70 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffae
[  340.420408] RAX: 0000000000000000 RBX: ffff8b1d96222500 RCX: 0000000000000000
[  340.420408] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[  340.420408] RBP: ffffb06bc0cbbe70 R08: 000000000679cadf R09: 0000000000000001
[  340.420408] R10: 000000000001fdfd R11: 0000000000000000 R12: 0000000000000003
[  340.420408] R13: ffff8b1d96222500 R14: 0000000000000000 R15: 0000000000000000
[  340.420408]  </IRQ>
[  340.420408]  default_idle+0x20/0x100
[  340.420408]  arch_cpu_idle+0xf/0x20
[  340.420408]  default_idle_call+0x23/0x30
[  340.420408]  do_idle+0x174/0x1e0
[  340.420408]  cpu_startup_entry+0x73/0x80
[  340.420408]  start_secondary+0x156/0x190
[  340.420408]  secondary_startup_64+0x9f/0x9f
[  340.420408] Code: 89 e5 41 57 41 56 41 55 41 54 49 89 f5 53 49 89 d6 41 89 fc 48 83 ec 18 89 7d d4 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 0a 03 00 00 <41> f6 85 02 01 00 00 01 0f 85 0c 03 00 00 48 b8 eb 83 b5 80 46
[  340.420408] RIP: __queue_work+0x32/0x420 RSP: ffff8b1d9fd83d18
[  340.420408] CR2: 0000000000000102
[  340.420408] ---[ end trace 4ae4f080188b0b36 ]---
[  340.420408] Kernel panic - not syncing: Fatal exception in interrupt
[  340.420408] Shutting down cpus with NMI
[  340.420408] Kernel Offset: 0x6000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  340.420408] ---[ end Kernel panic - not syncing: Fatal exception in interrupt



[Index of Archives]     [Linux RAID]     [Linux SCSI]     [Linux ATA RAID]     [IDE]     [Linux Wireless]     [Linux Kernel]     [ATH6KL]     [Linux Bluetooth]     [Linux Netdev]     [Kernel Newbies]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Device Mapper]

  Powered by Linux