Hi all,
Recently I noticed multiple crashes triggered by xfs/104 on ppc64 hosts
in my upstream 4.13 kernel testings. The block layer is involved in the
call trace so I add linux-block to cc list too. I append the full
console log to the end of this mail.
Now I can reproduce the crash on x86_64 hosts too by running xfs/104
many times (usually within 100 iterations). A git-bisect run (I ran it
for 500 iterations before calling it good in bisect run to be safe)
pointed the first bad commit to commit acdda3aae146 ("xfs: use
iomap_dio_rw").
I confirmed the bisect result by checking out a new branch with commit
acdda3aae146 as HEAD, xfs/104 would crash kernel within 100 iterations,
then reverting HEAD, xfs/104 passed 1500 iterations.
On one of my test vms, the crash happened as
[ 340.419429] BUG: unable to handle kernel NULL pointer dereference at 0000000000000102
[ 340.420408] IP: __queue_work+0x32/0x420
and that IP points to
(gdb) l *(__queue_work+0x32)
0x9cf32 is in __queue_work (kernel/workqueue.c:1383).
1378 WARN_ON_ONCE(!irqs_disabled());
1379
1380 debug_work_activate(work);
1381
1382 /* if draining, only works from the same workqueue are allowed */
1383 if (unlikely(wq->flags & __WQ_DRAINING) &&
1384 WARN_ON_ONCE(!is_chained_work(wq)))
1385 return;
1386 retry:
1387 if (req_cpu == WORK_CPU_UNBOUND)
So looks like "wq" is null. The test vm is a kvm guest running 4.13
kernel with 4 vcpus and 8G memory.
If more information is needed please let me know.
Thanks,
Eryu
P.S. console log when crashing
[ 339.746983] run fstests xfs/104 at 2017-09-13 17:38:26
[ 340.027352] XFS (vda6): Unmounting Filesystem
[ 340.207107] XFS (vda6): Mounting V5 Filesystem
[ 340.217553] XFS (vda6): Ending clean mount
[ 340.419429] BUG: unable to handle kernel NULL pointer dereference at 0000000000000102
[ 340.420408] IP: __queue_work+0x32/0x420
[ 340.420408] PGD 215373067
[ 340.420408] P4D 215373067
[ 340.420408] PUD 21210d067
[ 340.420408] PMD 0
[ 340.420408]
[ 340.420408] Oops: 0000 [#1] SMP
[ 340.420408] Modules linked in: xfs ip6t_rpfilter ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack libcrc32c ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter btrfs xor raid6_pq ppdev i2c_piix4 parport_pc i2c_core parport virtio_balloon pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi virtio_net virtio_blk ata_piix libata virtio_pci virtio_ring serio_raw virtio floppy
[ 340.420408] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.13.0 #64
[ 340.420408] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[ 340.420408] task: ffff8b1d96222500 task.stack: ffffb06bc0cb8000
[ 340.420408] RIP: 0010:__queue_work+0x32/0x420
[ 340.420408] RSP: 0018:ffff8b1d9fd83d18 EFLAGS: 00010046
[ 340.420408] RAX: 0000000000000096 RBX: 0000000000000002 RCX: ffff8b1d9489e6d8
[ 340.420408] RDX: ffff8b1d903c2090 RSI: 0000000000000000 RDI: 0000000000002000
[ 340.420408] RBP: ffff8b1d9fd83d58 R08: 0000000000000400 R09: 0000000000000009
[ 340.420408] R10: ffff8b1d9532b400 R11: 0000000000000000 R12: 0000000000002000
[ 340.420408] R13: 0000000000000000 R14: ffff8b1d903c2090 R15: 0000000000007800
[ 340.420408] FS: 0000000000000000(0000) GS:ffff8b1d9fd80000(0000) knlGS:0000000000000000
[ 340.420408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 340.420408] CR2: 0000000000000102 CR3: 00000002152ce000 CR4: 00000000000006e0
[ 340.420408] Call Trace:
[ 340.420408] <IRQ>
[ 340.420408] ? __slab_free+0x8e/0x260
[ 340.420408] queue_work_on+0x38/0x40
[ 340.420408] iomap_dio_bio_end_io+0x86/0x120
[ 340.420408] bio_endio+0x9f/0x120
[ 340.420408] blk_update_request+0xa8/0x2f0
[ 340.420408] blk_mq_end_request+0x1e/0x70
[ 340.420408] virtblk_request_done+0x22/0x30 [virtio_blk]
[ 340.420408] __blk_mq_complete_request+0x8f/0x140
[ 340.420408] blk_mq_complete_request+0x2a/0x30
[ 340.420408] virtblk_done+0x71/0x100 [virtio_blk]
[ 340.420408] vring_interrupt+0x34/0x80 [virtio_ring]
[ 340.420408] __handle_irq_event_percpu+0x7e/0x190
[ 340.420408] handle_irq_event_percpu+0x32/0x80
[ 340.420408] handle_irq_event+0x3b/0x60
[ 340.420408] handle_edge_irq+0x72/0x180
[ 340.420408] handle_irq+0x6f/0x110
[ 340.420408] do_IRQ+0x46/0xd0
[ 340.420408] common_interrupt+0x93/0x93
[ 340.420408] RIP: 0010:native_safe_halt+0x6/0x10
[ 340.420408] RSP: 0018:ffffb06bc0cbbe70 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffae
[ 340.420408] RAX: 0000000000000000 RBX: ffff8b1d96222500 RCX: 0000000000000000
[ 340.420408] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 340.420408] RBP: ffffb06bc0cbbe70 R08: 000000000679cadf R09: 0000000000000001
[ 340.420408] R10: 000000000001fdfd R11: 0000000000000000 R12: 0000000000000003
[ 340.420408] R13: ffff8b1d96222500 R14: 0000000000000000 R15: 0000000000000000
[ 340.420408] </IRQ>
[ 340.420408] default_idle+0x20/0x100
[ 340.420408] arch_cpu_idle+0xf/0x20
[ 340.420408] default_idle_call+0x23/0x30
[ 340.420408] do_idle+0x174/0x1e0
[ 340.420408] cpu_startup_entry+0x73/0x80
[ 340.420408] start_secondary+0x156/0x190
[ 340.420408] secondary_startup_64+0x9f/0x9f
[ 340.420408] Code: 89 e5 41 57 41 56 41 55 41 54 49 89 f5 53 49 89 d6 41 89 fc 48 83 ec 18 89 7d d4 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 0a 03 00 00 <41> f6 85 02 01 00 00 01 0f 85 0c 03 00 00 48 b8 eb 83 b5 80 46
[ 340.420408] RIP: __queue_work+0x32/0x420 RSP: ffff8b1d9fd83d18
[ 340.420408] CR2: 0000000000000102
[ 340.420408] ---[ end trace 4ae4f080188b0b36 ]---
[ 340.420408] Kernel panic - not syncing: Fatal exception in interrupt
[ 340.420408] Shutting down cpus with NMI
[ 340.420408] Kernel Offset: 0x6000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 340.420408] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
