BUG at IP: blk_mq_get_request+0x23e/0x390 on 4.16.0-rc7

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello
I got this kernel BUG on 4.16.0-rc7, here is the reproducer and log, let me know if you need more info, thanks.

Reproducer:
1. setup target
#nvmetcli restore /etc/rdma.json
2. connect target on host
#nvme connect-all -t rdma -a $IP -s 4420during my NVMeoF RDMA testing
3. do fio background on host
#fio -filename=/dev/nvme0n1 -iodepth=1 -thread -rw=randwrite -ioengine=psync -bssplit=5k/10:9k/10:13k/10:17k/10:21k/10:25k/10:29k/10:33k/10:37k/10:41k/10 -bs_unaligned -runtime=180 -size=-group_reporting -name=mytest -numjobs=60 &
4. offline cpu on host
#echo 0 > /sys/devices/system/cpu/cpu1/online
#echo 0 > /sys/devices/system/cpu/cpu2/online
#echo 0 > /sys/devices/system/cpu/cpu3/online
5. clear target
#nvmetcli clear
6. restore target
#nvmetcli restore /etc/rdma.json
7. check console log on host


[  167.054583] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[  167.108410] nvme nvme0: creating 40 I/O queues.
[  167.421694] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  256.496376] smpboot: CPU 1 is now offline
[  256.525102] IRQ 37: no longer affine to CPU2
[  256.529872] IRQ 54: no longer affine to CPU2
[  256.534637] IRQ 70: no longer affine to CPU2
[  256.539405] IRQ 98: no longer affine to CPU2
[  256.544175] IRQ 140: no longer affine to CPU2
[  256.549036] IRQ 141: no longer affine to CPU2
[  256.553905] IRQ 166: no longer affine to CPU2
[  256.561042] smpboot: CPU 2 is now offline
[  256.796920] smpboot: CPU 3 is now offline
[  258.649993] print_req_error: operation not supported error, dev nvme0n1, sector 60151856
[  258.650031] print_req_error: operation not supported error, dev nvme0n1, sector 512220944
[  258.650040] print_req_error: operation not supported error, dev nvme0n1, sector 221050984
[  258.650047] print_req_error: operation not supported error, dev nvme0n1, sector 160854616
[  258.650058] print_req_error: operation not supported error, dev nvme0n1, sector 471080288
[  258.650083] print_req_error: operation not supported error, dev nvme0n1, sector 242366208
[  258.650093] print_req_error: operation not supported error, dev nvme0n1, sector 363042304
[  258.650100] print_req_error: operation not supported error, dev nvme0n1, sector 55054168
[  258.650106] print_req_error: operation not supported error, dev nvme0n1, sector 261203184
[  258.650110] print_req_error: operation not supported error, dev nvme0n1, sector 318931552
[  259.401504] nvme nvme0: Reconnecting in 10 seconds...
[  259.401508] Buffer I/O error on dev nvme0n1, logical block 218, lost async page write
[  259.415933] Buffer I/O error on dev nvme0n1, logical block 219, lost async page write
[  259.424709] Buffer I/O error on dev nvme0n1, logical block 267, lost async page write
[  259.433479] Buffer I/O error on dev nvme0n1, logical block 268, lost async page write
[  259.442248] Buffer I/O error on dev nvme0n1, logical block 269, lost async page write
[  259.451017] Buffer I/O error on dev nvme0n1, logical block 270, lost async page write
[  259.459784] Buffer I/O error on dev nvme0n1, logical block 271, lost async page write
[  259.468550] Buffer I/O error on dev nvme0n1, logical block 272, lost async page write
[  259.477319] Buffer I/O error on dev nvme0n1, logical block 273, lost async page write
[  259.486095] Buffer I/O error on dev nvme0n1, logical block 341, lost async page write
[  264.003845] nvme nvme0: Identify namespace failed
[  264.009222] print_req_error: 391720 callbacks suppressed
[  264.009223] print_req_error: I/O error, dev nvme0n1, sector 0
[  264.021610] print_req_error: I/O error, dev nvme0n1, sector 0
[  264.028048] print_req_error: I/O error, dev nvme0n1, sector 0
[  264.034486] print_req_error: I/O error, dev nvme0n1, sector 0
[  264.040922] print_req_error: I/O error, dev nvme0n1, sector 0
[  264.047359] print_req_error: I/O error, dev nvme0n1, sector 0
[  264.053794] Dev nvme0n1: unable to read RDB block 0
[  264.059261] print_req_error: I/O error, dev nvme0n1, sector 0
[  264.065699] print_req_error: I/O error, dev nvme0n1, sector 0
[  264.072134]  nvme0n1: unable to read partition table
[  264.082672] print_req_error: I/O error, dev nvme0n1, sector 524287872
[  264.090339] print_req_error: I/O error, dev nvme0n1, sector 524287872
[  269.481193] nvme nvme0: creating 37 I/O queues.
[  269.787024] BUG: unable to handle kernel paging request at 0000473023d3b6c8
[  269.795246] IP: blk_mq_get_request+0x23e/0x390
[  269.800599] PGD 0 P4D 0 
[  269.803810] Oops: 0002 [#1] SMP PTI
[  269.808089] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc ib_isert iscsir
[  269.890870]  syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core ahci libahci tg3 libata crc32c_intel i2c_core devlink dm_mirror dm_region_hash dm_log dm_mod
[  269.908864] CPU: 36 PID: 680 Comm: kworker/u369:8 Not tainted 4.16.0-rc7 #3
[  269.917207] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016
[  269.926155] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
[  269.934239] RIP: 0010:blk_mq_get_request+0x23e/0x390
[  269.940392] RSP: 0018:ffffb237087cbca8 EFLAGS: 00010246
[  269.946841] RAX: 0000473023d3b680 RBX: ffff8b06546e0000 RCX: 000000000000001f
[  269.955443] RDX: 0000000000000000 RSI: ffffffdbc0ce8100 RDI: ffff8b0653431000
[  269.964053] RBP: ffffb237087cbce8 R08: ffffffffffffffff R09: 0000000000000002
[  269.972674] R10: ffff8af67eaa7160 R11: ffffd62c40186c00 R12: 0000000000000023
[  269.981285] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  269.989891] FS:  0000000000000000(0000) GS:ffff8af67ea80000(0000) knlGS:0000000000000000
[  269.999577] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  270.006654] CR2: 0000473023d3b6c8 CR3: 00000015ed40a001 CR4: 00000000001606e0
[  270.015300] Call Trace:
[  270.018716]  blk_mq_alloc_request_hctx+0xf2/0x140
[  270.024668]  nvme_alloc_request+0x36/0x60 [nvme_core]
[  270.031016]  __nvme_submit_sync_cmd+0x2b/0xd0 [nvme_core]
[  270.037762]  nvmf_connect_io_queue+0x10e/0x170 [nvme_fabrics]
[  270.044898]  nvme_rdma_start_queue+0x21/0x80 [nvme_rdma]
[  270.051566]  nvme_rdma_configure_io_queues+0x196/0x280 [nvme_rdma]
[  270.059199]  nvme_rdma_reconnect_ctrl_work+0x39/0xd0 [nvme_rdma]
[  270.066637]  process_one_work+0x158/0x360
[  270.071846]  worker_thread+0x47/0x3e0
[  270.076672]  kthread+0xf8/0x130
[  270.080918]  ? max_active_store+0x80/0x80
[  270.086142]  ? kthread_bind+0x10/0x10
[  270.090987]  ret_from_fork+0x35/0x40
[  270.095739] Code: 89 83 40 01 00 00 45 84 e4 48 c7 83 48 01 00 00 00 00 00 00 ba 01 00 00 00 48 8b 45 10 74 0c 31 d2 41 f7 c4 00 08 06 00 0f 95 c2 <48> 83 44 d0 48 01 41 81 e4 00 00 06  
[  270.118418] RIP: blk_mq_get_request+0x23e/0x390 RSP: ffffb237087cbca8
[  270.126422] CR2: 0000473023d3b6c8
[  270.130994] ---[ end trace 222e693b7ee07afa ]---
[  270.141098] Kernel panic - not syncing: Fatal exception
[  270.147812] Kernel Offset: 0x22800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  270.164696] ---[ end Kernel panic - not syncing: Fatal exception
[  270.172257] WARNING: CPU: 36 PID: 680 at kernel/sched/core.c:1189 set_task_cpu+0x18c/0x1a0
[  270.182333] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc ib_isert iscsir
[  270.268075]  syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx4_core ahci libahci tg3 libata crc32c_intel i2c_core devlink dm_mirror dm_region_hash dm_log dm_mod
[  270.286750] CPU: 36 PID: 680 Comm: kworker/u369:8 Tainted: G      D          4.16.0-rc7 #3
[  270.296862] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016
[  270.306088] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
[  270.314436] RIP: 0010:set_task_cpu+0x18c/0x1a0
[  270.320253] RSP: 0018:ffff8af67ea83ce0 EFLAGS: 00010046
[  270.326938] RAX: 0000000000000200 RBX: ffff8af65d9445c0 RCX: 0000005555555501
[  270.335764] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8af65d9445c0
[  270.344591] RBP: 0000000000022380 R08: 0000000000000000 R09: 0000000000000010
[  270.353409] R10: 000000005abdf5ea R11: 0000000016684c67 R12: 0000000000000000
[  270.362223] R13: 0000000000000000 R14: 0000000000000046 R15: 0000000000000000
[  270.371030] FS:  0000000000000000(0000) GS:ffff8af67ea80000(0000) knlGS:0000000000000000
[  270.380913] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  270.388166] CR2: 0000473023d3b6c8 CR3: 00000015ed40a001 CR4: 00000000001606e0
[  270.396985] Call Trace:
[  270.400557]  <IRQ>
[  270.403621]  try_to_wake_up+0x167/0x460
[  270.408730]  ? enqueue_task_fair+0x67/0xa00
[  270.414224]  __wake_up_common+0x8f/0x160
[  270.419417]  ep_poll_callback+0xc4/0x2f0
[  270.424609]  __wake_up_common+0x8f/0x160
[  270.429796]  __wake_up_common_lock+0x7a/0xc0
[  270.435368]  irq_work_run_list+0x4c/0x70
[  270.440547]  ? tick_sched_do_timer+0x60/0x60
[  270.446115]  update_process_times+0x3b/0x50
[  270.451579]  tick_sched_handle+0x26/0x60
[  270.456752]  tick_sched_timer+0x34/0x70
[  270.461826]  __hrtimer_run_queues+0xfb/0x270
[  270.467388]  hrtimer_interrupt+0x122/0x270
[  270.472756]  smp_apic_timer_interrupt+0x62/0x130
[  270.478712]  apic_timer_interrupt+0xf/0x20
[  270.484066]  </IRQ>
[  270.487167] RIP: 0010:panic+0x206/0x25c
[  270.492195] RSP: 0018:ffffb237087cba60 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff12
[  270.501406] RAX: 0000000000000034 RBX: 0000000000000000 RCX: 0000000000000006
[  270.510136] RDX: 0000000000000000 RSI: 0000000000000082 RDI: ffff8af67ea968b0
[  270.518863] RBP: ffffb237087cbad0 R08: 0000000000000000 R09: 0000000000000886
[  270.527578] R10: 00000000000003ff R11: 0000000000aaaaaa R12: ffffffffa4654b1a
[  270.536278] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
[  270.544970]  oops_end+0xb0/0xc0
[  270.549179]  no_context+0x1b3/0x430
[  270.553753]  ? account_entity_dequeue+0xa3/0xd0
[  270.559473]  __do_page_fault+0x97/0x4c0
[  270.564396]  do_page_fault+0x32/0x140
[  270.569103]  page_fault+0x25/0x50
[  270.573398] RIP: 0010:blk_mq_get_request+0x23e/0x390
[  270.579516] RSP: 0018:ffffb237087cbca8 EFLAGS: 00010246
[  270.585906] RAX: 0000473023d3b680 RBX: ffff8b06546e0000 RCX: 000000000000001f
[  270.594422] RDX: 0000000000000000 RSI: ffffffdbc0ce8100 RDI: ffff8b0653431000
[  270.602929] RBP: ffffb237087cbce8 R08: ffffffffffffffff R09: 0000000000000002
[  270.611432] R10: ffff8af67eaa7160 R11: ffffd62c40186c00 R12: 0000000000000023
[  270.619927] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  270.628409]  ? blk_mq_get_request+0x212/0x390
[  270.633795]  blk_mq_alloc_request_hctx+0xf2/0x140
[  270.639565]  nvme_alloc_request+0x36/0x60 [nvme_core]
[  270.645721]  __nvme_submit_sync_cmd+0x2b/0xd0 [nvme_core]
[  270.652269]  nvmf_connect_io_queue+0x10e/0x170 [nvme_fabrics]
[  270.659209]  nvme_rdma_start_queue+0x21/0x80 [nvme_rdma]
[  270.665668]  nvme_rdma_configure_io_queues+0x196/0x280 [nvme_rdma]
[  270.673087]  nvme_rdma_reconnect_ctrl_work+0x39/0xd0 [nvme_rdma]
[  270.680314]  process_one_work+0x158/0x360
[  270.685302]  worker_thread+0x47/0x3e0
[  270.689897]  kthread+0xf8/0x130
[  270.693906]  ? max_active_store+0x80/0x80
[  270.698880]  ? kthread_bind+0x10/0x10
[  270.703473]  ret_from_fork+0x35/0x40
[  270.707967] Code: 8b 9c 08 00 00 04 e9 28 ff ff ff 0f 0b 66 90 e9 bf fe ff ff f7 83 88 00 00 00 fd ff ff ff 0f 84 c9 fe ff ff 0f 0b e9 c2 fe ff ff <0f> 0b e9 d1 fe ff ff 0f 1f 00 66 2e  
[  270.730149] ---[ end trace 222e693b7ee07afb ]---

Best Regards,
  Yi Zhang





[Index of Archives]     [Linux RAID]     [Linux SCSI]     [Linux ATA RAID]     [IDE]     [Linux Wireless]     [Linux Kernel]     [ATH6KL]     [Linux Bluetooth]     [Linux Netdev]     [Kernel Newbies]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Device Mapper]

  Powered by Linux