Re: nvmeof rdma regression issue on 4.14.0-rc1 (or maybe mlx4?)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




Lets take this one step at a time, starting with this issue.

First, there is a reason why a simple create_ctrl fails, can we isolate
exactly which call fails? Was something else going on that might have
made the simple create_ctrl fail?

We don't see any "rdma_resolve_addr failed" or "failed to connect queue"
messages but we do see "creating I/O queues" which means that we either
failed at IO tagset allocation or initializing connect_q.

We have a missing error code assignment so can you try the following patch:

Hi Sagi
Sorry for the late response, here is the log with your patch, let me know if you need more test.

[ 236.980239] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[  237.131416] nvme_rdma: nvme_rdma_free_tagset: freeing ADMIN tagset
[  237.173614] nvme nvme0: creating 40 I/O queues.
[  237.688843] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  243.384596] nvme nvme0: rescanning
[  243.527922] nvme nvme0: Reconnecting in 10 seconds...
[  254.010547] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  254.018052] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  254.024482] nvme nvme0: Failed reconnect attempt 1
[  254.029844] nvme nvme0: Reconnecting in 10 seconds...
[  264.238223] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  264.245727] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  264.252159] nvme nvme0: Failed reconnect attempt 2
[  264.257545] nvme nvme0: Reconnecting in 10 seconds...
[  274.477977] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  274.485470] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  274.491905] nvme nvme0: Failed reconnect attempt 3
[  274.497287] nvme nvme0: Reconnecting in 10 seconds...
[  284.737691] nvme nvme0: creating 40 I/O queues.
[ 284.745163] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 100a1db001 not 1011432003)
[  284.754617] ------------[ cut here ]------------
[ 284.759790] WARNING: CPU: 12 PID: 6 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380 [ 284.770010] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma id [ 284.849181] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp libata crc32c_intel i2c_core pps_core devlink d [ 284.870886] CPU: 12 PID: 6 Comm: kworker/u368:0 Not tainted 4.14.0-rc4.10.02.sagi+ #30 [ 284.879746] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016
[  284.888133] Workqueue: ib_addr process_one_req [ib_core]
[  284.894074] task: ffff88018cb50000 task.stack: ffffc9000009c000
[  284.900699] RIP: 0010:__domain_mapping+0x367/0x380
[  284.906051] RSP: 0018:ffffc9000009fa98 EFLAGS: 00010202
[ 284.911899] RAX: 0000000000000004 RBX: 0000001011432003 RCX: 0000000000000000 [ 284.920643] RDX: 0000000000000000 RSI: ffff88103e78e058 RDI: ffff88103e78e058 [ 284.929371] RBP: ffffc9000009faf8 R08: 0000000000000000 R09: 0000000000000000 [ 284.938088] R10: 00000000000002f7 R11: 0000000001011432 R12: ffff88201ab634e8 [ 284.946814] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0e9d [ 284.955533] FS: 0000000000000000(0000) GS:ffff88103e780000(0000) knlGS:0000000000000000
[  284.965318] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 284.972491] CR2: 00007f9348b7b000 CR3: 000000202f7e7005 CR4: 00000000001606e0
[  284.981227] Call Trace:
[  284.984712]  __intel_map_single+0xeb/0x180
[  284.990059]  intel_alloc_coherent+0xb5/0x130
[  284.995584]  mlx4_buf_alloc+0xe5/0x1c0 [mlx4_core]
[  285.001690]  mlx4_ib_alloc_cq_buf.isra.9+0x38/0xd0 [mlx4_ib]
[  285.008754]  mlx4_ib_create_cq+0x223/0x450 [mlx4_ib]
[  285.015027]  ib_alloc_cq+0x49/0x170 [ib_core]
[  285.020634]  nvme_rdma_cm_handler+0x3a2/0x7ab [nvme_rdma]
[  285.027416]  ? cma_acquire_dev+0x1e3/0x3b0 [rdma_cm]
[  285.033689]  addr_handler+0xa4/0x1c0 [rdma_cm]
[  285.039417]  process_one_req+0x8d/0x120 [ib_core]
[  285.045408]  process_one_work+0x149/0x360
[  285.050620]  worker_thread+0x4d/0x3c0
[  285.055432]  kthread+0x109/0x140
[  285.059780]  ? rescuer_thread+0x380/0x380
[  285.064987]  ? kthread_park+0x60/0x60
[  285.069805]  ret_from_fork+0x25/0x30
[ 285.074511] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b
[  285.097088] ---[ end trace 501470fecce57311 ]---
[  285.103299] DMAR: DRHD: handling fault status reg 2
[ 285.103761] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 100a1db001 not 100526b002)
[  285.103766] ------------[ cut here ]------------
[ 285.103768] WARNING: CPU: 12 PID: 6 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380 [ 285.103768] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma id [ 285.103785] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp libata crc32c_intel i2c_core pps_core devlink d [ 285.103791] CPU: 12 PID: 6 Comm: kworker/u368:0 Tainted: G W 4.14.0-rc4.10.02.sagi+ #30 [ 285.103792] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016
[  285.103797] Workqueue: ib_addr process_one_req [ib_core]
[  285.103798] task: ffff88018cb50000 task.stack: ffffc9000009c000
[  285.103799] RIP: 0010:__domain_mapping+0x367/0x380
[  285.103800] RSP: 0018:ffffc9000009fc48 EFLAGS: 00010206
[ 285.103800] RAX: 0000000000000003 RBX: 000000100526b002 RCX: 0000000000000000 [ 285.103801] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000282 [ 285.103801] RBP: ffffc9000009fca8 R08: 0000000000000050 R09: 0000000000000000 [ 285.103802] R10: 0000000000000860 R11: 000000000100526b R12: ffff88201ab634e8 [ 285.103803] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0e9d [ 285.103803] FS: 0000000000000000(0000) GS:ffff88103e780000(0000) knlGS:0000000000000000
[  285.103804] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 285.103805] CR2: 00007f9348b7b000 CR3: 000000202f7e7005 CR4: 00000000001606e0
[  285.103805] Call Trace:
[  285.103808]  __intel_map_single+0xeb/0x180
[  285.103809]  intel_map_page+0x39/0x40
[  285.103811]  nvme_rdma_cm_handler+0x47b/0x7ab [nvme_rdma]
[  285.103813]  addr_handler+0xa4/0x1c0 [rdma_cm]
[  285.103817]  process_one_req+0x8d/0x120 [ib_core]
[  285.103818]  process_one_work+0x149/0x360
[  285.103819]  worker_thread+0x4d/0x3c0
[  285.103821]  kthread+0x109/0x140
[  285.103822]  ? rescuer_thread+0x380/0x380
[  285.103823]  ? kthread_park+0x60/0x60
[  285.103825]  ret_from_fork+0x25/0x30
[ 285.103826] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b
[  285.103839] ---[ end trace 501470fecce57312 ]---
[ 285.190580] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 100a1db001 not 100d51b001)
[  285.190586] ------------[ cut here ]------------
[ 285.190590] WARNING: CPU: 30 PID: 757 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380 [ 285.190590] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma id [ 285.190609] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp libata crc32c_intel i2c_core pps_core devlink d [ 285.190616] CPU: 30 PID: 757 Comm: kworker/u369:7 Tainted: G W 4.14.0-rc4.10.02.sagi+ #30 [ 285.190617] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016
[  285.190619] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
[  285.190620] task: ffff881032bd8000 task.stack: ffffc9000851c000
[  285.190622] RIP: 0010:__domain_mapping+0x367/0x380
[  285.190622] RSP: 0018:ffffc9000851fc60 EFLAGS: 00010202
[ 285.190623] RAX: 0000000000000002 RBX: 000000100d51b001 RCX: 0000000000000000 [ 285.190624] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000292 [ 285.190624] RBP: ffffc9000851fcc0 R08: 0000000000000050 R09: 0000000000000000 [ 285.190625] R10: 0000000000000881 R11: 000000000100d51b R12: ffff88201ab634e8 [ 285.190625] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0e9d [ 285.190626] FS: 0000000000000000(0000) GS:ffff88103e9c0000(0000) knlGS:0000000000000000
[  285.190626] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 285.190627] CR2: 00007f4a9e61fb70 CR3: 0000001011bdd001 CR4: 00000000001606e0
[  285.190628] Call Trace:
[  285.190630]  __intel_map_single+0xeb/0x180
[  285.190632]  intel_map_page+0x39/0x40
[  285.190636]  mlx4_ib_alloc_mr+0x141/0x220 [mlx4_ib]
[  285.190642]  ib_alloc_mr+0x26/0x50 [ib_core]
[  285.190644]  nvme_rdma_reinit_request+0x3a/0x70 [nvme_rdma]
[  285.190645]  ? nvme_rdma_free_ctrl+0xb0/0xb0 [nvme_rdma]
[  285.190647]  blk_mq_reinit_tagset+0x5c/0x90
[  285.190649]  nvme_rdma_configure_io_queues+0x204/0x280 [nvme_rdma]
[  285.190650]  nvme_rdma_reconnect_ctrl_work+0x5b/0xe0 [nvme_rdma]
[  285.190652]  process_one_work+0x149/0x360
[  285.190653]  worker_thread+0x4d/0x3c0
[  285.190655]  kthread+0x109/0x140
[  285.190656]  ? rescuer_thread+0x380/0x380
[  285.190657]  ? kthread_park+0x60/0x60
[  285.190660]  ret_from_fork+0x25/0x30
[ 285.190660] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b
[  285.190673] ---[ end trace 501470fecce57313 ]---
[ 285.819126] DMAR: [DMA Write] Request device [04:00.0] fault addr e0e9d000 [fault reason 05] PTE Write access is not set
[  286.035086] nvme nvme0: Successfully reconnected
[ 286.035143] nvme nvme0: MEMREG for CQE 0xffff88100a1e6f80 failed with status memory management operation error (6)
[  286.187508] nvme nvme0: Reconnecting in 10 seconds...
[  296.270043] nvme nvme0: creating 40 I/O queues.
[ 296.336505] DMAR: ERROR: DMA PTE for vPFN 0xe0e9d already set (to 1012082002 not 1012082002)
[  296.346704] ------------[ cut here ]------------
[ 296.352618] WARNING: CPU: 32 PID: 815 at drivers/iommu/intel-iommu.c:2305 __domain_mapping+0x367/0x380 [ 296.363755] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma id [ 296.447983] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp libata crc32c_intel i2c_core pps_core devlink d [ 296.471170] CPU: 32 PID: 815 Comm: kworker/u368:1 Tainted: G W 4.14.0-rc4.10.02.sagi+ #30 [ 296.482344] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016
[  296.491493] Workqueue: ib_addr process_one_req [ib_core]
[  296.498221] task: ffff8810373d2e80 task.stack: ffffc9000849c000
[  296.505613] RIP: 0010:__domain_mapping+0x367/0x380
[  296.511751] RSP: 0018:ffffc9000849fc48 EFLAGS: 00010202
[ 296.518380] RAX: 0000000000000001 RBX: 0000001012082002 RCX: 0000000000000000 [ 296.527146] RDX: 0000000000000000 RSI: ffff88103ea0e058 RDI: ffff88103ea0e058 [ 296.535918] RBP: ffffc9000849fca8 R08: 0000000000000000 R09: 0000000000000000 [ 296.544691] R10: 00000000000003ff R11: 0000000001012082 R12: ffff88201ab634e8 [ 296.553462] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000000e0e9d [ 296.562240] FS: 0000000000000000(0000) GS:ffff88103ea00000(0000) knlGS:0000000000000000
[  296.572078] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 296.579292] CR2: 00007fdc7ae90000 CR3: 0000001034df0002 CR4: 00000000001606e0
[  296.588070] Call Trace:
[  296.591620]  __intel_map_single+0xeb/0x180
[  296.597012]  intel_map_page+0x39/0x40
[  296.601912]  nvme_rdma_cm_handler+0x47b/0x7ab [nvme_rdma]
[  296.608760]  addr_handler+0xa4/0x1c0 [rdma_cm]
[  296.614540]  process_one_req+0x8d/0x120 [ib_core]
[  296.620602]  process_one_work+0x149/0x360
[  296.625883]  worker_thread+0x4d/0x3c0
[  296.630779]  kthread+0x109/0x140
[  296.635177]  ? rescuer_thread+0x380/0x380
[  296.640453]  ? kthread_park+0x60/0x60
[  296.645341]  ret_from_fork+0x25/0x30
[ 296.650132] Code: 10 ab 81 4c 89 5d a0 4c 89 4d a8 e8 27 ac c0 ff 8b 05 fe 39 87 00 4c 8b 4d a8 4c 8b 5d a0 85 c0 74 09 83 e8 01 89 05 e9 39 87 00 <0f> ff e9 b8 fd ff ff e8 ed 8e b
[  296.672876] ---[ end trace 501470fecce57314 ]---
[  297.231475] nvme nvme0: Successfully reconnected
[ 297.231523] nvme nvme0: MEMREG for CQE 0xffff88100a1e6f80 failed with status memory management operation error (6)
[  297.378026] nvme nvme0: Reconnecting in 10 seconds...
[  307.765849] ------------[ cut here ]------------
[  307.771793] kernel BUG at drivers/iommu/iova.c:836!
[  307.778007] invalid opcode: 0000 [#1] SMP
[ 307.783257] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma id [ 307.867936] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp libata crc32c_intel i2c_core pps_core devlink d [ 307.891229] CPU: 30 PID: 757 Comm: kworker/u369:7 Tainted: G W 4.14.0-rc4.10.02.sagi+ #30 [ 307.902427] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016
[  307.911579] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
[  307.919847] task: ffff881032bd8000 task.stack: ffffc9000851c000
[  307.927232] RIP: 0010:iova_magazine_free_pfns+0x76/0x80
[  307.933827] RSP: 0018:ffffc9000851fc58 EFLAGS: 00010046
[ 307.940415] RAX: 0000000000000000 RBX: 0000000000000017 RCX: 0000000100400038 [ 307.949132] RDX: 00000000000e0e9c RSI: 00000000000e0e9d RDI: ffff8810372c4e58 [ 307.957857] RBP: ffffc9000851fc78 R08: ffff88202fd41a40 R09: 0000000100400038 [ 307.966586] R10: 000000002fd41c01 R11: ffff88202fd41a40 R12: ffff880ffda45000 [ 307.975308] R13: ffff8810372c4e58 R14: 0000000000000046 R15: ffff880ffda45000 [ 307.984032] FS: 0000000000000000(0000) GS:ffff88103e9c0000(0000) knlGS:0000000000000000
[  307.993834] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 308.001012] CR2: 00007f4a9e61fb70 CR3: 0000001011bdd001 CR4: 00000000001606e0
[  308.009749] Call Trace:
[  308.013243]  free_iova_fast+0xc9/0x1d0
[  308.018198]  fq_ring_free+0x78/0xc0
[  308.022845]  queue_iova+0x111/0x130
[  308.027478]  intel_unmap+0xa5/0x130
[  308.032118]  intel_unmap_page+0xe/0x10
[  308.037044]  nvme_rdma_free_ring.constprop.61+0x4a/0x90 [nvme_rdma]
[  308.044789]  nvme_rdma_destroy_queue_ib+0x3a/0x50 [nvme_rdma]
[  308.051951]  nvme_rdma_free_queue+0x1a/0x30 [nvme_rdma]
[  308.058529]  nvme_rdma_destroy_io_queues+0x80/0xb0 [nvme_rdma]
[  308.065786]  nvme_rdma_reconnect_ctrl_work+0x30/0xe0 [nvme_rdma]
[  308.073240]  process_one_work+0x149/0x360
[  308.078470]  worker_thread+0x4d/0x3c0
[  308.083299]  kthread+0x109/0x140
[  308.087652]  ? rescuer_thread+0x380/0x380
[  308.092878]  ? kthread_park+0x60/0x60
[  308.097699]  ret_from_fork+0x25/0x30
[ 308.102407] Code: ef e8 ef fe ff ff 48 63 d3 49 3b 14 24 72 d7 4c 89 f6 4c 89 ef e8 3b 3d 2b 00 49 c7 04 24 00 00 00 00 5b 41 5c 41 5d 41 5e 5d c3 <0f> 0b 0f 1f 84 00 00 00 00 00 0
[  308.124980] RIP: iova_magazine_free_pfns+0x76/0x80 RSP: ffffc9000851fc58
[  308.133265] ---[ end trace 501470fecce57315 ]---
[  308.142811] Kernel panic - not syncing: Fatal exception
[  308.149425] Kernel Offset: disabled
[  308.158679] ---[ end Kernel panic - not syncing: Fatal exception
[  308.166135] ------------[ cut here ]------------
[ 308.172010] WARNING: CPU: 30 PID: 757 at kernel/sched/core.c:1179 set_task_cpu+0x191/0x1a0 [ 308.181959] Modules linked in: nvme_rdma nvme_fabrics nvme_core sch_mqprio ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge 8021q garp mrp stp llc rpcrdma id [ 308.266001] mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3 ahci mlx4_core libahci ptp libata crc32c_intel i2c_core pps_core devlink d [ 308.289112] CPU: 30 PID: 757 Comm: kworker/u369:7 Tainted: G D W 4.14.0-rc4.10.02.sagi+ #30 [ 308.300236] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.6.2 01/08/2016
[  308.309330] Workqueue: nvme-wq nvme_rdma_reconnect_ctrl_work [nvme_rdma]
[  308.317552] task: ffff881032bd8000 task.stack: ffffc9000851c000
[  308.324905] RIP: 0010:set_task_cpu+0x191/0x1a0
[  308.330621] RSP: 0018:ffff88103e9c3c38 EFLAGS: 00010046
[ 308.337191] RAX: 0000000000000200 RBX: ffff881034cc5d00 RCX: 0000000000000001 [ 308.345896] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff881034cc5d00 [ 308.354604] RBP: ffff88103e9c3c58 R08: 0000000000000000 R09: 0000000000000000 [ 308.363314] R10: 00000000000003ff R11: 0000000000000001 R12: ffff881034cc68ac [ 308.372018] R13: 0000000000000000 R14: 0000000000000000 R15: 000000000001b900 [ 308.380720] FS: 0000000000000000(0000) GS:ffff88103e9c0000(0000) knlGS:0000000000000000
[  308.390495] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 308.397632] CR2: 00007f4a9e61fb70 CR3: 0000001011bdd001 CR4: 00000000001606e0
[  308.406347] Call Trace:
[  308.409809]  <IRQ>
[  308.412789]  try_to_wake_up+0x166/0x470
[  308.417816]  default_wake_function+0x12/0x20
[  308.423325]  __wake_up_common+0x8a/0x160
[  308.428438]  __wake_up_locked+0x16/0x20
[  308.433450]  ep_poll_callback+0xd0/0x300
[  308.438562]  __wake_up_common+0x8a/0x160
[  308.443664]  __wake_up_common_lock+0x7e/0xc0
[  308.449150]  __wake_up+0x13/0x20
[  308.453455]  wake_up_klogd_work_func+0x40/0x60
[  308.459124]  irq_work_run_list+0x4d/0x70
[  308.464217]  ? tick_sched_do_timer+0x70/0x70
[  308.469701]  irq_work_tick+0x40/0x50
[  308.474409]  update_process_times+0x42/0x60
[  308.479800]  tick_sched_handle+0x2d/0x60
[  308.484881]  tick_sched_timer+0x39/0x70
[  308.489852]  __hrtimer_run_queues+0xe5/0x230
[  308.495291]  hrtimer_interrupt+0xa8/0x1a0
[  308.500466]  smp_apic_timer_interrupt+0x5f/0x130
[  308.506320]  apic_timer_interrupt+0x9d/0xb0
[  308.511661]  </IRQ>
[  308.514658] RIP: 0010:panic+0x1fd/0x245
[ 308.519603] RSP: 0018:ffffc9000851f9a8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10 [ 308.528740] RAX: 0000000000000034 RBX: 0000000000000000 RCX: 0000000000000006 [ 308.537352] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff88103e9ce050 [ 308.545943] RBP: ffffc9000851fa18 R08: 0000000000000000 R09: 00000000000008f9 [ 308.554511] R10: 00000000000003ff R11: 0000000000000001 R12: ffffffff81a3f340 [ 308.563067] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000004
[  308.571619]  ? panic+0x1f6/0x245
[  308.575773]  oops_end+0xb8/0xd0
[  308.579822]  die+0x42/0x50
[  308.583348]  do_trap+0x16a/0x180
[  308.587442]  do_error_trap+0x89/0x110
[  308.592007]  ? iova_magazine_free_pfns+0x76/0x80
[  308.597624]  ? rem_res_range+0x1ae/0x200 [mlx4_core]
[  308.603625]  do_invalid_op+0x20/0x30
[  308.608063]  invalid_op+0x18/0x20
[  308.612198] RIP: 0010:iova_magazine_free_pfns+0x76/0x80
[  308.618473] RSP: 0018:ffffc9000851fc58 EFLAGS: 00010046
[ 308.624735] RAX: 0000000000000000 RBX: 0000000000000017 RCX: 0000000100400038 [ 308.633146] RDX: 00000000000e0e9c RSI: 00000000000e0e9d RDI: ffff8810372c4e58 [ 308.641555] RBP: ffffc9000851fc78 R08: ffff88202fd41a40 R09: 0000000100400038 [ 308.649969] R10: 000000002fd41c01 R11: ffff88202fd41a40 R12: ffff880ffda45000 [ 308.658387] R13: ffff8810372c4e58 R14: 0000000000000046 R15: ffff880ffda45000
[  308.666799]  ? iova_magazine_free_pfns+0x3e/0x80
[  308.672399]  free_iova_fast+0xc9/0x1d0
[  308.677046]  fq_ring_free+0x78/0xc0
[  308.681370]  queue_iova+0x111/0x130
[  308.685691]  intel_unmap+0xa5/0x130
[  308.690011]  intel_unmap_page+0xe/0x10
[  308.694626]  nvme_rdma_free_ring.constprop.61+0x4a/0x90 [nvme_rdma]
[  308.702056]  nvme_rdma_destroy_queue_ib+0x3a/0x50 [nvme_rdma]
[  308.708909]  nvme_rdma_free_queue+0x1a/0x30 [nvme_rdma]
[  308.715182]  nvme_rdma_destroy_io_queues+0x80/0xb0 [nvme_rdma]
[  308.722148]  nvme_rdma_reconnect_ctrl_work+0x30/0xe0 [nvme_rdma]
[  308.729311]  process_one_work+0x149/0x360
[  308.734242]  worker_thread+0x4d/0x3c0
[  308.738783]  kthread+0x109/0x140
[  308.742840]  ? rescuer_thread+0x380/0x380
[  308.747774]  ? kthread_park+0x60/0x60
[  308.752309]  ret_from_fork+0x25/0x30
[ 308.756747] Code: ff 80 8b ac 08 00 00 04 e9 23 ff ff ff 0f ff e9 bf fe ff ff f7 83 84 00 00 00 fd ff ff ff 0f 84 c9 fe ff ff 0f ff e9 c2 fe ff ff <0f> ff e9 d1 fe ff ff 0f 1f 84 0
[  308.778800] ---[ end trace 501470fecce57316 ]---
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html



[Index of Archives]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Photo]     [Yosemite News]     [Yosemite Photos]     [Linux Kernel]     [Linux SCSI]     [XFree86]
  Powered by Linux