question about xfs soft lockup stuck

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Dear Brian and xfs experts,

Recently we have encountered an issue related to xfs. The log reports xfs soft lockup stuck which is similar to this bug:
http://bugs.centos.org/view.php?id=6287

I wan to ask a question. Is this issue already been fixed in kernel 358 or not(BZ#921961, BZ#896224)?
 

Thank you very much.

logs:
Jun 26 17:57:06 cqdx kernel: BUG: soft lockup - CPU#0 stuck for 67s! [xfslogd/0:2014]
Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Jun 26 17:57:06 cqdx kernel: CPU 0
Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Jun 26 17:57:06 cqdx kernel:
Jun 26 17:57:06 cqdx kernel: Pid: 2014, comm: xfslogd/0 Not tainted 2.6.32-279.19.1.01.qiyi.el6.x86_64 #1 Dell Inc. PowerEdge R720xd/0T0WRN
Jun 26 17:57:06 cqdx kernel: RIP: 0010:[<ffffffff814ec55c>]  [<ffffffff814ec55c>] _spin_lock+0x1c/0x30
Jun 26 17:57:06 cqdx kernel: RSP: 0000:ffff88101465bd50  EFLAGS: 00000293
Jun 26 17:57:06 cqdx kernel: RAX: 0000000000005468 RBX: ffff88101465bd50 RCX: ffffe8efaf600e88
Jun 26 17:57:06 cqdx kernel: RDX: 0000000000005466 RSI: ffff882011755898 RDI: ffff88201955b100
Jun 26 17:57:06 cqdx kernel: RBP: ffffffff8100bb8e R08: 0000000000000000 R09: 00000000ffffffff
Jun 26 17:57:06 cqdx kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff88101465bd70
Jun 26 17:57:06 cqdx kernel: R13: ffffffff8100b9ce R14: 0000000000000400 R15: 0000000000000094
Jun 26 17:57:06 cqdx kernel: FS:  0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
Jun 26 17:57:06 cqdx kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Jun 26 17:57:06 cqdx kernel: CR2: 00007f9e001dc0a0 CR3: 000000201311e000 CR4: 00000000000406f0
Jun 26 17:57:06 cqdx kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 26 17:57:06 cqdx kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jun 26 17:57:06 cqdx kernel: Process xfslogd/0 (pid: 2014, threadinfo ffff88101465a000, task ffff881015b75500)
Jun 26 17:57:06 cqdx kernel: Stack:
Jun 26 17:57:06 cqdx kernel: ffff88101465bd80 ffffffffa02a2b3b 0000000000000001 ffff8820137c7300
Jun 26 17:57:06 cqdx kernel: <d> ffff882016daee40 ffff882017e03800 ffff88101465bda0 ffffffffa02a2adc
Jun 26 17:57:06 cqdx kernel: <d> ffff882016d330a8 ffff882016daee40 ffff88101465be10 ffffffffa02a2d51
Jun 26 17:57:06 cqdx kernel: Call Trace:
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2b3b>] ? xfs_buf_iodone+0x2b/0x60 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2adc>] ? xfs_buf_do_callbacks+0x3c/0x50 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2d51>] ? xfs_buf_iodone_callbacks+0x51/0x270 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db190>] ? xfs_buf_iodone_work+0x0/0x50 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db1b6>] ? xfs_buf_iodone_work+0x26/0x50 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8108b120>] ? worker_thread+0x170/0x2a0
Jun 26 17:57:06 cqdx kernel: [<ffffffff81090990>] ? autoremove_wake_function+0x0/0x40
Jun 26 17:57:06 cqdx kernel: [<ffffffff8108afb0>] ? worker_thread+0x0/0x2a0
Jun 26 17:57:06 cqdx kernel: [<ffffffff81090626>] ? kthread+0x96/0xa0
Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Jun 26 17:57:06 cqdx kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jun 26 17:57:06 cqdx kernel: Code: 81 2f 00 00 00 01 74 05 e8 72 8c d8 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55
Jun 26 17:57:06 cqdx kernel: Call Trace:
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2b3b>] ? xfs_buf_iodone+0x2b/0x60 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2adc>] ? xfs_buf_do_callbacks+0x3c/0x50 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2d51>] ? xfs_buf_iodone_callbacks+0x51/0x270 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db190>] ? xfs_buf_iodone_work+0x0/0x50 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db1b6>] ? xfs_buf_iodone_work+0x26/0x50 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8108b120>] ? worker_thread+0x170/0x2a0
Jun 26 17:57:06 cqdx kernel: [<ffffffff81090990>] ? autoremove_wake_function+0x0/0x40
Jun 26 17:57:06 cqdx kernel: [<ffffffff8108afb0>] ? worker_thread+0x0/0x2a0
Jun 26 17:57:06 cqdx kernel: [<ffffffff81090626>] ? kthread+0x96/0xa0
Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Jun 26 17:57:06 cqdx kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jun 26 17:57:06 cqdx kernel: BUG: soft lockup - CPU#15 stuck for 67s! [glusterfsd:3245]
Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Jun 26 17:57:06 cqdx kernel: CPU 15
Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Jun 26 17:57:06 cqdx kernel:
Jun 26 17:57:06 cqdx kernel: Pid: 3245, comm: glusterfsd Not tainted 2.6.32-279.19.1.01.qiyi.el6.x86_64 #1 Dell Inc. PowerEdge R720xd/0T0WRN
Jun 26 17:57:06 cqdx kernel: RIP: 0010:[<ffffffff814ec55e>]  [<ffffffff814ec55e>] _spin_lock+0x1e/0x30
Jun 26 17:57:06 cqdx kernel: RSP: 0018:ffff880dceb5d528  EFLAGS: 00000297
Jun 26 17:57:06 cqdx kernel: RAX: 00000000000087eb RBX: ffff880dceb5d528 RCX: ffff8820137923e0
Jun 26 17:57:06 cqdx kernel: RDX: 00000000000087ea RSI: 0000000000006e5c RDI: ffff882016d330a8
Jun 26 17:57:06 cqdx kernel: RBP: ffffffff8100bb8e R08: ffff880dceb5d674 R09: ffff880dceb5d670
Jun 26 17:57:06 cqdx kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Jun 26 17:57:06 cqdx kernel: R13: 0000000000000cb5 R14: ffff8810788166e8 R15: ffff882016ea9400
Jun 26 17:57:06 cqdx kernel: FS:  00007f6cf3fff700(0000) GS:ffff8810788e0000(0000) knlGS:0000000000000000
Jun 26 17:57:06 cqdx kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 26 17:57:06 cqdx kernel: CR2: 0000000000481046 CR3: 000000201311e000 CR4: 00000000000406e0
Jun 26 17:57:06 cqdx kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 26 17:57:06 cqdx kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jun 26 17:57:06 cqdx kernel: Process glusterfsd (pid: 3245, threadinfo ffff880dceb5c000, task ffff8810191ecaa0)
Jun 26 17:57:06 cqdx kernel: Stack:
Jun 26 17:57:06 cqdx kernel: ffff880dceb5d5d8 ffffffffa02c27a8 ffff881078816680 ffff880dceb5d670
Jun 26 17:57:06 cqdx kernel: <d> ffff880dceb5d674 ffff880dceb5d668 00006e5c0000000f 0000000000000001
Jun 26 17:57:06 cqdx kernel: <d> 00000000000172a0 ffff882016d33080 ffff880dceb5d588 0000000000000082
Jun 26 17:57:06 cqdx kernel: Call Trace:
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c27a8>] ? xlog_state_get_iclog_space+0x68/0x2e0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8105f8ac>] ? try_to_wake_up+0x24c/0x3e0
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2bf1>] ? xlog_write+0x181/0x6d0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8126f9f5>] ? rwsem_wake+0x75/0x170
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c3c41>] ? xlog_cil_push+0x261/0x3d0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c43a5>] ? xlog_cil_force_lsn+0x115/0x120 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa029a30d>] ? xfs_bmap_add_extent+0xad/0x3c0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2308>] ? _xfs_log_force+0x68/0x240 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2718>] ? xfs_log_force+0x18/0x40 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d99d0>] ? xfs_buf_cond_lock+0x90/0xc0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02dadb3>] ? _xfs_buf_find+0xe3/0x280 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02dafbb>] ? xfs_buf_get+0x6b/0x1a0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d16d8>] ? xfs_trans_get_buf+0xe8/0x180 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a4a67>] ? xfs_da_do_buf+0x4c7/0x770 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a5fda>] ? xfs_da_get_buf+0x2a/0x30 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02aa49d>] ? xfs_dir2_data_init+0x4d/0xf0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02aa49d>] ? xfs_dir2_data_init+0x4d/0xf0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a8c23>] ? xfs_dir2_sf_to_block+0xe3/0x5e0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02b5ce6>] ? xfs_iunlock+0x86/0x100 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8118ecaa>] ? dput+0x9a/0x150
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02b0c18>] ? xfs_dir2_sf_addname+0x458/0x5b0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d6687>] ? kmem_zone_alloc+0x77/0xf0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a81bc>] ? xfs_dir_createname+0x14c/0x1b0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d3390>] ? xfs_link+0x1f0/0x300 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff81195560>] ? mntput_no_expire+0x30/0x110
Jun 26 17:57:06 cqdx kernel: [<ffffffff8118e7a7>] ? __d_lookup+0xa7/0x150
Jun 26 17:57:06 cqdx kernel: [<ffffffff8107721a>] ? capable+0x2a/0x60
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02e1120>] ? xfs_vn_link+0x40/0x70 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8118379c>] ? vfs_link+0x11c/0x190
Jun 26 17:57:06 cqdx kernel: [<ffffffff81182aaa>] ? lookup_hash+0x3a/0x50
Jun 26 17:57:06 cqdx kernel: [<ffffffff81186b71>] ? sys_linkat+0x141/0x160
Jun 26 17:57:06 cqdx kernel: [<ffffffff8117b83b>] ? vfs_stat+0x1b/0x20
Jun 26 17:57:06 cqdx kernel: [<ffffffff8117b864>] ? sys_newstat+0x24/0x50
Jun 26 17:57:06 cqdx kernel: [<ffffffff81186bae>] ? sys_link+0x1e/0x20
Jun 26 17:57:06 cqdx kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
Jun 26 17:57:06 cqdx kernel: Code: 00 00 00 01 74 05 e8 72 8c d8 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89
Jun 26 17:57:06 cqdx kernel: Call Trace:
Jun 26 17:57:06 cqdx kernel: [<ffffffff8106210b>] ? enqueue_task_fair+0xfb/0x100
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c27a8>] ? xlog_state_get_iclog_space+0x68/0x2e0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8105f8ac>] ? try_to_wake_up+0x24c/0x3e0
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2bf1>] ? xlog_write+0x181/0x6d0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8126f9f5>] ? rwsem_wake+0x75/0x170
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c3c41>] ? xlog_cil_push+0x261/0x3d0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c43a5>] ? xlog_cil_force_lsn+0x115/0x120 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa029a30d>] ? xfs_bmap_add_extent+0xad/0x3c0 [xfs]


--
符永涛 
_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs
[Index of Archives]     [Linux XFS Devel]     [Linux Filesystem Development]     [Filesystem Testing]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux