Hello, I'm working on OpenStack, and we hit a bug on Swift (distributed storage). The Linux kernel 3.2 logged many "soft lockup" messages which looks to be related to XFS: see kernel messages at the end of this email (it's only an extract of first messages, there are more later). I'm trying to find with google if the bug is known or not. "soft lockup" looks to be a "common" bug on Linux, but it's hard (for me) to check if existing bug reports are similar to our issue or not. And how can I check if it's an hardware issue or a Linux kernel (XFS?) issue? The following issues look similiar: - http://oss.sgi.com/archives/xfs/2013-06/msg00865.html : Centos, Linux kernel 2.6.32 This email mentions two bugzilla issues, probably the following ones, but I'm not allowed to see them :-/ https://bugzilla.redhat.com/show_bug.cgi?id=896224 https://bugzilla.redhat.com/show_bug.cgi?id=921961 - https://retrace.fedoraproject.org/faf/reports/98448/ : Fedora 18, Linux kernel 3.8 - https://bugs.centos.org/view.php?id=6287 : Linux 2.6.32 - https://lkml.org/lkml/2013/10/16/263 : RC version of Linux (3.12.0-rc1), maybe not reliable Some info about the server: - Dell PowerEdge R720xd/0X6H47 - Ubuntu 12.04 - Linux kernel 3.2.0-51-generic - Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz : 32 cores - 2 NUMA nodes There are 26 disks, HDD and SSD. All partitions used to store Swift objects are formatted with XFS, only a few system partitions are formatted as ext4 (and ext3 for /boot). There are 2 disk controllers: internal and a PCI Express controller. Victor Linux kernel messages: BUG: soft lockup - CPU#28 stuck for 22s! [swift-object-se:4354] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 28 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 4354, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8802a0d95b68 EFLAGS: 00000293 RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff88080830a960 RDX: 0000000000007c1e RSI: ffff8801c6235c00 RDI: ffff88082420fea8 RBP: ffff8802a0d95b68 R08: ffff8802a0d95ca0 R09: ffff8802a0d95cac R10: 0000000000002134 R11: 000000000000000a R12: ffff8802a0d95b50 R13: ffff88082420fe00 R14: ffff880424d676f8 R15: ffff8802a0d95ae8 FS: 00007f3d8dffb700(0000) GS:ffff88042fdc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swift-object-se (pid: 4354, threadinfo ffff8802a0d94000, task ffff8801c6235c00) Stack: ffff8802a0d95b78 ffffffff816606be ffff8802a0d95c18 ffffffffa01cc548 ffff8802a0d95cac ffff8802a0d95ca0 ffff8802a0d95c90 000022546322c000 00000000000144c0 ffff8801c6235c00 ffff8802a0d95df5 000000000000000e Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffff8119e031>] ? setxattr+0xd1/0x120 [<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [<ffffffff8165f986>] ? down_read+0x16/0x2b [<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs] [<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs] [<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs] [<ffffffff811a8c56>] do_fsync+0x56/0x80 [<ffffffff811a8f80>] sys_fsync+0x10/0x20 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffff8119e031>] ? setxattr+0xd1/0x120 [<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [<ffffffff8165f986>] ? down_read+0x16/0x2b [<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs] [<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs] [<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs] [<ffffffff811a8c56>] do_fsync+0x56/0x80 [<ffffffff811a8f80>] sys_fsync+0x10/0x20 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#30 stuck for 22s! [xfsaild/sdf1:2876] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 30 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 2876, comm: xfsaild/sdf1 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff88041116bc20 EFLAGS: 00000297 RAX: 00000000000007e5 RBX: 0000000000000000 RCX: 0000000000000231 RDX: 00000000000007e6 RSI: ffff88082420fea8 RDI: ffff880824b32e78 RBP: ffff88041116bc20 R08: 0000000000000001 R09: 0000000000000000 R10: ffff88041116ba24 R11: 0000000000000000 R12: 0000000000000001 R13: ffff8807eeedad80 R14: 0003000100000000 R15: ffff88041116bb88 FS: 0000000000000000(0000) GS:ffff88042fde0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f3dbcd1b000 CR3: 0000000001c05000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process xfsaild/sdf1 (pid: 2876, threadinfo ffff88041116a000, task ffff880819d35c00) Stack: ffff88041116bc30 ffffffff816606be ffff88041116bc50 ffffffffa01d13d4 ffff88082420fe00 ffff88082420fe00 ffff88041116bc70 ffffffffa01cc3fe ffff88041116bcc0 ffff88042121c800 ffff88041116bca0 ffffffffa01cc4d3 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs] [<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs] [<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs] [<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs] [<ffffffff810916d6>] ? down_trylock+0x36/0x50 [<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs] [<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs] [<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs] [<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs] [<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs] [<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs] [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13 Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs] [<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs] [<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs] [<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs] [<ffffffff810916d6>] ? down_trylock+0x36/0x50 [<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs] [<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs] [<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs] [<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs] [<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs] [<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs] [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13 BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:3:27580] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 0 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 27580, comm: kworker/0:3 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8801aeae3d40 EFLAGS: 00000293 RAX: 00000000000007e5 RBX: ffff8801aeae3cd0 RCX: 0000000000000003 RDX: 00000000000007e7 RSI: ffff880823a117d0 RDI: ffff880824b32e78 RBP: ffff8801aeae3d40 R08: ffff8801aeae2000 R09: 0000000000000093 R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002 R13: ffff88042fc137c0 R14: ffff880425a37a00 R15: 000000000000014b FS: 0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007effa8b52ef7 CR3: 0000000001c05000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/0:3 (pid: 27580, threadinfo ffff8801aeae2000, task ffff8800a5015c00) Stack: ffff8801aeae3d50 ffffffff816606be ffff8801aeae3d80 ffffffffa01cfe1b ffff8806e3d3dc48 ffff8800339f1600 ffff880421b76940 ffff8801bbe53500 ffff8801aeae3da0 ffffffffa01cef1c 0000000000000000 ffff880421b76940 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs] [<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs] [<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs] [<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs] [<ffffffff81085e37>] process_one_work+0x127/0x470 [<ffffffff81086f44>] worker_thread+0x164/0x370 [<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130 [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13 Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs] [<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs] [<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs] [<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs] [<ffffffff81085e37>] process_one_work+0x127/0x470 [<ffffffff81086f44>] worker_thread+0x164/0x370 [<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130 [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13 BUG: soft lockup - CPU#7 stuck for 23s! [swift-object-se:4340] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 7 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 4340, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8803362a1498 EFLAGS: 00000297 RAX: 0000000000007c1c RBX: 0000000700000006 RCX: ffff8804186a7ce8 RDX: 0000000000007c1d RSI: ffff8804112c9700 RDI: ffff88082420fea8 RBP: ffff8803362a1498 R08: ffff8803362a15d0 R09: ffff8803362a15dc R10: 000000000000121c R11: 0000000000000006 R12: ffffffffa01a6587 R13: ffff8803362a1428 R14: ffff8803362a1544 R15: 0000000000000000 FS: 00007f3db1ffb700(0000) GS:ffff88083fc60000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3dbcd1b000 CR3: 00000002955f6000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swift-object-se (pid: 4340, threadinfo ffff8803362a0000, task ffff8804112c9700) Stack: ffff8803362a14a8 ffffffff816606be ffff8803362a1548 ffffffffa01cc548 ffff8803362a15dc ffff8803362a15d0 ffff8803362a15c0 000013248103ec59 00000000000144c0 ffff8804112c9700 0000000000000286 ffff880556253d78 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff810918a2>] ? up+0x32/0x50 [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffffa01bd97e>] ? xfs_iext_insert+0x7e/0x100 [xfs] [<ffffffffa019b662>] ? xfs_bmap_add_extent_hole_real+0x112/0xad0 [xfs] [<ffffffffa01cd098>] _xfs_log_force+0x68/0x2a0 [xfs] [<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs] [<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs] [<ffffffffa017621b>] ? _xfs_buf_find+0x4b/0x240 [xfs] [<ffffffffa017628c>] _xfs_buf_find+0xbc/0x240 [xfs] [<ffffffffa0176694>] xfs_buf_get+0x34/0x1c0 [xfs] [<ffffffffa01d2507>] xfs_trans_get_buf+0x127/0x1a0 [xfs] [<ffffffffa01abce9>] xfs_da_do_buf+0x3d9/0x630 [xfs] [<ffffffffa01abfd1>] xfs_da_get_buf+0x21/0x30 [xfs] [<ffffffffa01949b7>] xfs_attr_leaf_create.isra.2+0x47/0x100 [xfs] [<ffffffffa0197264>] xfs_attr_shortform_to_leaf+0x104/0x240 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa018c929>] ? kmem_zone_zalloc+0x39/0x50 [xfs] [<ffffffffa0193306>] xfs_attr_set_int+0x406/0x4e0 [xfs] [<ffffffffa019394f>] xfs_attr_set+0x7f/0x90 [xfs] [<ffffffffa0188057>] xfs_xattr_set+0x47/0x60 [xfs] [<ffffffff8119d4fb>] generic_setxattr+0x6b/0x90 [<ffffffff8119dd5b>] __vfs_setxattr_noperm+0x7b/0x1c0 [<ffffffff812decce>] ? evm_inode_setxattr+0xe/0x10 [<ffffffff8119df5c>] vfs_setxattr+0xbc/0xc0 [<ffffffff8119e026>] setxattr+0xc6/0x120 [<ffffffff811b8a62>] ? fsnotify+0x1c2/0x2a0 [<ffffffff810a1e6c>] ? do_futex+0x7c/0x1b0 [<ffffffff8119e43b>] sys_fsetxattr+0xbb/0xe0 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff810918a2>] ? up+0x32/0x50 [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffffa01bd97e>] ? xfs_iext_insert+0x7e/0x100 [xfs] [<ffffffffa019b662>] ? xfs_bmap_add_extent_hole_real+0x112/0xad0 [xfs] [<ffffffffa01cd098>] _xfs_log_force+0x68/0x2a0 [xfs] [<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs] [<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs] [<ffffffffa017621b>] ? _xfs_buf_find+0x4b/0x240 [xfs] [<ffffffffa017628c>] _xfs_buf_find+0xbc/0x240 [xfs] [<ffffffffa0176694>] xfs_buf_get+0x34/0x1c0 [xfs] [<ffffffffa01d2507>] xfs_trans_get_buf+0x127/0x1a0 [xfs] [<ffffffffa01abce9>] xfs_da_do_buf+0x3d9/0x630 [xfs] [<ffffffffa01abfd1>] xfs_da_get_buf+0x21/0x30 [xfs] [<ffffffffa01949b7>] xfs_attr_leaf_create.isra.2+0x47/0x100 [xfs] [<ffffffffa0197264>] xfs_attr_shortform_to_leaf+0x104/0x240 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa018c929>] ? kmem_zone_zalloc+0x39/0x50 [xfs] [<ffffffffa0193306>] xfs_attr_set_int+0x406/0x4e0 [xfs] [<ffffffffa019394f>] xfs_attr_set+0x7f/0x90 [xfs] [<ffffffffa0188057>] xfs_xattr_set+0x47/0x60 [xfs] [<ffffffff8119d4fb>] generic_setxattr+0x6b/0x90 [<ffffffff8119dd5b>] __vfs_setxattr_noperm+0x7b/0x1c0 [<ffffffff812decce>] ? evm_inode_setxattr+0xe/0x10 [<ffffffff8119df5c>] vfs_setxattr+0xbc/0xc0 [<ffffffff8119e026>] setxattr+0xc6/0x120 [<ffffffff811b8a62>] ? fsnotify+0x1c2/0x2a0 [<ffffffff810a1e6c>] ? do_futex+0x7c/0x1b0 [<ffffffff8119e43b>] sys_fsetxattr+0xbb/0xe0 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#9 stuck for 23s! [swift-object-se:4362] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 9 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 4362, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8803df863b68 EFLAGS: 00000293 RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff880418632708 RDX: 0000000000007c1f RSI: ffff880110e99700 RDI: ffff88082420fea8 RBP: ffff8803df863b68 R08: ffff8803df863ca0 R09: ffff8803df863cac R10: 00000000000021c4 R11: 000000000000000a R12: ffff8803df863b50 R13: ffff88082420fe00 R14: ffff880808307a40 R15: ffff8803df863ae8 FS: 00007f3d6f7fe700(0000) GS:ffff88083fc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swift-object-se (pid: 4362, threadinfo ffff8803df862000, task ffff880110e99700) Stack: ffff8803df863b78 ffffffff816606be ffff8803df863c18 ffffffffa01cc548 ffff8803df863cac ffff8803df863ca0 ffff8803df863c90 000022e4244506c0 00000000000144c0 ffff880110e99700 ffff8803df863df5 000000000000000e Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffff8119e031>] ? setxattr+0xd1/0x120 [<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [<ffffffff8165f986>] ? down_read+0x16/0x2b [<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs] [<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs] [<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs] [<ffffffff811a8c56>] do_fsync+0x56/0x80 [<ffffffff811a8f80>] sys_fsync+0x10/0x20 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffff8119e031>] ? setxattr+0xd1/0x120 [<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [<ffffffff8165f986>] ? down_read+0x16/0x2b [<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs] [<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs] [<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs] [<ffffffff811a8c56>] do_fsync+0x56/0x80 [<ffffffff811a8f80>] sys_fsync+0x10/0x20 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#28 stuck for 22s! [swift-object-se:4354] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 28 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 4354, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8802a0d95b68 EFLAGS: 00000293 RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff88080830a960 RDX: 0000000000007c1e RSI: ffff8801c6235c00 RDI: ffff88082420fea8 RBP: ffff8802a0d95b68 R08: ffff8802a0d95ca0 R09: ffff8802a0d95cac R10: 0000000000002134 R11: 000000000000000a R12: ffff8802a0d95b50 R13: ffff88082420fe00 R14: ffff880424d676f8 R15: ffff8802a0d95ae8 FS: 00007f3d8dffb700(0000) GS:ffff88042fdc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swift-object-se (pid: 4354, threadinfo ffff8802a0d94000, task ffff8801c6235c00) Stack: ffff8802a0d95b78 ffffffff816606be ffff8802a0d95c18 ffffffffa01cc548 ffff8802a0d95cac ffff8802a0d95ca0 ffff8802a0d95c90 000022546322c000 00000000000144c0 ffff8801c6235c00 ffff8802a0d95df5 000000000000000e Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffff8119e031>] ? setxattr+0xd1/0x120 [<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [<ffffffff8165f986>] ? down_read+0x16/0x2b [<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs] [<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs] [<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs] [<ffffffff811a8c56>] do_fsync+0x56/0x80 [<ffffffff811a8f80>] sys_fsync+0x10/0x20 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffff8119e031>] ? setxattr+0xd1/0x120 [<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [<ffffffff8165f986>] ? down_read+0x16/0x2b [<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs] [<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs] [<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs] [<ffffffff811a8c56>] do_fsync+0x56/0x80 [<ffffffff811a8f80>] sys_fsync+0x10/0x20 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#30 stuck for 22s! [xfsaild/sdf1:2876] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 30 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 2876, comm: xfsaild/sdf1 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff88041116bc20 EFLAGS: 00000297 RAX: 00000000000007e5 RBX: 0000000000000000 RCX: 0000000000000231 RDX: 00000000000007e6 RSI: ffff88082420fea8 RDI: ffff880824b32e78 RBP: ffff88041116bc20 R08: 0000000000000001 R09: 0000000000000000 R10: ffff88041116ba24 R11: 0000000000000000 R12: 0000000000000001 R13: ffff8807eeedad80 R14: 0003000100000000 R15: ffff88041116bb88 FS: 0000000000000000(0000) GS:ffff88042fde0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f3dbcd1b000 CR3: 0000000001c05000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process xfsaild/sdf1 (pid: 2876, threadinfo ffff88041116a000, task ffff880819d35c00) Stack: ffff88041116bc30 ffffffff816606be ffff88041116bc50 ffffffffa01d13d4 ffff88082420fe00 ffff88082420fe00 ffff88041116bc70 ffffffffa01cc3fe ffff88041116bcc0 ffff88042121c800 ffff88041116bca0 ffffffffa01cc4d3 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs] [<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs] [<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs] [<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs] [<ffffffff810916d6>] ? down_trylock+0x36/0x50 [<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs] [<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs] [<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs] [<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs] [<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs] [<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs] [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13 Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs] [<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs] [<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs] [<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs] [<ffffffff810916d6>] ? down_trylock+0x36/0x50 [<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs] [<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs] [<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs] [<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs] [<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs] [<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs] [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13 BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:3:27580] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 0 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 27580, comm: kworker/0:3 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc2>] [<ffffffff8103ebc2>] __ticket_spin_lock+0x22/0x30 RSP: 0018:ffff8801aeae3d40 EFLAGS: 00000293 RAX: 00000000000007e5 RBX: ffff8801aeae3cd0 RCX: 0000000000000003 RDX: 00000000000007e7 RSI: ffff880823a117d0 RDI: ffff880824b32e78 RBP: ffff8801aeae3d40 R08: ffff8801aeae2000 R09: 0000000000000093 R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002 R13: ffff88042fc137c0 R14: ffff880425a37a00 R15: 000000000000014b FS: 0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007effa8b52ef7 CR3: 0000000001c05000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/0:3 (pid: 27580, threadinfo ffff8801aeae2000, task ffff8800a5015c00) Stack: ffff8801aeae3d50 ffffffff816606be ffff8801aeae3d80 ffffffffa01cfe1b ffff8806e3d3dc48 ffff8800339f1600 ffff880421b76940 ffff8801bbe53500 ffff8801aeae3da0 ffffffffa01cef1c 0000000000000000 ffff880421b76940 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs] [<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs] [<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs] [<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs] [<ffffffff81085e37>] process_one_work+0x127/0x470 [<ffffffff81086f44>] worker_thread+0x164/0x370 [<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130 [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13 Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs] [<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs] [<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs] [<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs] [<ffffffff81085e37>] process_one_work+0x127/0x470 [<ffffffff81086f44>] worker_thread+0x164/0x370 [<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130 [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13
BUG: soft lockup - CPU#28 stuck for 22s! [swift-object-se:4354] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 28 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 4354, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8802a0d95b68 EFLAGS: 00000293 RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff88080830a960 RDX: 0000000000007c1e RSI: ffff8801c6235c00 RDI: ffff88082420fea8 RBP: ffff8802a0d95b68 R08: ffff8802a0d95ca0 R09: ffff8802a0d95cac R10: 0000000000002134 R11: 000000000000000a R12: ffff8802a0d95b50 R13: ffff88082420fe00 R14: ffff880424d676f8 R15: ffff8802a0d95ae8 FS: 00007f3d8dffb700(0000) GS:ffff88042fdc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swift-object-se (pid: 4354, threadinfo ffff8802a0d94000, task ffff8801c6235c00) Stack: ffff8802a0d95b78 ffffffff816606be ffff8802a0d95c18 ffffffffa01cc548 ffff8802a0d95cac ffff8802a0d95ca0 ffff8802a0d95c90 000022546322c000 00000000000144c0 ffff8801c6235c00 ffff8802a0d95df5 000000000000000e Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffff8119e031>] ? setxattr+0xd1/0x120 [<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [<ffffffff8165f986>] ? down_read+0x16/0x2b [<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs] [<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs] [<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs] [<ffffffff811a8c56>] do_fsync+0x56/0x80 [<ffffffff811a8f80>] sys_fsync+0x10/0x20 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffff8119e031>] ? setxattr+0xd1/0x120 [<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [<ffffffff8165f986>] ? down_read+0x16/0x2b [<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs] [<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs] [<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs] [<ffffffff811a8c56>] do_fsync+0x56/0x80 [<ffffffff811a8f80>] sys_fsync+0x10/0x20 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#30 stuck for 22s! [xfsaild/sdf1:2876] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 30 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 2876, comm: xfsaild/sdf1 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff88041116bc20 EFLAGS: 00000297 RAX: 00000000000007e5 RBX: 0000000000000000 RCX: 0000000000000231 RDX: 00000000000007e6 RSI: ffff88082420fea8 RDI: ffff880824b32e78 RBP: ffff88041116bc20 R08: 0000000000000001 R09: 0000000000000000 R10: ffff88041116ba24 R11: 0000000000000000 R12: 0000000000000001 R13: ffff8807eeedad80 R14: 0003000100000000 R15: ffff88041116bb88 FS: 0000000000000000(0000) GS:ffff88042fde0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f3dbcd1b000 CR3: 0000000001c05000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process xfsaild/sdf1 (pid: 2876, threadinfo ffff88041116a000, task ffff880819d35c00) Stack: ffff88041116bc30 ffffffff816606be ffff88041116bc50 ffffffffa01d13d4 ffff88082420fe00 ffff88082420fe00 ffff88041116bc70 ffffffffa01cc3fe ffff88041116bcc0 ffff88042121c800 ffff88041116bca0 ffffffffa01cc4d3 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs] [<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs] [<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs] [<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs] [<ffffffff810916d6>] ? down_trylock+0x36/0x50 [<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs] [<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs] [<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs] [<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs] [<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs] [<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs] [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13 Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs] [<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs] [<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs] [<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs] [<ffffffff810916d6>] ? down_trylock+0x36/0x50 [<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs] [<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs] [<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs] [<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs] [<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs] [<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs] [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13 BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:3:27580] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 0 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 27580, comm: kworker/0:3 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8801aeae3d40 EFLAGS: 00000293 RAX: 00000000000007e5 RBX: ffff8801aeae3cd0 RCX: 0000000000000003 RDX: 00000000000007e7 RSI: ffff880823a117d0 RDI: ffff880824b32e78 RBP: ffff8801aeae3d40 R08: ffff8801aeae2000 R09: 0000000000000093 R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002 R13: ffff88042fc137c0 R14: ffff880425a37a00 R15: 000000000000014b FS: 0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007effa8b52ef7 CR3: 0000000001c05000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/0:3 (pid: 27580, threadinfo ffff8801aeae2000, task ffff8800a5015c00) Stack: ffff8801aeae3d50 ffffffff816606be ffff8801aeae3d80 ffffffffa01cfe1b ffff8806e3d3dc48 ffff8800339f1600 ffff880421b76940 ffff8801bbe53500 ffff8801aeae3da0 ffffffffa01cef1c 0000000000000000 ffff880421b76940 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs] [<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs] [<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs] [<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs] [<ffffffff81085e37>] process_one_work+0x127/0x470 [<ffffffff81086f44>] worker_thread+0x164/0x370 [<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130 [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13 Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs] [<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs] [<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs] [<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs] [<ffffffff81085e37>] process_one_work+0x127/0x470 [<ffffffff81086f44>] worker_thread+0x164/0x370 [<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130 [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13 BUG: soft lockup - CPU#7 stuck for 23s! [swift-object-se:4340] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 7 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 4340, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8803362a1498 EFLAGS: 00000297 RAX: 0000000000007c1c RBX: 0000000700000006 RCX: ffff8804186a7ce8 RDX: 0000000000007c1d RSI: ffff8804112c9700 RDI: ffff88082420fea8 RBP: ffff8803362a1498 R08: ffff8803362a15d0 R09: ffff8803362a15dc R10: 000000000000121c R11: 0000000000000006 R12: ffffffffa01a6587 R13: ffff8803362a1428 R14: ffff8803362a1544 R15: 0000000000000000 FS: 00007f3db1ffb700(0000) GS:ffff88083fc60000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3dbcd1b000 CR3: 00000002955f6000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swift-object-se (pid: 4340, threadinfo ffff8803362a0000, task ffff8804112c9700) Stack: ffff8803362a14a8 ffffffff816606be ffff8803362a1548 ffffffffa01cc548 ffff8803362a15dc ffff8803362a15d0 ffff8803362a15c0 000013248103ec59 00000000000144c0 ffff8804112c9700 0000000000000286 ffff880556253d78 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff810918a2>] ? up+0x32/0x50 [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffffa01bd97e>] ? xfs_iext_insert+0x7e/0x100 [xfs] [<ffffffffa019b662>] ? xfs_bmap_add_extent_hole_real+0x112/0xad0 [xfs] [<ffffffffa01cd098>] _xfs_log_force+0x68/0x2a0 [xfs] [<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs] [<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs] [<ffffffffa017621b>] ? _xfs_buf_find+0x4b/0x240 [xfs] [<ffffffffa017628c>] _xfs_buf_find+0xbc/0x240 [xfs] [<ffffffffa0176694>] xfs_buf_get+0x34/0x1c0 [xfs] [<ffffffffa01d2507>] xfs_trans_get_buf+0x127/0x1a0 [xfs] [<ffffffffa01abce9>] xfs_da_do_buf+0x3d9/0x630 [xfs] [<ffffffffa01abfd1>] xfs_da_get_buf+0x21/0x30 [xfs] [<ffffffffa01949b7>] xfs_attr_leaf_create.isra.2+0x47/0x100 [xfs] [<ffffffffa0197264>] xfs_attr_shortform_to_leaf+0x104/0x240 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa018c929>] ? kmem_zone_zalloc+0x39/0x50 [xfs] [<ffffffffa0193306>] xfs_attr_set_int+0x406/0x4e0 [xfs] [<ffffffffa019394f>] xfs_attr_set+0x7f/0x90 [xfs] [<ffffffffa0188057>] xfs_xattr_set+0x47/0x60 [xfs] [<ffffffff8119d4fb>] generic_setxattr+0x6b/0x90 [<ffffffff8119dd5b>] __vfs_setxattr_noperm+0x7b/0x1c0 [<ffffffff812decce>] ? evm_inode_setxattr+0xe/0x10 [<ffffffff8119df5c>] vfs_setxattr+0xbc/0xc0 [<ffffffff8119e026>] setxattr+0xc6/0x120 [<ffffffff811b8a62>] ? fsnotify+0x1c2/0x2a0 [<ffffffff810a1e6c>] ? do_futex+0x7c/0x1b0 [<ffffffff8119e43b>] sys_fsetxattr+0xbb/0xe0 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff810918a2>] ? up+0x32/0x50 [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffffa01bd97e>] ? xfs_iext_insert+0x7e/0x100 [xfs] [<ffffffffa019b662>] ? xfs_bmap_add_extent_hole_real+0x112/0xad0 [xfs] [<ffffffffa01cd098>] _xfs_log_force+0x68/0x2a0 [xfs] [<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs] [<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs] [<ffffffffa017621b>] ? _xfs_buf_find+0x4b/0x240 [xfs] [<ffffffffa017628c>] _xfs_buf_find+0xbc/0x240 [xfs] [<ffffffffa0176694>] xfs_buf_get+0x34/0x1c0 [xfs] [<ffffffffa01d2507>] xfs_trans_get_buf+0x127/0x1a0 [xfs] [<ffffffffa01abce9>] xfs_da_do_buf+0x3d9/0x630 [xfs] [<ffffffffa01abfd1>] xfs_da_get_buf+0x21/0x30 [xfs] [<ffffffffa01949b7>] xfs_attr_leaf_create.isra.2+0x47/0x100 [xfs] [<ffffffffa0197264>] xfs_attr_shortform_to_leaf+0x104/0x240 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa018c929>] ? kmem_zone_zalloc+0x39/0x50 [xfs] [<ffffffffa0193306>] xfs_attr_set_int+0x406/0x4e0 [xfs] [<ffffffffa019394f>] xfs_attr_set+0x7f/0x90 [xfs] [<ffffffffa0188057>] xfs_xattr_set+0x47/0x60 [xfs] [<ffffffff8119d4fb>] generic_setxattr+0x6b/0x90 [<ffffffff8119dd5b>] __vfs_setxattr_noperm+0x7b/0x1c0 [<ffffffff812decce>] ? evm_inode_setxattr+0xe/0x10 [<ffffffff8119df5c>] vfs_setxattr+0xbc/0xc0 [<ffffffff8119e026>] setxattr+0xc6/0x120 [<ffffffff811b8a62>] ? fsnotify+0x1c2/0x2a0 [<ffffffff810a1e6c>] ? do_futex+0x7c/0x1b0 [<ffffffff8119e43b>] sys_fsetxattr+0xbb/0xe0 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#9 stuck for 23s! [swift-object-se:4362] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 9 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 4362, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8803df863b68 EFLAGS: 00000293 RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff880418632708 RDX: 0000000000007c1f RSI: ffff880110e99700 RDI: ffff88082420fea8 RBP: ffff8803df863b68 R08: ffff8803df863ca0 R09: ffff8803df863cac R10: 00000000000021c4 R11: 000000000000000a R12: ffff8803df863b50 R13: ffff88082420fe00 R14: ffff880808307a40 R15: ffff8803df863ae8 FS: 00007f3d6f7fe700(0000) GS:ffff88083fc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swift-object-se (pid: 4362, threadinfo ffff8803df862000, task ffff880110e99700) Stack: ffff8803df863b78 ffffffff816606be ffff8803df863c18 ffffffffa01cc548 ffff8803df863cac ffff8803df863ca0 ffff8803df863c90 000022e4244506c0 00000000000144c0 ffff880110e99700 ffff8803df863df5 000000000000000e Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffff8119e031>] ? setxattr+0xd1/0x120 [<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [<ffffffff8165f986>] ? down_read+0x16/0x2b [<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs] [<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs] [<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs] [<ffffffff811a8c56>] do_fsync+0x56/0x80 [<ffffffff811a8f80>] sys_fsync+0x10/0x20 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffff8119e031>] ? setxattr+0xd1/0x120 [<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [<ffffffff8165f986>] ? down_read+0x16/0x2b [<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs] [<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs] [<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs] [<ffffffff811a8c56>] do_fsync+0x56/0x80 [<ffffffff811a8f80>] sys_fsync+0x10/0x20 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#28 stuck for 22s! [swift-object-se:4354] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 28 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 4354, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8802a0d95b68 EFLAGS: 00000293 RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff88080830a960 RDX: 0000000000007c1e RSI: ffff8801c6235c00 RDI: ffff88082420fea8 RBP: ffff8802a0d95b68 R08: ffff8802a0d95ca0 R09: ffff8802a0d95cac R10: 0000000000002134 R11: 000000000000000a R12: ffff8802a0d95b50 R13: ffff88082420fe00 R14: ffff880424d676f8 R15: ffff8802a0d95ae8 FS: 00007f3d8dffb700(0000) GS:ffff88042fdc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swift-object-se (pid: 4354, threadinfo ffff8802a0d94000, task ffff8801c6235c00) Stack: ffff8802a0d95b78 ffffffff816606be ffff8802a0d95c18 ffffffffa01cc548 ffff8802a0d95cac ffff8802a0d95ca0 ffff8802a0d95c90 000022546322c000 00000000000144c0 ffff8801c6235c00 ffff8802a0d95df5 000000000000000e Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffff8119e031>] ? setxattr+0xd1/0x120 [<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [<ffffffff8165f986>] ? down_read+0x16/0x2b [<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs] [<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs] [<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs] [<ffffffff811a8c56>] do_fsync+0x56/0x80 [<ffffffff811a8f80>] sys_fsync+0x10/0x20 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cc548>] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [<ffffffff8116655d>] ? kmem_cache_alloc+0x11d/0x140 [<ffffffffa01ccb99>] xlog_write+0xe9/0x400 [xfs] [<ffffffffa018c877>] ? kmem_zone_alloc+0x67/0xe0 [xfs] [<ffffffffa01ce40c>] xlog_cil_push+0x20c/0x380 [xfs] [<ffffffffa01ced2e>] xlog_cil_force_lsn+0x10e/0x120 [xfs] [<ffffffff8119e031>] ? setxattr+0xd1/0x120 [<ffffffffa01cd367>] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [<ffffffff8165f986>] ? down_read+0x16/0x2b [<ffffffffa017d116>] ? xfs_iunlock+0xe6/0x150 [xfs] [<ffffffffa0179123>] ? xfs_file_fsync+0x113/0x300 [xfs] [<ffffffffa017920c>] xfs_file_fsync+0x1fc/0x300 [xfs] [<ffffffff811a8c56>] do_fsync+0x56/0x80 [<ffffffff811a8f80>] sys_fsync+0x10/0x20 [<ffffffff81668d02>] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#30 stuck for 22s! [xfsaild/sdf1:2876] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 30 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 2876, comm: xfsaild/sdf1 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc5>] [<ffffffff8103ebc5>] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff88041116bc20 EFLAGS: 00000297 RAX: 00000000000007e5 RBX: 0000000000000000 RCX: 0000000000000231 RDX: 00000000000007e6 RSI: ffff88082420fea8 RDI: ffff880824b32e78 RBP: ffff88041116bc20 R08: 0000000000000001 R09: 0000000000000000 R10: ffff88041116ba24 R11: 0000000000000000 R12: 0000000000000001 R13: ffff8807eeedad80 R14: 0003000100000000 R15: ffff88041116bb88 FS: 0000000000000000(0000) GS:ffff88042fde0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f3dbcd1b000 CR3: 0000000001c05000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process xfsaild/sdf1 (pid: 2876, threadinfo ffff88041116a000, task ffff880819d35c00) Stack: ffff88041116bc30 ffffffff816606be ffff88041116bc50 ffffffffa01d13d4 ffff88082420fe00 ffff88082420fe00 ffff88041116bc70 ffffffffa01cc3fe ffff88041116bcc0 ffff88042121c800 ffff88041116bca0 ffffffffa01cc4d3 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs] [<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs] [<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs] [<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs] [<ffffffff810916d6>] ? down_trylock+0x36/0x50 [<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs] [<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs] [<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs] [<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs] [<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs] [<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs] [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13 Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01d13d4>] xfs_ail_min_lsn+0x24/0x60 [xfs] [<ffffffffa01cc3fe>] xlog_assign_tail_lsn+0x1e/0x40 [xfs] [<ffffffffa01cc4d3>] xlog_state_release_iclog+0xb3/0xd0 [xfs] [<ffffffffa01cd156>] _xfs_log_force+0x126/0x2a0 [xfs] [<ffffffff810916d6>] ? down_trylock+0x36/0x50 [<ffffffffa01cd2e8>] xfs_log_force+0x18/0x40 [xfs] [<ffffffffa01760b0>] xfs_buf_trylock+0xd0/0xe0 [xfs] [<ffffffffa01cf1c8>] xfs_buf_item_trylock+0x58/0xc0 [xfs] [<ffffffffa01d177a>] xfsaild_push+0x24a/0x680 [xfs] [<ffffffffa01d1c12>] xfsaild+0x62/0xc0 [xfs] [<ffffffffa01d1bb0>] ? xfsaild_push+0x680/0x680 [xfs] [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13 BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:3:27580] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 0 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 27580, comm: kworker/0:3 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[<ffffffff8103ebc2>] [<ffffffff8103ebc2>] __ticket_spin_lock+0x22/0x30 RSP: 0018:ffff8801aeae3d40 EFLAGS: 00000293 RAX: 00000000000007e5 RBX: ffff8801aeae3cd0 RCX: 0000000000000003 RDX: 00000000000007e7 RSI: ffff880823a117d0 RDI: ffff880824b32e78 RBP: ffff8801aeae3d40 R08: ffff8801aeae2000 R09: 0000000000000093 R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002 R13: ffff88042fc137c0 R14: ffff880425a37a00 R15: 000000000000014b FS: 0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007effa8b52ef7 CR3: 0000000001c05000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/0:3 (pid: 27580, threadinfo ffff8801aeae2000, task ffff8800a5015c00) Stack: ffff8801aeae3d50 ffffffff816606be ffff8801aeae3d80 ffffffffa01cfe1b ffff8806e3d3dc48 ffff8800339f1600 ffff880421b76940 ffff8801bbe53500 ffff8801aeae3da0 ffffffffa01cef1c 0000000000000000 ffff880421b76940 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs] [<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs] [<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs] [<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs] [<ffffffff81085e37>] process_one_work+0x127/0x470 [<ffffffff81086f44>] worker_thread+0x164/0x370 [<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130 [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13 Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 Call Trace: [<ffffffff816606be>] _raw_spin_lock+0xe/0x20 [<ffffffffa01cfe1b>] xfs_buf_iodone+0x2b/0x60 [xfs] [<ffffffffa01cef1c>] xfs_buf_do_callbacks+0x3c/0x50 [xfs] [<ffffffffa01cef96>] xfs_buf_iodone_callbacks+0x46/0x220 [xfs] [<ffffffffa0176843>] xfs_buf_iodone_work+0x23/0x50 [xfs] [<ffffffff81085e37>] process_one_work+0x127/0x470 [<ffffffff81086f44>] worker_thread+0x164/0x370 [<ffffffff81086de0>] ? manage_workers.isra.31+0x130/0x130 [<ffffffff8108b78c>] kthread+0x8c/0xa0 [<ffffffff8166ae74>] kernel_thread_helper+0x4/0x10 [<ffffffff8108b700>] ? flush_kthread_worker+0xa0/0xa0 [<ffffffff8166ae70>] ? gs_change+0x13/0x13
_______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
