Freeze with cryptsetup->mdadm->lvm2->xfs(->nfs) get_active_stripe?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

after upgrading from 2.6.39.1 in December, I've been having freezes
with 3.1.x, 3.2.2.
In an attempt to find the cause, when upgrading to 3.3.1 in the
beginning of April I enabled all the debug settings I could find in
the kernel, but it never froze and I assumed that the bug had been
fixed.
4-5 days ago I upgraded to 3.4.3 without debug flags (they made
everything horribly slow), and today it froze again!

I can ssh in, but don't get further than "Last login: Tue Jun 26
11:26:10 CEST 2012 from 10.0.0.2 on pts/4". I assume it tries to write
new info to /var/log/ and hangs.

I got around 10 disks connected to the Intel motherboard controller
(ahci) and two sata_sil24 PCI-e cards.
They're not partitioned very wisely: All the disks have a cryptsetup
partition. Mdadm runs on top of the encrypted volumes. Lvm2 on the big
/dev/md0 device. With xfs filesystems inside. And this is also served
over nfs.
That is: cryptsetup -> mdadm -> lvm2 -> xfs (-> nfs).

Here is Show Blocked State from netconsole.  To me it looks like the
problem might be in get_active_stripe, but I don't understand much of
threads/synchronization.

Jun 26 22:27:29 [368353.502333] SysRq : Show Blocked State
Jun 26 22:27:29 [368353.502414]   task                        PC stack
  pid father
Jun 26 22:27:29 [368353.502478] flush-254:12    D ffff88010db48cc0
0  5522      2 0x00000000
Jun 26 22:27:29 [368353.502600]  ffff88010cfc9640 0000000000000046
ffff88010cfc95d0 ffff88010c3ccc80
Jun 26 22:27:29 [368353.502764]  ffff88010f0029c0 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.502924]  ffff88010db48cc0 00000000000107c0
ffff88010cfc9fd8 ffff88010cfc9fd8
Jun 26 22:27:29 [368353.503085] Call Trace:
Jun 26 22:27:29 [368353.503127]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.503176]  [<ffffffffa00b38ff>]
get_active_stripe+0x256/0x564 [raid456]
Jun 26 22:27:29 [368353.503220]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:29 [368353.503392]  [<ffffffffa00b5976>]
make_request+0x1dd/0x3bc [raid456]
Jun 26 22:27:29 [368353.503564]  [<ffffffff8103ffc0>] ? wake_up_bit+0x25/0x25
Jun 26 22:27:29 [368353.503738]  [<ffffffffa0069ef1>]
md_make_request+0xd0/0x1c2 [md_mod]
Jun 26 22:27:29 [368353.503970]  [<ffffffff811c6b93>] ? dm_request+0x1b4/0x1c3
Jun 26 22:27:29 [368353.504140]  [<ffffffff811c9c8c>] ? linear_merge+0x4d/0x58
Jun 26 22:27:29 [368353.504310]  [<ffffffff81119ad1>]
generic_make_request+0x97/0xd9
Jun 26 22:27:29 [368353.504480]  [<ffffffff81119bee>] submit_bio+0xdb/0xfa
Jun 26 22:27:29 [368353.504659]  [<ffffffffa01023e5>]
xfs_submit_ioend_bio.clone.19+0x2e/0x30 [xfs]
Jun 26 22:27:29 [368353.504967]  [<ffffffffa01024b5>]
xfs_submit_ioend+0xce/0xee [xfs]
Jun 26 22:27:29 [368353.505144]  [<ffffffffa0102859>]
xfs_vm_writepage+0x384/0x412 [xfs]
Jun 26 22:27:29 [368353.505316]  [<ffffffff8108a23e>] __writepage+0x12/0x2b
Jun 26 22:27:29 [368353.505485]  [<ffffffff8108aa5c>]
write_cache_pages+0x225/0x377
Jun 26 22:27:29 [368353.505656]  [<ffffffff8108a22c>] ? set_page_dirty+0x5b/0x5b
Jun 26 22:27:29 [368353.505826]  [<ffffffff8108abee>]
generic_writepages+0x40/0x57
Jun 26 22:27:29 [368353.506001]  [<ffffffffa0101c06>]
xfs_vm_writepages+0x47/0x4e [xfs]
Jun 26 22:27:29 [368353.506172]  [<ffffffff8108bbec>] do_writepages+0x1c/0x25
Jun 26 22:27:29 [368353.506342]  [<ffffffff810d1c17>]
writeback_single_inode+0xcf/0x232
Jun 26 22:27:29 [368353.506513]  [<ffffffff810d2015>]
writeback_sb_inodes+0x163/0x1fe
Jun 26 22:27:29 [368353.506684]  [<ffffffff810d240e>]
__writeback_inodes_wb+0x6d/0xab
Jun 26 22:27:29 [368353.506854]  [<ffffffff810d26dd>] wb_writeback+0xfc/0x182
Jun 26 22:27:29 [368353.507024]  [<ffffffff810c91b9>] ?
get_nr_inodes_unused+0x43/0x5a
Jun 26 22:27:29 [368353.507195]  [<ffffffff8125f1d9>] ?
_raw_spin_lock_irqsave+0xd/0x27
Jun 26 22:27:29 [368353.507365]  [<ffffffff810d2dbd>]
wb_do_writeback+0x111/0x16d
Jun 26 22:27:29 [368353.507537]  [<ffffffff81033322>] ?
run_timer_softirq+0x1ef/0x1ef
Jun 26 22:27:29 [368353.507707]  [<ffffffff810d2e9e>]
bdi_writeback_thread+0x85/0x14a
Jun 26 22:27:29 [368353.507877]  [<ffffffff810d2e19>] ?
wb_do_writeback+0x16d/0x16d
Jun 26 22:27:29 [368353.508047]  [<ffffffff810d2e19>] ?
wb_do_writeback+0x16d/0x16d
Jun 26 22:27:29 [368353.508217]  [<ffffffff8103fba9>] kthread+0x84/0x8c
Jun 26 22:27:29 [368353.508386]  [<ffffffff81260b54>]
kernel_thread_helper+0x4/0x10
Jun 26 22:27:29 [368353.508556]  [<ffffffff8103fb25>] ?
kthread_freezable_should_stop+0x41/0x41
Jun 26 22:27:29 [368353.508728]  [<ffffffff81260b50>] ? gs_change+0xb/0xb
Jun 26 22:27:29 [368353.508899] dhcpcd          D ffff88010b2d2ca0
0  6706      1 0x00000000
Jun 26 22:27:29 [368353.509131]  ffff88010c8c7828 0000000000000082
ffff88010c8c77b8 ffff88010dd1d940
Jun 26 22:27:29 [368353.509636]  0000000000000005 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.510056]  ffff88010b2d2ca0 00000000000107c0
ffff88010c8c7fd8 ffff88010c8c7fd8
Jun 26 22:27:29 [368353.510475] Call Trace:
Jun 26 22:27:29 [368353.510640]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.510811]  [<ffffffffa00b38ff>]
get_active_stripe+0x256/0x564 [raid456]
Jun 26 22:27:29 [368353.510982]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:29 [368353.511153]  [<ffffffffa00b5976>]
make_request+0x1dd/0x3bc [raid456]
Jun 26 22:27:29 [368353.511323]  [<ffffffff8103ffc0>] ? wake_up_bit+0x25/0x25
Jun 26 22:27:29 [368353.511496]  [<ffffffffa0069ef1>]
md_make_request+0xd0/0x1c2 [md_mod]
Jun 26 22:27:29 [368353.511666]  [<ffffffff811c6b93>] ? dm_request+0x1b4/0x1c3
Jun 26 22:27:29 [368353.511835]  [<ffffffff811c9c8c>] ? linear_merge+0x4d/0x58
Jun 26 22:27:29 [368353.512005]  [<ffffffff81119ad1>]
generic_make_request+0x97/0xd9
Jun 26 22:27:29 [368353.512176]  [<ffffffff81119bee>] submit_bio+0xdb/0xfa
Jun 26 22:27:29 [368353.512325]  [<ffffffffa01023e5>]
xfs_submit_ioend_bio.clone.19+0x2e/0x30 [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa01024b5>]
xfs_submit_ioend+0xce/0xee [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0102859>]
xfs_vm_writepage+0x384/0x412 [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffff8108a23e>] __writepage+0x12/0x2b
Jun 26 22:27:29 [368353.512325]  [<ffffffff8108aa5c>]
write_cache_pages+0x225/0x377
Jun 26 22:27:29 [368353.512325]  [<ffffffff8108a22c>] ? set_page_dirty+0x5b/0x5b
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0105bb2>] ?
xfs_rw_iunlock+0x24/0x29 [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa01067c5>] ?
xfs_file_buffered_aio_write+0xfd/0x110 [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffff8108abee>]
generic_writepages+0x40/0x57
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0101c06>]
xfs_vm_writepages+0x47/0x4e [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffff8108bbec>] do_writepages+0x1c/0x25
Jun 26 22:27:29 [368353.512325]  [<ffffffff81084a71>]
__filemap_fdatawrite_range+0x4e/0x50
Jun 26 22:27:29 [368353.512325]  [<ffffffff810855e3>]
filemap_fdatawrite_range+0xe/0x10
Jun 26 22:27:29 [368353.512325]  [<ffffffffa010822a>]
xfs_flush_pages+0x5c/0x8e [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0112ebd>]
xfs_release+0xe8/0x1ab [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0105eb4>]
xfs_file_release+0x10/0x14 [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffff810b729c>] fput+0xf9/0x1bd
Jun 26 22:27:29 [368353.512325]  [<ffffffff810b449a>] filp_close+0x6b/0x76
Jun 26 22:27:29 [368353.512325]  [<ffffffff810b4523>] sys_close+0x7e/0xbb
Jun 26 22:27:29 [368353.512325]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.512325] nfsd            D ffffffffffffffff
0  7164      2 0x00000000
Jun 26 22:27:29 [368353.512325]  ffff880107f99b18 0000000000000046
ffff88010a330d80 ffff88010dd1bfc0
Jun 26 22:27:29 [368353.512325]  ffffffff810c7402 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.512325]  ffff88010c8ea640 00000000000107c0
ffff880107f99fd8 ffff880107f99fd8
Jun 26 22:27:29 [368353.512325] Call Trace:
Jun 26 22:27:29 [368353.512325]  [<ffffffff810c7402>] ?
d_obtain_alias+0x142/0x14e
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0105975>] ?
xfs_dir_fsync+0x77/0x77 [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.512325]  [<ffffffff8125efa0>]
rwsem_down_failed_common+0xd7/0x109
Jun 26 22:27:29 [368353.512325]  [<ffffffff8125eff9>]
rwsem_down_read_failed+0x12/0x14
Jun 26 22:27:29 [368353.512325]  [<ffffffff81136a84>]
call_rwsem_down_read_failed+0x14/0x30
Jun 26 22:27:29 [368353.512325]  [<ffffffff810b6eb8>] ?
get_empty_filp+0x66/0x113
Jun 26 22:27:29 [368353.512325]  [<ffffffff8125de47>] ? down_read+0x12/0x14
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0108483>] xfs_ilock+0x59/0x5e [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa01084b8>]
xfs_ilock_map_shared+0x30/0x36 [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa010599e>]
xfs_dir_open+0x29/0x58 [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffff810b4899>]
__dentry_open.clone.14+0x149/0x263
Jun 26 22:27:29 [368353.512325]  [<ffffffff810b4a04>] dentry_open+0x51/0x5a
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0315c76>]
nfsd_open+0x11f/0x158 [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0316797>]
nfsd_readdir+0x43/0x1cf [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa031d750>] ?
nfs3svc_encode_entry+0xe/0xe [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa027b4b7>] ?
cache_check+0x1f4/0x216 [sunrpc]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa031b59c>]
nfsd3_proc_readdirplus+0x8b/0x16f [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa031279e>]
nfsd_dispatch+0xa5/0x15c [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0272b39>]
svc_process+0x3a5/0x567 [sunrpc]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa031205f>] ?
nfsd_get_default_max_blksize+0x3a/0x3a [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0312149>] nfsd+0xea/0x12e [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa031205f>] ?
nfsd_get_default_max_blksize+0x3a/0x3a [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffff8103fba9>] kthread+0x84/0x8c
Jun 26 22:27:29 [368353.512325]  [<ffffffff81260b54>]
kernel_thread_helper+0x4/0x10
Jun 26 22:27:29 [368353.512325]  [<ffffffff8103fb25>] ?
kthread_freezable_should_stop+0x41/0x41
Jun 26 22:27:29 [368353.512325]  [<ffffffff81260b50>] ? gs_change+0xb/0xb
Jun 26 22:27:29 [368353.512325] nfsd            D ffffffffffffffff
0  7166      2 0x00000000
Jun 26 22:27:29 [368353.512325]  ffff8801028a5b18 0000000000000046
ffff88010a330d80 ffff88010dd1bfc0
Jun 26 22:27:29 [368353.512325]  ffffffff810c7402 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.512325]  ffff88010c8e8660 00000000000107c0
ffff8801028a5fd8 ffff8801028a5fd8
Jun 26 22:27:29 [368353.512325] Call Trace:
Jun 26 22:27:29 [368353.512325]  [<ffffffff810c7402>] ?
d_obtain_alias+0x142/0x14e
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0105975>] ?
xfs_dir_fsync+0x77/0x77 [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.512325]  [<ffffffff8125efa0>]
rwsem_down_failed_common+0xd7/0x109
Jun 26 22:27:29 [368353.512325]  [<ffffffff8125eff9>]
rwsem_down_read_failed+0x12/0x14
Jun 26 22:27:29 [368353.512325]  [<ffffffff81136a84>]
call_rwsem_down_read_failed+0x14/0x30
Jun 26 22:27:29 [368353.512325]  [<ffffffff810b6eb8>] ?
get_empty_filp+0x66/0x113
Jun 26 22:27:29 [368353.512325]  [<ffffffff8125de47>] ? down_read+0x12/0x14
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0108483>] xfs_ilock+0x59/0x5e [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa01084b8>]
xfs_ilock_map_shared+0x30/0x36 [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa010599e>]
xfs_dir_open+0x29/0x58 [xfs]
Jun 26 22:27:29 [368353.512325]  [<ffffffff810b4899>]
__dentry_open.clone.14+0x149/0x263
Jun 26 22:27:29 [368353.512325]  [<ffffffff810b4a04>] dentry_open+0x51/0x5a
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0315c76>]
nfsd_open+0x11f/0x158 [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0316797>]
nfsd_readdir+0x43/0x1cf [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa031d750>] ?
nfs3svc_encode_entry+0xe/0xe [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa027b4b7>] ?
cache_check+0x1f4/0x216 [sunrpc]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa031b59c>]
nfsd3_proc_readdirplus+0x8b/0x16f [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa031279e>]
nfsd_dispatch+0xa5/0x15c [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0272b39>]
svc_process+0x3a5/0x567 [sunrpc]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa031205f>] ?
nfsd_get_default_max_blksize+0x3a/0x3a [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa0312149>] nfsd+0xea/0x12e [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffffa031205f>] ?
nfsd_get_default_max_blksize+0x3a/0x3a [nfsd]
Jun 26 22:27:29 [368353.512325]  [<ffffffff8103fba9>] kthread+0x84/0x8c
Jun 26 22:27:29 [368353.512325]  [<ffffffff81260b54>]
kernel_thread_helper+0x4/0x10
Jun 26 22:27:29 [368353.512325]  [<ffffffff8103fb25>] ?
kthread_freezable_should_stop+0x41/0x41
Jun 26 22:27:29 [368353.512325]  [<ffffffff81260b50>] ? gs_change+0xb/0xb
Jun 26 22:27:29 [368353.512325] agetty          D ffff880101e3bb18
0  7323      1 0x00000000
Jun 26 22:27:29 [368353.512325]  ffff880101e3ba68 0000000000000082
ffff880101e3b9e8 ffff88010dd1d940
Jun 26 22:27:29 [368353.530119]  ffff880101e3ba18 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010c8bb960 00000000000107c0
ffff880101e3bfd8 ffff880101e3bfd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108403a>] ? sleep_on_page+0xd/0xd
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e81e>] io_schedule+0x5e/0x79
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084043>]
sleep_on_page_killable+0x9/0x23
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125d7cb>]
__wait_on_bit_lock+0x41/0x8a
Jun 26 22:27:29 [368353.530119]  [<ffffffff81083f77>]
__lock_page_killable+0x61/0x68
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103fff4>] ?
autoremove_wake_function+0x34/0x34
Jun 26 22:27:29 [368353.530119]  [<ffffffff8113144a>] ?
radix_tree_lookup_slot+0xe/0x10
Jun 26 22:27:29 [368353.530119]  [<ffffffff81083f9f>]
lock_page_killable+0x21/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffff810853a3>]
generic_file_aio_read+0x454/0x61e
Jun 26 22:27:29 [368353.530119]  [<ffffffff8104ac69>] ? update_curr+0x42/0x79
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01062a1>]
xfs_file_aio_read+0x1a4/0x1fa [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b592c>] do_sync_read+0xc9/0x106
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0110fdf>] ?
xfs_xattr_get+0x37/0x40 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810cf77c>] ?
generic_getxattr+0x4e/0x5b
Jun 26 22:27:29 [368353.530119]  [<ffffffff811166b4>] ?
get_vfs_caps_from_disk+0x54/0xcd
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125f01c>] ? _raw_spin_lock+0x9/0xb
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b6032>] vfs_read+0xa4/0xf7
Jun 26 22:27:29 [368353.530119]  [<ffffffff810ba035>] kernel_read+0x3c/0x4c
Jun 26 22:27:29 [368353.530119]  [<ffffffff810ba126>] prepare_binprm+0xe1/0xe5
Jun 26 22:27:29 [368353.530119]  [<ffffffff810bb998>]
do_execve_common.clone.35+0x223/0x392
Jun 26 22:27:29 [368353.530119]  [<ffffffff810bbb1d>] do_execve+0x16/0x18
Jun 26 22:27:29 [368353.530119]  [<ffffffff81009c54>] sys_execve+0x3e/0x55
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fe2c>] stub_execve+0x6c/0xc0
Jun 26 22:27:29 [368353.530119] agetty          D ffff88010c8b8000
0  7324      1 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff880101e5b588 0000000000000086
ffff880101e5b518 ffff88010dd1f2c0
Jun 26 22:27:29 [368353.530119]  0000000000000400 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010c8b8000 00000000000107c0
ffff880101e5bfd8 ffff880101e5bfd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b38ff>]
get_active_stripe+0x256/0x564 [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b5976>]
make_request+0x1dd/0x3bc [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103ffc0>] ? wake_up_bit+0x25/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0069ef1>]
md_make_request+0xd0/0x1c2 [md_mod]
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c6b93>] ? dm_request+0x1b4/0x1c3
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119ad1>]
generic_make_request+0x97/0xd9
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119bee>] submit_bio+0xdb/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffff810defa8>]
do_mpage_readpage+0x443/0x4f7
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108cb59>] ?
__lru_cache_add+0x2f/0x56
Jun 26 22:27:29 [368353.530119]  [<ffffffff810df1aa>] mpage_readpages+0xe9/0x134
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101a00>] ?
xfs_get_blocks_direct+0x11/0x11 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01037a4>] ?
xfs_buf_rele+0x32/0x8c [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101a00>] ?
xfs_get_blocks_direct+0x11/0x11 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101bbd>]
xfs_vm_readpages+0x18/0x1a [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108be39>]
__do_page_cache_readahead+0x135/0x1d2
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108c14c>] ra_submit+0x1c/0x20
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108c307>]
ondemand_readahead+0x1b7/0x1c6
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c5f00>] ?
dentry_update_name_case+0x1c/0x56
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108c3cb>]
page_cache_sync_readahead+0x38/0x3d
Jun 26 22:27:29 [368353.530119]  [<ffffffff810851ec>]
generic_file_aio_read+0x29d/0x61e
Jun 26 22:27:29 [368353.530119]  [<ffffffff8104ac69>] ? update_curr+0x42/0x79
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01062a1>]
xfs_file_aio_read+0x1a4/0x1fa [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b592c>] do_sync_read+0xc9/0x106
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0110fdf>] ?
xfs_xattr_get+0x37/0x40 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810cf77c>] ?
generic_getxattr+0x4e/0x5b
Jun 26 22:27:29 [368353.530119]  [<ffffffff811166b4>] ?
get_vfs_caps_from_disk+0x54/0xcd
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125f01c>] ? _raw_spin_lock+0x9/0xb
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b6032>] vfs_read+0xa4/0xf7
Jun 26 22:27:29 [368353.530119]  [<ffffffff810ba035>] kernel_read+0x3c/0x4c
Jun 26 22:27:29 [368353.530119]  [<ffffffff810ba126>] prepare_binprm+0xe1/0xe5
Jun 26 22:27:29 [368353.530119]  [<ffffffff810bb998>]
do_execve_common.clone.35+0x223/0x392
Jun 26 22:27:29 [368353.530119]  [<ffffffff810bbb1d>] do_execve+0x16/0x18
Jun 26 22:27:29 [368353.530119]  [<ffffffff81009c54>] sys_execve+0x3e/0x55
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fe2c>] stub_execve+0x6c/0xc0
Jun 26 22:27:29 [368353.530119] irssi           D ffffffff8108402d
0  7343      1 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff880100487a38 0000000000000082
ffff8801004879b8 ffff88010c3c8cc0
Jun 26 22:27:29 [368353.530119]  ffff8801004879e8 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010c8bd940 00000000000107c0
ffff880100487fd8 ffff880100487fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108402d>] ? lock_page+0x22/0x22
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e81e>] io_schedule+0x5e/0x79
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084036>] sleep_on_page+0x9/0xd
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125d8cb>] __wait_on_bit+0x43/0x76
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0140b02>] ?
xfs_trans_free_item_desc+0x2e/0x30 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810841e3>] wait_on_page_bit+0x6d/0x74
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103fff4>] ?
autoremove_wake_function+0x34/0x34
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084109>] ? find_lock_page+0x32/0x56
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084200>]
wait_on_page_writeback+0x16/0x18
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108443a>]
grab_cache_page_write_begin+0xa9/0xbb
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d83b6>]
block_write_begin+0x27/0x77
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101a00>] ?
xfs_get_blocks_direct+0x11/0x11 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0102393>]
xfs_vm_write_begin+0x37/0x5b [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d197a>] ? inode_to_bdi+0x21/0x3f
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108370b>]
generic_file_buffered_write+0x105/0x255
Jun 26 22:27:29 [368353.530119]  [<ffffffff810cb70c>] ?
mnt_drop_write_file+0xd/0xf
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c89d8>] ?
file_update_time+0xd4/0xeb
Jun 26 22:27:29 [368353.530119]  [<ffffffffa010677b>]
xfs_file_buffered_aio_write+0xb3/0x110 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0106934>]
xfs_file_aio_write+0x15c/0x1b0 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8109e1ac>] ? vma_link+0x85/0xa7
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5826>] do_sync_write+0xc9/0x106
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5f3b>] vfs_write+0xa7/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b6133>] sys_write+0x45/0x69
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] flush-254:11    D ffff88010c8b9980
0  7345      2 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff8801004a1640 0000000000000046
ffff8801004a15d0 ffff88010c3cd2e0
Jun 26 22:27:29 [368353.530119]  ffff88010f0029c0 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010c8b9980 00000000000107c0
ffff8801004a1fd8 ffff8801004a1fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b38ff>]
get_active_stripe+0x256/0x564 [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b5976>]
make_request+0x1dd/0x3bc [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103ffc0>] ? wake_up_bit+0x25/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0069ef1>]
md_make_request+0xd0/0x1c2 [md_mod]
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c6b93>] ? dm_request+0x1b4/0x1c3
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c9c8c>] ? linear_merge+0x4d/0x58
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119ad1>]
generic_make_request+0x97/0xd9
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119bee>] submit_bio+0xdb/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01023e5>]
xfs_submit_ioend_bio.clone.19+0x2e/0x30 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01024b5>]
xfs_submit_ioend+0xce/0xee [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0102859>]
xfs_vm_writepage+0x384/0x412 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108a23e>] __writepage+0x12/0x2b
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108aa5c>]
write_cache_pages+0x225/0x377
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108a22c>] ? set_page_dirty+0x5b/0x5b
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0067408>] ?
plugger_unplug+0x38/0x3c [md_mod]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8111ba86>] ?
blk_flush_plug_list+0x94/0x1b1
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108abee>]
generic_writepages+0x40/0x57
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101c06>]
xfs_vm_writepages+0x47/0x4e [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108bbec>] do_writepages+0x1c/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d1c17>]
writeback_single_inode+0xcf/0x232
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d2015>]
writeback_sb_inodes+0x163/0x1fe
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d240e>]
__writeback_inodes_wb+0x6d/0xab
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d26dd>] wb_writeback+0xfc/0x182
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c91b9>] ?
get_nr_inodes_unused+0x43/0x5a
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125f1d9>] ?
_raw_spin_lock_irqsave+0xd/0x27
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d2dbd>]
wb_do_writeback+0x111/0x16d
Jun 26 22:27:29 [368353.530119]  [<ffffffff81033322>] ?
run_timer_softirq+0x1ef/0x1ef
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d2e9e>]
bdi_writeback_thread+0x85/0x14a
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d2e19>] ?
wb_do_writeback+0x16d/0x16d
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d2e19>] ?
wb_do_writeback+0x16d/0x16d
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103fba9>] kthread+0x84/0x8c
Jun 26 22:27:29 [368353.530119]  [<ffffffff81260b54>]
kernel_thread_helper+0x4/0x10
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103fb25>] ?
kthread_freezable_should_stop+0x41/0x41
Jun 26 22:27:29 [368353.530119]  [<ffffffff81260b50>] ? gs_change+0xb/0xb
Jun 26 22:27:29 [368353.530119] irssi           D ffffffff8108402d
0  7382      1 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff88007f8e3a38 0000000000000082
0000000000000004 ffff88010dd1bfc0
Jun 26 22:27:29 [368353.530119]  ffff88007f8e3a08 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010c8b8660 00000000000107c0
ffff88007f8e3fd8 ffff88007f8e3fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108402d>] ? lock_page+0x22/0x22
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e81e>] io_schedule+0x5e/0x79
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084036>] sleep_on_page+0x9/0xd
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125d8cb>] __wait_on_bit+0x43/0x76
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0140b02>] ?
xfs_trans_free_item_desc+0x2e/0x30 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810841e3>] wait_on_page_bit+0x6d/0x74
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103fff4>] ?
autoremove_wake_function+0x34/0x34
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084109>] ? find_lock_page+0x32/0x56
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084200>]
wait_on_page_writeback+0x16/0x18
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108443a>]
grab_cache_page_write_begin+0xa9/0xbb
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d83b6>]
block_write_begin+0x27/0x77
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101a00>] ?
xfs_get_blocks_direct+0x11/0x11 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0102393>]
xfs_vm_write_begin+0x37/0x5b [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d197a>] ? inode_to_bdi+0x21/0x3f
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108370b>]
generic_file_buffered_write+0x105/0x255
Jun 26 22:27:29 [368353.530119]  [<ffffffff810cb70c>] ?
mnt_drop_write_file+0xd/0xf
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c89d8>] ?
file_update_time+0xd4/0xeb
Jun 26 22:27:29 [368353.530119]  [<ffffffffa010677b>]
xfs_file_buffered_aio_write+0xb3/0x110 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0106934>]
xfs_file_aio_write+0x15c/0x1b0 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5826>] do_sync_write+0xc9/0x106
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5f3b>] vfs_write+0xa7/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b6133>] sys_write+0x45/0x69
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] irssi           D ffffffff8108402d
0  7383      1 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff88007f9f9a38 0000000000000082
0000000000000004 ffff88010dd1f2c0
Jun 26 22:27:29 [368353.530119]  ffff88007f9f9a08 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010db4aca0 00000000000107c0
ffff88007f9f9fd8 ffff88007f9f9fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108402d>] ? lock_page+0x22/0x22
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e81e>] io_schedule+0x5e/0x79
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084036>] sleep_on_page+0x9/0xd
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125d8cb>] __wait_on_bit+0x43/0x76
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0140b02>] ?
xfs_trans_free_item_desc+0x2e/0x30 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810841e3>] wait_on_page_bit+0x6d/0x74
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103fff4>] ?
autoremove_wake_function+0x34/0x34
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084109>] ? find_lock_page+0x32/0x56
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084200>]
wait_on_page_writeback+0x16/0x18
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108443a>]
grab_cache_page_write_begin+0xa9/0xbb
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d83b6>]
block_write_begin+0x27/0x77
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101a00>] ?
xfs_get_blocks_direct+0x11/0x11 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0102393>]
xfs_vm_write_begin+0x37/0x5b [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d197a>] ? inode_to_bdi+0x21/0x3f
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108370b>]
generic_file_buffered_write+0x105/0x255
Jun 26 22:27:29 [368353.530119]  [<ffffffff810cb70c>] ?
mnt_drop_write_file+0xd/0xf
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c89d8>] ?
file_update_time+0xd4/0xeb
Jun 26 22:27:29 [368353.530119]  [<ffffffffa010677b>]
xfs_file_buffered_aio_write+0xb3/0x110 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0106934>]
xfs_file_aio_write+0x15c/0x1b0 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5826>] do_sync_write+0xc9/0x106
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5f3b>] vfs_write+0xa7/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b6133>] sys_write+0x45/0x69
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] ping.py         D ffff88010c5fcc80
0  4347   4337 0x00000004
Jun 26 22:27:29 [368353.530119]  ffff88002c791758 0000000000000082
ffff88002c7916e8 ffff88010c3c8cc0
Jun 26 22:27:29 [368353.530119]  0000000000000400 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010c5fcc80 00000000000107c0
ffff88002c791fd8 ffff88002c791fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b38ff>]
get_active_stripe+0x256/0x564 [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b5976>]
make_request+0x1dd/0x3bc [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103ffc0>] ? wake_up_bit+0x25/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0069ef1>]
md_make_request+0xd0/0x1c2 [md_mod]
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c6b93>] ? dm_request+0x1b4/0x1c3
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c9c8c>] ? linear_merge+0x4d/0x58
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119ad1>]
generic_make_request+0x97/0xd9
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119bee>] submit_bio+0xdb/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01023e5>]
xfs_submit_ioend_bio.clone.19+0x2e/0x30 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01024b5>]
xfs_submit_ioend+0xce/0xee [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0102859>]
xfs_vm_writepage+0x384/0x412 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108a23e>] __writepage+0x12/0x2b
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108aa5c>]
write_cache_pages+0x225/0x377
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108a22c>] ? set_page_dirty+0x5b/0x5b
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0102338>] ?
xfs_vm_write_end+0x22/0x46 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810837e2>] ?
generic_file_buffered_write+0x1dc/0x255
Jun 26 22:27:29 [368353.530119]  [<ffffffff810cb70c>] ?
mnt_drop_write_file+0xd/0xf
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108abee>]
generic_writepages+0x40/0x57
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101c06>]
xfs_vm_writepages+0x47/0x4e [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108bbec>] do_writepages+0x1c/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084a71>]
__filemap_fdatawrite_range+0x4e/0x50
Jun 26 22:27:29 [368353.530119]  [<ffffffff810855e3>]
filemap_fdatawrite_range+0xe/0x10
Jun 26 22:27:29 [368353.530119]  [<ffffffffa010822a>]
xfs_flush_pages+0x5c/0x8e [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa010c6b0>]
xfs_setattr_size+0x135/0x348 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8102e526>] ?
current_fs_time+0x22/0x29
Jun 26 22:27:29 [368353.530119]  [<ffffffffa010c8de>]
xfs_vn_setattr+0x1b/0x2f [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810ca449>] notify_change+0x1a1/0x27a
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b4bac>] do_truncate+0x65/0x82
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5707>] ?
fsnotify_modify+0x5d/0x65
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b6dba>] ? fget+0x37/0x4c
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b4d7b>] sys_ftruncate+0xa4/0xa6
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] kworker/3:3     D ffff880113d907c0
0 29654      2 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff880106145cd0 0000000000000046
00000e3100000000 ffff880072938660
Jun 26 22:27:29 [368353.530119]  0000000081049200 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010b2d3960 00000000000107c0
ffff880106145fd8 ffff880106145fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffffa014210f>] xlog_wait+0x58/0x70 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01442a9>]
_xfs_log_force_lsn+0x1bf/0x1f5 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0141a69>]
xfs_trans_commit+0xdc/0x1d8 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa011007b>] ?
xfs_ifunlock+0x3f/0x3f [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0108011>]
xfs_fs_log_dummy+0x6f/0x76 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0143482>] ?
xfs_log_need_covered+0x52/0x94 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01100b9>]
xfs_sync_worker+0x3e/0x64 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103bbd0>]
process_one_work+0x161/0x22b
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103c724>] worker_thread+0xfe/0x182
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103c626>] ?
manage_workers.clone.22+0x16d/0x16d
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103c626>] ?
manage_workers.clone.22+0x16d/0x16d
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103fba9>] kthread+0x84/0x8c
Jun 26 22:27:29 [368353.530119]  [<ffffffff81260b54>]
kernel_thread_helper+0x4/0x10
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103fb25>] ?
kthread_freezable_should_stop+0x41/0x41
Jun 26 22:27:29 [368353.530119]  [<ffffffff81260b50>] ? gs_change+0xb/0xb
Jun 26 22:27:29 [368353.530119] kworker/0:0     D ffff88010cfa6c60
0  1339      2 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff880035d53cd0 0000000000000046
0000023100000000 ffff88010c3c8cc0
Jun 26 22:27:29 [368353.530119]  0000000081049200 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010cfa6c60 00000000000107c0
ffff880035d53fd8 ffff880035d53fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffffa014210f>] xlog_wait+0x58/0x70 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01442a9>]
_xfs_log_force_lsn+0x1bf/0x1f5 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0141a69>]
xfs_trans_commit+0xdc/0x1d8 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa011007b>] ?
xfs_ifunlock+0x3f/0x3f [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0108011>]
xfs_fs_log_dummy+0x6f/0x76 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0143493>] ?
xfs_log_need_covered+0x63/0x94 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01100b9>]
xfs_sync_worker+0x3e/0x64 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103bbd0>]
process_one_work+0x161/0x22b
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103c724>] worker_thread+0xfe/0x182
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103c626>] ?
manage_workers.clone.22+0x16d/0x16d
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103c626>] ?
manage_workers.clone.22+0x16d/0x16d
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103fba9>] kthread+0x84/0x8c
Jun 26 22:27:29 [368353.530119]  [<ffffffff81260b54>]
kernel_thread_helper+0x4/0x10
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103fb25>] ?
kthread_freezable_should_stop+0x41/0x41
Jun 26 22:27:29 [368353.530119]  [<ffffffff81260b50>] ? gs_change+0xb/0xb
Jun 26 22:27:29 [368353.530119] kworker/3:2     D ffff88010cfa0000
0  1968      2 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff8800e6f57ac0 0000000000000046
ffffea00040b7900 ffff88010c3cec60
Jun 26 22:27:29 [368353.530119]  ffff8800e6f57ae0 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010cfa0000 00000000000107c0
ffff8800e6f57fd8 ffff8800e6f57fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b38ff>]
get_active_stripe+0x256/0x564 [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b5976>]
make_request+0x1dd/0x3bc [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00d52a9>] ?
ata_scsi_em_message_show+0x2f/0x2f [libata]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103ffc0>] ? wake_up_bit+0x25/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0069ef1>]
md_make_request+0xd0/0x1c2 [md_mod]
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119a05>] ?
generic_make_request_checks+0x154/0x189
Jun 26 22:27:29 [368353.530119]  [<ffffffff810daed8>] ?
bio_alloc_bioset+0x71/0xbf
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c6ba2>] ? dm_request+0x1c3/0x1c3
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119ad1>]
generic_make_request+0x97/0xd9
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c5c7e>] __map_bio+0x4f/0xb0
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c67ae>]
__split_and_process_bio+0x246/0x477
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c5c13>] ? dec_pending+0x24c/0x268
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c6ba2>] ? dm_request+0x1c3/0x1c3
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c6c33>] dm_wq_work+0x91/0xb2
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103bbd0>]
process_one_work+0x161/0x22b
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103c724>] worker_thread+0xfe/0x182
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103c626>] ?
manage_workers.clone.22+0x16d/0x16d
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103c626>] ?
manage_workers.clone.22+0x16d/0x16d
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103fba9>] kthread+0x84/0x8c
Jun 26 22:27:29 [368353.530119]  [<ffffffff81260b54>]
kernel_thread_helper+0x4/0x10
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103fb25>] ?
kthread_freezable_should_stop+0x41/0x41
Jun 26 22:27:29 [368353.530119]  [<ffffffff81260b50>] ? gs_change+0xb/0xb
Jun 26 22:27:29 [368353.530119] cleanup         D ffff88010de5aca0
0  7088   7286 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff880038865868 0000000000000082
ffff8800388657f8 ffffffff81359020
Jun 26 22:27:29 [368353.530119]  0000000000000005 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010de5aca0 00000000000107c0
ffff880038865fd8 ffff880038865fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b38ff>]
get_active_stripe+0x256/0x564 [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b5976>]
make_request+0x1dd/0x3bc [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103ffc0>] ? wake_up_bit+0x25/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0069ef1>]
md_make_request+0xd0/0x1c2 [md_mod]
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c6b93>] ? dm_request+0x1b4/0x1c3
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c9c8c>] ? linear_merge+0x4d/0x58
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119ad1>]
generic_make_request+0x97/0xd9
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119bee>] submit_bio+0xdb/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01023e5>]
xfs_submit_ioend_bio.clone.19+0x2e/0x30 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01024b5>]
xfs_submit_ioend+0xce/0xee [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0102859>]
xfs_vm_writepage+0x384/0x412 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108a23e>] __writepage+0x12/0x2b
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108aa5c>]
write_cache_pages+0x225/0x377
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108a22c>] ? set_page_dirty+0x5b/0x5b
Jun 26 22:27:29 [368353.530119]  [<ffffffffa014d43a>] ?
xfs_get_acl+0x23/0x104 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa014d669>] ?
xfs_acl_chmod+0x2c/0x7b [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108abee>]
generic_writepages+0x40/0x57
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101c06>]
xfs_vm_writepages+0x47/0x4e [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108bbec>] do_writepages+0x1c/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084a71>]
__filemap_fdatawrite_range+0x4e/0x50
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084a9b>]
filemap_write_and_wait_range+0x28/0x51
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0105d97>]
xfs_file_fsync+0x3c/0x149 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d589e>] vfs_fsync_range+0x1e/0x20
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d58b7>] vfs_fsync+0x17/0x19
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d5923>] do_fsync+0x29/0x3e
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d5b52>] sys_fsync+0xb/0xf
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] deliver         D ffff88007293aca0
0  7095   7094 0x00000004
Jun 26 22:27:29 [368353.530119]  ffff880035d21868 0000000000000086
ffff880035d217f8 ffff88010dd1d940
Jun 26 22:27:29 [368353.530119]  0000000000000005 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88007293aca0 00000000000107c0
ffff880035d21fd8 ffff880035d21fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b38ff>]
get_active_stripe+0x256/0x564 [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b5976>]
make_request+0x1dd/0x3bc [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103ffc0>] ? wake_up_bit+0x25/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0069ef1>]
md_make_request+0xd0/0x1c2 [md_mod]
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c6b93>] ? dm_request+0x1b4/0x1c3
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c9c8c>] ? linear_merge+0x4d/0x58
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119ad1>]
generic_make_request+0x97/0xd9
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119bee>] submit_bio+0xdb/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01023e5>]
xfs_submit_ioend_bio.clone.19+0x2e/0x30 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01024b5>]
xfs_submit_ioend+0xce/0xee [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0102859>]
xfs_vm_writepage+0x384/0x412 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108a23e>] __writepage+0x12/0x2b
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108aa5c>]
write_cache_pages+0x225/0x377
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108a22c>] ? set_page_dirty+0x5b/0x5b
Jun 26 22:27:29 [368353.530119]  [<ffffffff8109afb4>] ?
handle_pte_fault+0x25d/0x615
Jun 26 22:27:29 [368353.530119]  [<ffffffffa010c517>] ?
xfs_setattr_nonsize+0x433/0x497 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108abee>]
generic_writepages+0x40/0x57
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101c06>]
xfs_vm_writepages+0x47/0x4e [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108bbec>] do_writepages+0x1c/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084a71>]
__filemap_fdatawrite_range+0x4e/0x50
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084a9b>]
filemap_write_and_wait_range+0x28/0x51
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0105d97>]
xfs_file_fsync+0x3c/0x149 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d589e>] vfs_fsync_range+0x1e/0x20
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d58b7>] vfs_fsync+0x17/0x19
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d5923>] do_fsync+0x29/0x3e
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d5b52>] sys_fsync+0xb/0xf
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] fetchmail       D ffff88010ca1ec60
0  7104   7099 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff8800388b36d8 0000000000000086
ffff88007293dfe8 ffff88010b2d4620
Jun 26 22:27:29 [368353.530119]  ffff8800388b3678 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010ca1ec60 00000000000107c0
ffff8800388b3fd8 ffff8800388b3fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125f1d9>] ?
_raw_spin_lock_irqsave+0xd/0x27
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125d686>] schedule_timeout+0x22/0xd5
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0114bd2>] ?
kmem_zone_alloc+0x38/0x82 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125defb>] __down_common+0x90/0xe5
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125dfaf>] __down+0x18/0x1a
Jun 26 22:27:29 [368353.530119]  [<ffffffff81043ef9>] down+0x28/0x38
Jun 26 22:27:29 [368353.530119]  [<ffffffffa010399f>]
xfs_buf_lock+0x37/0x3b [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0103b1b>]
_xfs_buf_find+0x178/0x1d6 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0103d0d>]
xfs_buf_get+0x25/0x13b [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0104019>]
xfs_buf_read+0x13/0x80 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01478b9>]
xfs_trans_read_buf+0x181/0x248 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa012b2ec>]
xfs_da_do_buf+0x2fe/0x518 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa012b558>]
xfs_da_read_buf+0x1c/0x1e [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103feed>] ? bit_waitqueue+0x12/0x92
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01309ed>]
xfs_dir2_leaf_addname+0x43/0x59e [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0121665>] ?
xfs_bmap_last_extent.clone.23+0x62/0x75 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa012d714>]
xfs_dir_createname+0x14f/0x16b [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0113723>]
xfs_create+0x31e/0x4a6 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa010bf55>]
xfs_vn_mknod+0xd1/0x153 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa010bff8>]
xfs_vn_create+0xe/0x10 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c05a5>] vfs_create+0x52/0x72
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c0d82>] do_last+0x220/0x573
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c17c3>] path_openat+0xcb/0x319
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c1aff>] do_filp_open+0x38/0x84
Jun 26 22:27:29 [368353.530119]  [<ffffffff810cb079>] ? alloc_fd+0x6f/0xfe
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b54bf>] do_sys_open+0x70/0x102
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b556d>] sys_open+0x1c/0x1e
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] munin-update    D ffffffff8108402d
0  7113   7105 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff88005ea15a38 0000000000000082
0000000000000018 ffffffff81359020
Jun 26 22:27:29 [368353.530119]  ffff88005ea15a28 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010de59320 00000000000107c0
ffff88005ea15fd8 ffff88005ea15fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108402d>] ? lock_page+0x22/0x22
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e81e>] io_schedule+0x5e/0x79
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084036>] sleep_on_page+0x9/0xd
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125d8cb>] __wait_on_bit+0x43/0x76
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0140b02>] ?
xfs_trans_free_item_desc+0x2e/0x30 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810841e3>] wait_on_page_bit+0x6d/0x74
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103fff4>] ?
autoremove_wake_function+0x34/0x34
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084109>] ? find_lock_page+0x32/0x56
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084200>]
wait_on_page_writeback+0x16/0x18
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108443a>]
grab_cache_page_write_begin+0xa9/0xbb
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d83b6>]
block_write_begin+0x27/0x77
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101a00>] ?
xfs_get_blocks_direct+0x11/0x11 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0102393>]
xfs_vm_write_begin+0x37/0x5b [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d197a>] ? inode_to_bdi+0x21/0x3f
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108370b>]
generic_file_buffered_write+0x105/0x255
Jun 26 22:27:29 [368353.530119]  [<ffffffff810cb70c>] ?
mnt_drop_write_file+0xd/0xf
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c89d8>] ?
file_update_time+0xd4/0xeb
Jun 26 22:27:29 [368353.530119]  [<ffffffffa010677b>]
xfs_file_buffered_aio_write+0xb3/0x110 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0106934>]
xfs_file_aio_write+0x15c/0x1b0 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5826>] do_sync_write+0xc9/0x106
Jun 26 22:27:29 [368353.530119]  [<ffffffff8109b5a5>] ?
handle_mm_fault+0xe1/0xf6
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5f3b>] vfs_write+0xa7/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b6133>] sys_write+0x45/0x69
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] fetchmail       D 0000000000000002
0  7144   7138 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff88007f8cbc78 0000000000000082
ffff88010ede7b40 ffffffff81359020
Jun 26 22:27:29 [368353.530119]  ffff88007f8cbd48 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88007293f920 00000000000107c0
ffff88007f8cbfd8 ffff88007f8cbfd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e95b>]
schedule_preempt_disabled+0x9/0xb
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125db87>]
__mutex_lock_common.clone.5+0x11d/0x17e
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125dbf6>]
__mutex_lock_slowpath+0xe/0x10
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125d988>] mutex_lock+0x16/0x27
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c0cf7>] do_last+0x195/0x573
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c17c3>] path_openat+0xcb/0x319
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c1aff>] do_filp_open+0x38/0x84
Jun 26 22:27:29 [368353.530119]  [<ffffffff810cb079>] ? alloc_fd+0x6f/0xfe
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b54bf>] do_sys_open+0x70/0x102
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b556d>] sys_open+0x1c/0x1e
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] munin-update    D 0000000000000002
0  7145   7141 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff8800389ddbe8 0000000000000086
ffff8800389ddbd8 ffff88010dd1d940
Jun 26 22:27:29 [368353.530119]  0000000000000002 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010b2d5fa0 00000000000107c0
ffff8800389ddfd8 ffff8800389ddfd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e95b>]
schedule_preempt_disabled+0x9/0xb
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125db87>]
__mutex_lock_common.clone.5+0x11d/0x17e
Jun 26 22:27:29 [368353.530119]  [<ffffffff810858d0>] ?
filemap_fault+0x1ee/0x34f
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125dbf6>]
__mutex_lock_slowpath+0xe/0x10
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125d988>] mutex_lock+0x16/0x27
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0105b7c>]
xfs_rw_ilock+0x21/0x33 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0106715>]
xfs_file_buffered_aio_write+0x4d/0x110 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0106934>]
xfs_file_aio_write+0x15c/0x1b0 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5826>] do_sync_write+0xc9/0x106
Jun 26 22:27:29 [368353.530119]  [<ffffffff8109b5a5>] ?
handle_mm_fault+0xe1/0xf6
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5f3b>] vfs_write+0xa7/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b6133>] sys_write+0x45/0x69
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] fetchmail       D 0000000000000002
0  7181   7175 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff880038901c78 0000000000000086
ffff88010ede7b40 ffff88007293c620
Jun 26 22:27:29 [368353.530119]  ffff880038901d48 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88007293bfc0 00000000000107c0
ffff880038901fd8 ffff880038901fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e95b>]
schedule_preempt_disabled+0x9/0xb
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125db87>]
__mutex_lock_common.clone.5+0x11d/0x17e
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125dbf6>]
__mutex_lock_slowpath+0xe/0x10
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125d988>] mutex_lock+0x16/0x27
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c0cf7>] do_last+0x195/0x573
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c17c3>] path_openat+0xcb/0x319
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c1aff>] do_filp_open+0x38/0x84
Jun 26 22:27:29 [368353.530119]  [<ffffffff810cb079>] ? alloc_fd+0x6f/0xfe
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b54bf>] do_sys_open+0x70/0x102
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b556d>] sys_open+0x1c/0x1e
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] munin-update    D 0000000000000002
0  7189   7183 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff880037793be8 0000000000000082
ffff880037793bd8 ffff88010dd1bfc0
Jun 26 22:27:29 [368353.530119]  0000000000000002 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010de59fe0 00000000000107c0
ffff880037793fd8 ffff880037793fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e95b>]
schedule_preempt_disabled+0x9/0xb
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125db87>]
__mutex_lock_common.clone.5+0x11d/0x17e
Jun 26 22:27:29 [368353.530119]  [<ffffffff810858d0>] ?
filemap_fault+0x1ee/0x34f
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125dbf6>]
__mutex_lock_slowpath+0xe/0x10
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125d988>] mutex_lock+0x16/0x27
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0105b7c>]
xfs_rw_ilock+0x21/0x33 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0106715>]
xfs_file_buffered_aio_write+0x4d/0x110 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0106934>]
xfs_file_aio_write+0x15c/0x1b0 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5826>] do_sync_write+0xc9/0x106
Jun 26 22:27:29 [368353.530119]  [<ffffffff8109b5a5>] ?
handle_mm_fault+0xe1/0xf6
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5f3b>] vfs_write+0xa7/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b6133>] sys_write+0x45/0x69
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] apache2         D ffff88010de5ec60
0  7208  17519 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff8800e440d778 0000000000000086
ffff8800e440d708 ffff88010dd1f2c0
Jun 26 22:27:29 [368353.530119]  0000000000000000 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010de5ec60 00000000000107c0
ffff8800e440dfd8 ffff8800e440dfd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b38ff>]
get_active_stripe+0x256/0x564 [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b5976>]
make_request+0x1dd/0x3bc [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103ffc0>] ? wake_up_bit+0x25/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0069ef1>]
md_make_request+0xd0/0x1c2 [md_mod]
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c6b93>] ? dm_request+0x1b4/0x1c3
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119ad1>]
generic_make_request+0x97/0xd9
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119bee>] submit_bio+0xdb/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108cb59>] ?
__lru_cache_add+0x2f/0x56
Jun 26 22:27:29 [368353.530119]  [<ffffffff810df1e1>]
mpage_readpages+0x120/0x134
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101a00>] ?
xfs_get_blocks_direct+0x11/0x11 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101a00>] ?
xfs_get_blocks_direct+0x11/0x11 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101bbd>]
xfs_vm_readpages+0x18/0x1a [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108be39>]
__do_page_cache_readahead+0x135/0x1d2
Jun 26 22:27:29 [368353.530119]  [<ffffffff810afc43>] ? signal_pending+0x12/0x1c
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108c14c>] ra_submit+0x1c/0x20
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108c307>]
ondemand_readahead+0x1b7/0x1c6
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108c3cb>]
page_cache_sync_readahead+0x38/0x3d
Jun 26 22:27:29 [368353.530119]  [<ffffffff810851ec>]
generic_file_aio_read+0x29d/0x61e
Jun 26 22:27:29 [368353.530119]  [<ffffffff810a239d>] ?
page_add_new_anon_rmap+0x5b/0x6e
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01062a1>]
xfs_file_aio_read+0x1a4/0x1fa [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b592c>] do_sync_read+0xc9/0x106
Jun 26 22:27:29 [368353.530119]  [<ffffffff8109b5a5>] ?
handle_mm_fault+0xe1/0xf6
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b6032>] vfs_read+0xa4/0xf7
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b60ca>] sys_read+0x45/0x69
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] cleanup         D ffff88010bd6e600
0  7210   7286 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff88003ac71868 0000000000000082
ffff88003ac717f8 ffff88010dd1bfc0
Jun 26 22:27:29 [368353.530119]  0000000000000005 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88010bd6e600 00000000000107c0
ffff88003ac71fd8 ffff88003ac71fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b38ff>]
get_active_stripe+0x256/0x564 [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b5976>]
make_request+0x1dd/0x3bc [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103ffc0>] ? wake_up_bit+0x25/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0069ef1>]
md_make_request+0xd0/0x1c2 [md_mod]
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c6b93>] ? dm_request+0x1b4/0x1c3
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c9c8c>] ? linear_merge+0x4d/0x58
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119ad1>]
generic_make_request+0x97/0xd9
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119bee>] submit_bio+0xdb/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01023e5>]
xfs_submit_ioend_bio.clone.19+0x2e/0x30 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01024b5>]
xfs_submit_ioend+0xce/0xee [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0102859>]
xfs_vm_writepage+0x384/0x412 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108a23e>] __writepage+0x12/0x2b
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108aa5c>]
write_cache_pages+0x225/0x377
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108a22c>] ? set_page_dirty+0x5b/0x5b
Jun 26 22:27:29 [368353.530119]  [<ffffffffa014d669>] ?
xfs_acl_chmod+0x2c/0x7b [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108abee>]
generic_writepages+0x40/0x57
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101c06>]
xfs_vm_writepages+0x47/0x4e [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108bbec>] do_writepages+0x1c/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084a71>]
__filemap_fdatawrite_range+0x4e/0x50
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084a9b>]
filemap_write_and_wait_range+0x28/0x51
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0105d97>]
xfs_file_fsync+0x3c/0x149 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d589e>] vfs_fsync_range+0x1e/0x20
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d58b7>] vfs_fsync+0x17/0x19
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d5923>] do_fsync+0x29/0x3e
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d5b52>] sys_fsync+0xb/0xf
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] local           D ffffffff8108402d
0  7213   7286 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff8800e6e17a38 0000000000000086
ffff8800e6e179b8 ffff88010dd1d940
Jun 26 22:27:29 [368353.530119]  ffff88010f002240 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff880072938cc0 00000000000107c0
ffff8800e6e17fd8 ffff8800e6e17fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108402d>] ? lock_page+0x22/0x22
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e81e>] io_schedule+0x5e/0x79
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084036>] sleep_on_page+0x9/0xd
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125d8cb>] __wait_on_bit+0x43/0x76
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0140b02>] ?
xfs_trans_free_item_desc+0x2e/0x30 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810841e3>] wait_on_page_bit+0x6d/0x74
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103fff4>] ?
autoremove_wake_function+0x34/0x34
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084109>] ? find_lock_page+0x32/0x56
Jun 26 22:27:29 [368353.530119]  [<ffffffff81084200>]
wait_on_page_writeback+0x16/0x18
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108443a>]
grab_cache_page_write_begin+0xa9/0xbb
Jun 26 22:27:29 [368353.530119]  [<ffffffff810d83b6>]
block_write_begin+0x27/0x77
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0101a00>] ?
xfs_get_blocks_direct+0x11/0x11 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0102393>]
xfs_vm_write_begin+0x37/0x5b [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8108370b>]
generic_file_buffered_write+0x105/0x255
Jun 26 22:27:29 [368353.530119]  [<ffffffff810cb70c>] ?
mnt_drop_write_file+0xd/0xf
Jun 26 22:27:29 [368353.530119]  [<ffffffff810c89d8>] ?
file_update_time+0xd4/0xeb
Jun 26 22:27:29 [368353.530119]  [<ffffffffa010677b>]
xfs_file_buffered_aio_write+0xb3/0x110 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0106934>]
xfs_file_aio_write+0x15c/0x1b0 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5826>] do_sync_write+0xc9/0x106
Jun 26 22:27:29 [368353.530119]  [<ffffffff8109b5a5>] ?
handle_mm_fault+0xe1/0xf6
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b5f3b>] vfs_write+0xa7/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffff810b6133>] sys_write+0x45/0x69
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:29 [368353.530119] sshd            D ffff88007293d2e0
0  7221      1 0x00000000
Jun 26 22:27:29 [368353.530119]  ffff880035da5868 0000000000000082
ffff880035da57f8 ffff88010dd1d940
Jun 26 22:27:29 [368353.530119]  ffff88010f0029c0 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:29 [368353.530119]  ffff88007293d2e0 00000000000107c0
ffff880035da5fd8 ffff880035da5fd8
Jun 26 22:27:29 [368353.530119] Call Trace:
Jun 26 22:27:29 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b38ff>]
get_active_stripe+0x256/0x564 [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:29 [368353.530119]  [<ffffffffa00b5976>]
make_request+0x1dd/0x3bc [raid456]
Jun 26 22:27:29 [368353.530119]  [<ffffffff8103ffc0>] ? wake_up_bit+0x25/0x25
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0069ef1>]
md_make_request+0xd0/0x1c2 [md_mod]
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c6b93>] ? dm_request+0x1b4/0x1c3
Jun 26 22:27:29 [368353.530119]  [<ffffffff811c9c8c>] ? linear_merge+0x4d/0x58
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119ad1>]
generic_make_request+0x97/0xd9
Jun 26 22:27:29 [368353.530119]  [<ffffffff81119bee>] submit_bio+0xdb/0xfa
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01023e5>]
xfs_submit_ioend_bio.clone.19+0x2e/0x30 [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa01024b5>]
xfs_submit_ioend+0xce/0xee [xfs]
Jun 26 22:27:29 [368353.530119]  [<ffffffffa0102859>]
xfs_vm_writepage+0x384/0x412 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffff8108a23e>] __writepage+0x12/0x2b
Jun 26 22:27:30 [368353.530119]  [<ffffffff8108aa5c>]
write_cache_pages+0x225/0x377
Jun 26 22:27:30 [368353.530119]  [<ffffffff8108a22c>] ? set_page_dirty+0x5b/0x5b
Jun 26 22:27:30 [368353.530119]  [<ffffffffa010696b>] ?
xfs_file_aio_write+0x193/0x1b0 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffff8108abee>]
generic_writepages+0x40/0x57
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0101c06>]
xfs_vm_writepages+0x47/0x4e [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffff8108bbec>] do_writepages+0x1c/0x25
Jun 26 22:27:30 [368353.530119]  [<ffffffff81084a71>]
__filemap_fdatawrite_range+0x4e/0x50
Jun 26 22:27:30 [368353.530119]  [<ffffffff81084a9b>]
filemap_write_and_wait_range+0x28/0x51
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0105d97>]
xfs_file_fsync+0x3c/0x149 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffff810b5707>] ?
fsnotify_modify+0x5d/0x65
Jun 26 22:27:30 [368353.530119]  [<ffffffff810d589e>] vfs_fsync_range+0x1e/0x20
Jun 26 22:27:30 [368353.530119]  [<ffffffff810d58b7>] vfs_fsync+0x17/0x19
Jun 26 22:27:30 [368353.530119]  [<ffffffff810d5923>] do_fsync+0x29/0x3e
Jun 26 22:27:30 [368353.530119]  [<ffffffff810d5b52>] sys_fsync+0xb/0xf
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:30 [368353.530119] bash            D fffffffeffffffff
0  7226   7225 0x00000000
Jun 26 22:27:30 [368353.530119]  ffff880073ebfb70 0000000000000082
4000000000000000 ffffffff81359020
Jun 26 22:27:30 [368353.530119]  0000000000000202 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:30 [368353.530119]  ffff88010bd6f920 00000000000107c0
ffff880073ebffd8 ffff880073ebffd8
Jun 26 22:27:30 [368353.530119] Call Trace:
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0105975>] ?
xfs_dir_fsync+0x77/0x77 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125efa0>]
rwsem_down_failed_common+0xd7/0x109
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125eff9>]
rwsem_down_read_failed+0x12/0x14
Jun 26 22:27:30 [368353.530119]  [<ffffffff81136a84>]
call_rwsem_down_read_failed+0x14/0x30
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125de47>] ? down_read+0x12/0x14
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0108483>] xfs_ilock+0x59/0x5e [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa01084b8>]
xfs_ilock_map_shared+0x30/0x36 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa010599e>]
xfs_dir_open+0x29/0x58 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffff810b4899>]
__dentry_open.clone.14+0x149/0x263
Jun 26 22:27:30 [368353.530119]  [<ffffffff810cbe68>] ? mntget+0x1b/0x21
Jun 26 22:27:30 [368353.530119]  [<ffffffff810b5448>]
nameidata_to_filp+0x43/0x4a
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c10c1>] do_last+0x55f/0x573
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c17c3>] path_openat+0xcb/0x319
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c1aff>] do_filp_open+0x38/0x84
Jun 26 22:27:30 [368353.530119]  [<ffffffff810cb079>] ? alloc_fd+0x6f/0xfe
Jun 26 22:27:30 [368353.530119]  [<ffffffff810b54bf>] do_sys_open+0x70/0x102
Jun 26 22:27:30 [368353.530119]  [<ffffffff810b557e>] sys_openat+0xf/0x11
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:30 [368353.530119] fetchmail       D 0000000000000002
0  7237   7231 0x00000000
Jun 26 22:27:30 [368353.530119]  ffff8800e6fedc78 0000000000000086
ffff88010ede7b40 ffffffff81359020
Jun 26 22:27:30 [368353.530119]  ffff8800e6fedd48 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:30 [368353.530119]  ffff880072939980 00000000000107c0
ffff8800e6fedfd8 ffff8800e6fedfd8
Jun 26 22:27:30 [368353.530119] Call Trace:
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125e95b>]
schedule_preempt_disabled+0x9/0xb
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125db87>]
__mutex_lock_common.clone.5+0x11d/0x17e
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125dbf6>]
__mutex_lock_slowpath+0xe/0x10
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125d988>] mutex_lock+0x16/0x27
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c0cf7>] do_last+0x195/0x573
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c17c3>] path_openat+0xcb/0x319
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c1aff>] do_filp_open+0x38/0x84
Jun 26 22:27:30 [368353.530119]  [<ffffffff810cb079>] ? alloc_fd+0x6f/0xfe
Jun 26 22:27:30 [368353.530119]  [<ffffffff810b54bf>] do_sys_open+0x70/0x102
Jun 26 22:27:30 [368353.530119]  [<ffffffff810b556d>] sys_open+0x1c/0x1e
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:30 [368353.530119] munin-update    D 0000000000000002
0  7238   7235 0x00000000
Jun 26 22:27:30 [368353.530119]  ffff88002c7fbbe8 0000000000000082
ffff88002c7fbbd8 ffff88010dd1d940
Jun 26 22:27:30 [368353.530119]  0000000000000002 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:30 [368353.530119]  ffff88010b2d0000 00000000000107c0
ffff88002c7fbfd8 ffff88002c7fbfd8
Jun 26 22:27:30 [368353.530119] Call Trace:
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125e95b>]
schedule_preempt_disabled+0x9/0xb
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125db87>]
__mutex_lock_common.clone.5+0x11d/0x17e
Jun 26 22:27:30 [368353.530119]  [<ffffffff810858d0>] ?
filemap_fault+0x1ee/0x34f
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125dbf6>]
__mutex_lock_slowpath+0xe/0x10
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125d988>] mutex_lock+0x16/0x27
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0105b7c>]
xfs_rw_ilock+0x21/0x33 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0106715>]
xfs_file_buffered_aio_write+0x4d/0x110 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0106934>]
xfs_file_aio_write+0x15c/0x1b0 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffff810b5826>] do_sync_write+0xc9/0x106
Jun 26 22:27:30 [368353.530119]  [<ffffffff8109b5a5>] ?
handle_mm_fault+0xe1/0xf6
Jun 26 22:27:30 [368353.530119]  [<ffffffff810b5f3b>] vfs_write+0xa7/0xfa
Jun 26 22:27:30 [368353.530119]  [<ffffffff810b6133>] sys_write+0x45/0x69
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:30 [368353.530119] ping.py         D ffff88010b2d3fc0
0  7248   7247 0x00000000
Jun 26 22:27:30 [368353.530119]  ffff880035cf1288 0000000000000082
0000000000000010 ffff88010dd1f2c0
Jun 26 22:27:30 [368353.530119]  0000000000000000 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:30 [368353.530119]  ffff88010b2d3fc0 00000000000107c0
ffff880035cf1fd8 ffff880035cf1fd8
Jun 26 22:27:30 [368353.530119] Call Trace:
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:30 [368353.530119]  [<ffffffffa00b38ff>]
get_active_stripe+0x256/0x564 [raid456]
Jun 26 22:27:30 [368353.530119]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:30 [368353.530119]  [<ffffffffa00b5976>]
make_request+0x1dd/0x3bc [raid456]
Jun 26 22:27:30 [368353.530119]  [<ffffffff8103ffc0>] ? wake_up_bit+0x25/0x25
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0069ef1>]
md_make_request+0xd0/0x1c2 [md_mod]
Jun 26 22:27:30 [368353.530119]  [<ffffffff811c6b93>] ? dm_request+0x1b4/0x1c3
Jun 26 22:27:30 [368353.530119]  [<ffffffff81119ad1>]
generic_make_request+0x97/0xd9
Jun 26 22:27:30 [368353.530119]  [<ffffffff81119bee>] submit_bio+0xdb/0xfa
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0102e32>]
_xfs_buf_ioapply+0x157/0x182 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0103f86>]
xfs_buf_iorequest+0x31/0x4a [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa01420b1>]
xlog_bdstrat+0x3c/0x42 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0142b4b>]
xlog_sync+0x1ea/0x2de [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0143583>]
xlog_state_release_iclog+0x8f/0x94 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0143fe5>]
_xfs_log_force+0xef/0x1be [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa01440c7>]
xfs_log_force+0x13/0x36 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0103960>]
xfs_buf_trylock+0x40/0x48 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0103a4c>]
_xfs_buf_find+0xa9/0x1d6 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0103d0d>]
xfs_buf_get+0x25/0x13b [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa01476ae>]
xfs_trans_get_buf+0x9b/0xc3 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffff81049200>] ?
try_to_wake_up+0x1ae/0x1ae
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0134403>]
xfs_ialloc_inode_init+0xd1/0x18b [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0134a13>]
xfs_ialloc_ag_alloc+0x2ef/0x446 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0134bd1>] ?
xfs_read_agi+0x67/0xe1 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0117c29>] ?
__xfs_alloc_vextent+0x388/0x388 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa013571c>]
xfs_dialloc+0x123/0x75a [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0137096>]
xfs_ialloc+0x53/0x4e2 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa01427db>] ?
xlog_grant_head_check+0x4b/0xbc [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa01119b8>]
xfs_dir_ialloc+0x6c/0x238 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa01136bb>]
xfs_create+0x2b6/0x4a6 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c5fe9>] ? __d_rehash+0x1e/0x54
Jun 26 22:27:30 [368353.530119]  [<ffffffffa010bf55>]
xfs_vn_mknod+0xd1/0x153 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa010bff8>]
xfs_vn_create+0xe/0x10 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c05a5>] vfs_create+0x52/0x72
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c0d82>] do_last+0x220/0x573
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c17c3>] path_openat+0xcb/0x319
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c1aff>] do_filp_open+0x38/0x84
Jun 26 22:27:30 [368353.530119]  [<ffffffff810cb079>] ? alloc_fd+0x6f/0xfe
Jun 26 22:27:30 [368353.530119]  [<ffffffff810b54bf>] do_sys_open+0x70/0x102
Jun 26 22:27:30 [368353.530119]  [<ffffffff810b556d>] sys_open+0x1c/0x1e
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
Jun 26 22:27:30 [368353.530119] cleanup         D ffff88010b2d1320
0  7250   7286 0x00000000
Jun 26 22:27:30 [368353.530119]  ffff88003779f658 0000000000000086
01ff880113c107c0 ffffffff81359020
Jun 26 22:27:30 [368353.530119]  ffff880113d0d450 00000000000107c0
00000000000107c0 00000000000107c0
Jun 26 22:27:30 [368353.530119]  ffff88010b2d1320 00000000000107c0
ffff88003779ffd8 ffff88003779ffd8
Jun 26 22:27:30 [368353.530119] Call Trace:
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125e7be>] schedule+0x5f/0x61
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125d686>] schedule_timeout+0x22/0xd5
Jun 26 22:27:30 [368353.530119]  [<ffffffff8104c1f7>] ?
update_group_power+0xd5/0x134
Jun 26 22:27:30 [368353.530119]  [<ffffffff8112df54>] ?
cpumask_next_and+0x2c/0x39
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125defb>] __down_common+0x90/0xe5
Jun 26 22:27:30 [368353.530119]  [<ffffffff810afc43>] ? signal_pending+0x12/0x1c
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125dfaf>] __down+0x18/0x1a
Jun 26 22:27:30 [368353.530119]  [<ffffffff81043ef9>] down+0x28/0x38
Jun 26 22:27:30 [368353.530119]  [<ffffffffa010399f>]
xfs_buf_lock+0x37/0x3b [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0103b1b>]
_xfs_buf_find+0x178/0x1d6 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0103d0d>]
xfs_buf_get+0x25/0x13b [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0104019>]
xfs_buf_read+0x13/0x80 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa01478b9>]
xfs_trans_read_buf+0x181/0x248 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0134bd1>]
xfs_read_agi+0x67/0xe1 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0134c64>]
xfs_ialloc_read_agi+0x19/0x66 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0135564>]
xfs_ialloc_ag_select+0x16d/0x202 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa013563c>]
xfs_dialloc+0x43/0x75a [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffff8104b47e>] ?
select_task_rq_fair+0x320/0x5ab
Jun 26 22:27:30 [368353.530119]  [<ffffffff810840d2>] ? unlock_page+0x25/0x2a
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125f132>] ?
_raw_read_unlock_bh+0x12/0x14
Jun 26 22:27:30 [368353.530119]  [<ffffffffa0137096>]
xfs_ialloc+0x53/0x4e2 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa01427db>] ?
xlog_grant_head_check+0x4b/0xbc [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa01119b8>]
xfs_dir_ialloc+0x6c/0x238 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa01136bb>]
xfs_create+0x2b6/0x4a6 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c5fe9>] ? __d_rehash+0x1e/0x54
Jun 26 22:27:30 [368353.530119]  [<ffffffffa010bf55>]
xfs_vn_mknod+0xd1/0x153 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffffa010bff8>]
xfs_vn_create+0xe/0x10 [xfs]
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c05a5>] vfs_create+0x52/0x72
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c0d82>] do_last+0x220/0x573
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c17c3>] path_openat+0xcb/0x319
Jun 26 22:27:30 [368353.530119]  [<ffffffff810c1aff>] do_filp_open+0x38/0x84
Jun 26 22:27:30 [368353.530119]  [<ffffffff810cb079>] ? alloc_fd+0x6f/0xfe
Jun 26 22:27:30 [368353.530119]  [<ffffffff810b54bf>] do_sys_open+0x70/0x102
Jun 26 22:27:30 [368353.530119]  [<ffffffff810b556d>] sys_open+0x1c/0x1e
Jun 26 22:27:30 [368353.530119]  [<ffffffff8125fa22>]
system_call_fastpath+0x16/0x1b
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [Linux Ext4 Filesystem]     [Union Filesystem]     [Filesystem Testing]     [Ceph Users]     [Ecryptfs]     [AutoFS]     [Kernel Newbies]     [Share Photos]     [Security]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux Cachefs]     [Reiser Filesystem]     [Linux RAID]     [Samba]     [Device Mapper]     [CEPH Development]
  Powered by Linux