Hello!
I have tried to google for similar problems, but have not found any
relevant or helpful information.
I have server with gluster-3.7.4 installed. Now it is configured as 3
pairs of 2 replicas on 6 local hard drives. In current setup it writes
log files from several network sources in directory structure like:
process/server/month/hour.log. Periodically other process traverses the
tree and compress relatively old files. The problem is that server is
hung up at different intervals - it can work for 2 weeks without a
problem or it can hung several days in a row. Looks like it bacame
IO-hung, because kernel responses to pings and to ssh port, but I can
not login to it or do anything else. I do not know how to properly debug
it. Can somebody help with it?
Originally bricks was located on ext4 filesystem, I have tried to change
it to xfs, but it does not helped. I have setup netconsole logging from
kernel, file is attached. Here is some additional information:
# gluster --version
glusterfs 3.7.4 built on Sep 19 2015 11:44:12
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU
General Public License.
# gluster volume info gv0
Volume Name: gv0
Type: Distributed-Replicate
Volume ID: b3167dd1-dbc1-48dd-8c8e-ca56a37f78a8
Status: Started
Number of Bricks: 3 x 2 = 6
Transport-type: tcp
Bricks:
Brick1: log0:/data/brick/b/gv0
Brick2: log0:/data/brick/a/gv0
Brick3: log0:/data/brick/d/gv0
Brick4: log0:/data/brick/c/gv0
Brick5: log0:/data/brick/e/gv0
Brick6: log0:/data/brick/f/gv0
Options Reconfigured:
performance.readdir-ahead: on
cluster.self-heal-daemon: enable
# gluster volume status gv0
Status of volume: gv0
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick log0:/data/brick/b/gv0 49163 0 Y
4385
Brick log0:/data/brick/a/gv0 49166 0 Y
4407
Brick log0:/data/brick/d/gv0 49164 0 Y
4397
Brick log0:/data/brick/c/gv0 49167 0 Y
4418
Brick log0:/data/brick/e/gv0 49165 0 Y
4379
Brick log0:/data/brick/f/gv0 49168 0 Y
4391
NFS Server on localhost N/A N/A N
N/A
Self-heal Daemon on localhost N/A N/A Y
4358
Task Status of Volume gv0
------------------------------------------------------------------------------
There are no active volume tasks
# uname -a
Linux log0 4.1.4-hardened #1 SMP Fri Aug 14 10:32:50 MSK 2015 x86_64
Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz GenuineIntel GNU/Linux
(it is gentoo-hardened kernel)
# cat /proc/mounts | grep /data/brick
/dev/sda4 /data/brick/a xfs rw,noatime,attr2,inode64,noquota 0 0
/dev/sdb4 /data/brick/b xfs rw,noatime,attr2,inode64,noquota 0 0
/dev/sdc4 /data/brick/c xfs rw,noatime,attr2,inode64,noquota 0 0
/dev/sdd4 /data/brick/d xfs rw,noatime,attr2,inode64,noquota 0 0
/dev/sde4 /data/brick/e xfs rw,noatime,attr2,inode64,noquota 0 0
/dev/sdf4 /data/brick/f xfs rw,noatime,attr2,inode64,noquota 0 0
/dev/sdg4 /data/brick/g xfs rw,noatime,attr2,inode64,noquota 0 0
I can provide additional information if needed.
Oct 2 20:11:33 logserver XFS (sdb4): Mounting V5 Filesystem
Oct 2 20:11:34 logserver XFS (sdb4): Ending clean mount
Oct 2 20:11:34 logserver XFS (sdd4): Mounting V5 Filesystem
Oct 2 20:11:34 logserver XFS (sdd4): Ending clean mount
Oct 2 20:11:34 logserver XFS (sde4): Mounting V5 Filesystem
Oct 2 20:11:34 logserver XFS (sde4): Ending clean mount
Oct 2 20:11:34 logserver XFS (sdg4): Mounting V5 Filesystem
Oct 2 20:11:34 logserver XFS (sdg4): Ending clean mount
Oct 6 08:14:21 logserver XFS (sda4): Mounting V5 Filesystem
Oct 6 08:14:21 logserver XFS (sda4): Ending clean mount
Oct 6 08:14:21 logserver XFS (sdc4): Mounting V5 Filesystem
Oct 6 08:14:21 logserver XFS (sdc4): Ending clean mount
Oct 6 08:14:21 logserver XFS (sdf4): Mounting V5 Filesystem
Oct 6 08:14:21 logserver XFS (sdf4): Ending clean mount
Oct 9 15:14:45 logserver perf interrupt took too long (2506 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
Oct 14 00:16:05 logserver INFO: task xfsaild/sde4:3551 blocked for more than 120 seconds.
Oct 14 00:16:05 logserver Not tainted 4.1.4-hardened #1
Oct 14 00:16:05 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 14 00:16:05 logserver xfsaild/sde4 D
Oct 14 00:16:05 logserver ffff8807f894fb08
Oct 14 00:16:05 logserver 0 3551 2 0x00000000
Oct 14 00:16:05 logserver ffff8807f894fb08
Oct 14 00:16:05 logserver 0000000000000000
Oct 14 00:16:05 logserver ffff8807f72a3720
Oct 14 00:16:05 logserver 0000000000000000
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffff8807f72a3ce0
Oct 14 00:16:05 logserver ffff8807f894fc48
Oct 14 00:16:05 logserver 7fffffffffffffff
Oct 14 00:16:05 logserver ffff8807f72a3720
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffffffffffffffff
Oct 14 00:16:05 logserver ffff8807f894fb28
Oct 14 00:16:05 logserver ffffffff815c3592
Oct 14 00:16:05 logserver ffff8807f894fb28
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver Call Trace:
Oct 14 00:16:05 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 14 00:16:05 logserver [<ffffffff815c6735>] schedule_timeout+0x155/0x1a0
Oct 14 00:16:05 logserver [<ffffffff8109cb7e>] ? load_balance+0x1be/0x8d0
Oct 14 00:16:05 logserver [<ffffffff815c3ff3>] wait_for_completion+0x123/0x180
Oct 14 00:16:05 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 14 00:16:05 logserver [<ffffffff8107d888>] flush_work+0xf8/0x1e0
Oct 14 00:16:05 logserver [<ffffffff8107a9a0>] ? destroy_worker+0xa0/0xa0
Oct 14 00:16:05 logserver [<ffffffff81090202>] ? cpuset_cpu_inactive+0xe2/0x120
Oct 14 00:16:05 logserver [<ffffffffc0a63858>] xlog_cil_force_lsn+0x88/0x210 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a61d1b>] _xfs_log_force+0x6b/0x280 [xfs]
Oct 14 00:16:05 logserver [<ffffffff810c3550>] ? migrate_timer_list+0x60/0x60
Oct 14 00:16:05 logserver [<ffffffffc0a61f43>] xfs_log_force+0x13/0x40 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a6bdcb>] xfsaild+0x13b/0x4c0 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a6bc90>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs]
Oct 14 00:16:05 logserver [<ffffffff81083273>] kthread+0xd3/0xf0
Oct 14 00:16:05 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 14 00:16:05 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 14 00:16:05 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 14 00:16:05 logserver INFO: task glusterfsd:4399 blocked for more than 120 seconds.
Oct 14 00:16:05 logserver Not tainted 4.1.4-hardened #1
Oct 14 00:16:05 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 14 00:16:05 logserver glusterfsd D
Oct 14 00:16:05 logserver ffff8807f16c3768
Oct 14 00:16:05 logserver 0 4399 1 0x00000000
Oct 14 00:16:05 logserver ffff8807f16c3768
Oct 14 00:16:05 logserver ffff88081fa52c80
Oct 14 00:16:05 logserver ffff8807f837d2b0
Oct 14 00:16:05 logserver ffff88081fa52c80
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffff8807f837d870
Oct 14 00:16:05 logserver ffff8807f16c38a8
Oct 14 00:16:05 logserver 7fffffffffffffff
Oct 14 00:16:05 logserver ffff8807f837d2b0
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffffffffffffffff
Oct 14 00:16:05 logserver ffff8807f16c3788
Oct 14 00:16:05 logserver ffffffff815c3592
Oct 14 00:16:05 logserver ffff8807f16c3798
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver Call Trace:
Oct 14 00:16:05 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 14 00:16:05 logserver [<ffffffff815c6735>] schedule_timeout+0x155/0x1a0
Oct 14 00:16:05 logserver [<ffffffff8108e0b3>] ? try_to_wake_up+0x1d3/0x300
Oct 14 00:16:05 logserver [<ffffffff815c3ff3>] wait_for_completion+0x123/0x180
Oct 14 00:16:05 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 14 00:16:05 logserver [<ffffffff8107d888>] flush_work+0xf8/0x1e0
Oct 14 00:16:05 logserver [<ffffffff8107a9a0>] ? destroy_worker+0xa0/0xa0
Oct 14 00:16:05 logserver [<ffffffffc0a30202>] ? xfs_ialloc_next_rec+0x62/0xd0 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a63858>] xlog_cil_force_lsn+0x88/0x210 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a10382>] ? xfs_bmap_search_multi_extents+0xb2/0x150 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a61d1b>] _xfs_log_force+0x6b/0x280 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a17312>] ? xfs_bmapi_read+0x202/0x300 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a61f43>] xfs_log_force+0x13/0x40 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a40a22>] xfs_buf_lock+0x42/0x60 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a40be5>] _xfs_buf_find+0x1a5/0x270 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a40cd5>] xfs_buf_get_map+0x25/0x130 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a6c9f1>] xfs_trans_get_buf_map+0xa1/0xd0 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a223e2>] xfs_da_get_buf+0xc2/0x100 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a28578>] xfs_dir3_data_init+0x58/0x1d0 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a2a446>] xfs_dir2_leaf_addname+0x5e6/0x790 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a198b3>] ? xfs_bmbt_get_all+0x13/0x20 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a256a9>] xfs_dir_createname+0x169/0x1d0 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a5394d>] xfs_link+0x1cd/0x2f0 [xfs]
Oct 14 00:16:05 logserver [<ffffffff8106b1ff>] ? ns_capable+0x3f/0x70
Oct 14 00:16:05 logserver [<ffffffffc0a4f771>] xfs_vn_link+0x61/0xa0 [xfs]
Oct 14 00:16:05 logserver [<ffffffff81189bb4>] vfs_link+0x1c4/0x2b0
Oct 14 00:16:05 logserver [<ffffffff811917b7>] SyS_linkat+0x2e7/0x390
Oct 14 00:16:05 logserver [<ffffffff815c7876>] system_call_fastpath+0x16/0x79
Oct 14 00:16:05 logserver INFO: task kworker/1:0:25667 blocked for more than 120 seconds.
Oct 14 00:16:05 logserver Not tainted 4.1.4-hardened #1
Oct 14 00:16:05 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 14 00:16:05 logserver kworker/1:0 D
Oct 14 00:16:05 logserver ffff8800976f3b68
Oct 14 00:16:05 logserver 0 25667 2 0x00000000
Oct 14 00:16:05 logserver Workqueue: xfs-cil/sdb4 xlog_cil_push_work [xfs]
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffff8800976f3b68
Oct 14 00:16:05 logserver ffff8800976f3b58
Oct 14 00:16:05 logserver ffff8807f8518000
Oct 14 00:16:05 logserver 0000000000000000
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffff8807f85185c0
Oct 14 00:16:05 logserver ffff8807fa125928
Oct 14 00:16:05 logserver ffff8805d18ff508
Oct 14 00:16:05 logserver ffff8807f8518000
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffff8807fa125900
Oct 14 00:16:05 logserver ffff8800976f3b88
Oct 14 00:16:05 logserver ffffffff815c3592
Oct 14 00:16:05 logserver ffff8807fa125928
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver Call Trace:
Oct 14 00:16:05 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 14 00:16:05 logserver [<ffffffffc0a6108b>] xlog_state_get_iclog_space+0xeb/0x300 [xfs]
Oct 14 00:16:05 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 14 00:16:05 logserver [<ffffffffc0a61581>] xlog_write+0x1d1/0x800 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a62f8f>] xlog_cil_push+0x23f/0x420 [xfs]
Oct 14 00:16:05 logserver [<ffffffff81017689>] ? sched_clock+0x9/0x10
Oct 14 00:16:05 logserver [<ffffffffc0a63180>] xlog_cil_push_work+0x10/0x20 [xfs]
Oct 14 00:16:05 logserver [<ffffffff8107ca16>] process_one_work+0x146/0x3d0
Oct 14 00:16:05 logserver [<ffffffff8107e1ac>] worker_thread+0x1bc/0x530
Oct 14 00:16:05 logserver [<ffffffff8107dff0>] ? rescuer_thread+0x3a0/0x3a0
Oct 14 00:16:05 logserver [<ffffffff81083273>] kthread+0xd3/0xf0
Oct 14 00:16:05 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 14 00:16:05 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 14 00:16:05 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 14 00:16:05 logserver INFO: task kworker/7:2:26453 blocked for more than 120 seconds.
Oct 14 00:16:05 logserver Not tainted 4.1.4-hardened #1
Oct 14 00:16:05 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 14 00:16:05 logserver kworker/7:2 D
Oct 14 00:16:05 logserver ffff880109fb7b68
Oct 14 00:16:05 logserver 0 26453 2 0x00000000
Oct 14 00:16:05 logserver Workqueue: xfs-cil/sde4 xlog_cil_push_work [xfs]
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffff880109fb7b68
Oct 14 00:16:05 logserver ffff880109fb7b58
Oct 14 00:16:05 logserver ffff88007faddbe0
Oct 14 00:16:05 logserver 0000000000000000
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffff88007fade1a0
Oct 14 00:16:05 logserver ffff8807fbd14d28
Oct 14 00:16:05 logserver ffff880385964678
Oct 14 00:16:05 logserver ffff88007faddbe0
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffff8807fbd14d00
Oct 14 00:16:05 logserver ffff880109fb7b88
Oct 14 00:16:05 logserver ffffffff815c3592
Oct 14 00:16:05 logserver ffff8807fbd14d28
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver Call Trace:
Oct 14 00:16:05 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 14 00:16:05 logserver [<ffffffffc0a6108b>] xlog_state_get_iclog_space+0xeb/0x300 [xfs]
Oct 14 00:16:05 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 14 00:16:05 logserver [<ffffffffc0a61581>] xlog_write+0x1d1/0x800 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a62f8f>] xlog_cil_push+0x23f/0x420 [xfs]
Oct 14 00:16:05 logserver [<ffffffff81017689>] ? sched_clock+0x9/0x10
Oct 14 00:16:05 logserver [<ffffffffc0a63180>] xlog_cil_push_work+0x10/0x20 [xfs]
Oct 14 00:16:05 logserver [<ffffffff8107ca16>] process_one_work+0x146/0x3d0
Oct 14 00:16:05 logserver [<ffffffff8107e1ac>] worker_thread+0x1bc/0x530
Oct 14 00:16:05 logserver [<ffffffff8107dff0>] ? rescuer_thread+0x3a0/0x3a0
Oct 14 00:16:05 logserver [<ffffffff81083273>] kthread+0xd3/0xf0
Oct 14 00:16:05 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 14 00:16:05 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 14 00:16:05 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 14 00:16:05 logserver INFO: task kworker/2:0H:3350 blocked for more than 120 seconds.
Oct 14 00:16:05 logserver Not tainted 4.1.4-hardened #1
Oct 14 00:16:05 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 14 00:16:05 logserver kworker/2:0H D
Oct 14 00:16:05 logserver ffff8801189e7ae8
Oct 14 00:16:05 logserver 0 3350 2 0x00000000
Oct 14 00:16:05 logserver Workqueue: xfs-log/sde4 xfs_log_worker [xfs]
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffff8801189e7ae8
Oct 14 00:16:05 logserver 0000000000000000
Oct 14 00:16:05 logserver ffff88007fad9b90
Oct 14 00:16:05 logserver 0000000000000000
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffff88007fada150
Oct 14 00:16:05 logserver ffff8801189e7c28
Oct 14 00:16:05 logserver 7fffffffffffffff
Oct 14 00:16:05 logserver ffff88007fad9b90
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffffffffffffffff
Oct 14 00:16:05 logserver ffff8801189e7b08
Oct 14 00:16:05 logserver ffffffff815c3592
Oct 14 00:16:05 logserver ffff88081fad2c80
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver Call Trace:
Oct 14 00:16:05 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 14 00:16:05 logserver [<ffffffff815c6735>] schedule_timeout+0x155/0x1a0
Oct 14 00:16:05 logserver [<ffffffff8109c4e2>] ? find_busiest_group+0x42/0x520
Oct 14 00:16:05 logserver [<ffffffff815c3ff3>] wait_for_completion+0x123/0x180
Oct 14 00:16:05 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 14 00:16:05 logserver [<ffffffff8107d888>] flush_work+0xf8/0x1e0
Oct 14 00:16:05 logserver [<ffffffff8107a9a0>] ? destroy_worker+0xa0/0xa0
Oct 14 00:16:05 logserver [<ffffffff81090202>] ? cpuset_cpu_inactive+0xe2/0x120
Oct 14 00:16:05 logserver [<ffffffffc0a63858>] xlog_cil_force_lsn+0x88/0x210 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a61d1b>] _xfs_log_force+0x6b/0x280 [xfs]
Oct 14 00:16:05 logserver [<ffffffff8109d7f0>] ? pick_next_task_fair+0x560/0x980
Oct 14 00:16:05 logserver [<ffffffff8100c70f>] ? __switch_to+0x28f/0x610
Oct 14 00:16:05 logserver [<ffffffffc0a61f43>] xfs_log_force+0x13/0x40 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a61f93>] xfs_log_worker+0x23/0x50 [xfs]
Oct 14 00:16:05 logserver [<ffffffff8107ca16>] process_one_work+0x146/0x3d0
Oct 14 00:16:05 logserver [<ffffffff8107e043>] worker_thread+0x53/0x530
Oct 14 00:16:05 logserver [<ffffffff8107dff0>] ? rescuer_thread+0x3a0/0x3a0
Oct 14 00:16:05 logserver [<ffffffff81083273>] kthread+0xd3/0xf0
Oct 14 00:16:05 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 14 00:16:05 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 14 00:16:05 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 14 00:16:05 logserver INFO: task kworker/1:0H:3724 blocked for more than 120 seconds.
Oct 14 00:16:05 logserver Not tainted 4.1.4-hardened #1
Oct 14 00:16:05 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 14 00:16:05 logserver kworker/1:0H D
Oct 14 00:16:05 logserver ffff8801024e3ae8
Oct 14 00:16:05 logserver 0 3724 2 0x00000000
Oct 14 00:16:05 logserver Workqueue: xfs-log/sdb4 xfs_log_worker [xfs]
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffff8801024e3ae8
Oct 14 00:16:05 logserver ffff8801024e3ac8
Oct 14 00:16:05 logserver ffff8807fb9e24c0
Oct 14 00:16:05 logserver ffff88081fa52c80
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffff8807fb9e2a80
Oct 14 00:16:05 logserver ffff8801024e3c28
Oct 14 00:16:05 logserver 7fffffffffffffff
Oct 14 00:16:05 logserver ffff8807fb9e24c0
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver ffffffffffffffff
Oct 14 00:16:05 logserver ffff8801024e3b08
Oct 14 00:16:05 logserver ffffffff815c3592
Oct 14 00:16:05 logserver ffff8801024e3b28
Oct 14 00:16:05 logserver
Oct 14 00:16:05 logserver Call Trace:
Oct 14 00:16:05 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 14 00:16:05 logserver [<ffffffff815c6735>] schedule_timeout+0x155/0x1a0
Oct 14 00:16:05 logserver [<ffffffff8108e0b3>] ? try_to_wake_up+0x1d3/0x300
Oct 14 00:16:05 logserver [<ffffffff815c3ff3>] wait_for_completion+0x123/0x180
Oct 14 00:16:05 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 14 00:16:05 logserver [<ffffffff8107d888>] flush_work+0xf8/0x1e0
Oct 14 00:16:05 logserver [<ffffffff8107a9a0>] ? destroy_worker+0xa0/0xa0
Oct 14 00:16:05 logserver [<ffffffffc0a63858>] xlog_cil_force_lsn+0x88/0x210 [xfs]
Oct 14 00:16:05 logserver [<ffffffff8109861f>] ? dequeue_entity+0x11f/0x6e0
Oct 14 00:16:05 logserver [<ffffffffc0a61d1b>] _xfs_log_force+0x6b/0x280 [xfs]
Oct 14 00:16:05 logserver [<ffffffff8109d46c>] ? pick_next_task_fair+0x1dc/0x980
Oct 14 00:16:05 logserver [<ffffffff8100c70f>] ? __switch_to+0x28f/0x610
Oct 14 00:16:05 logserver [<ffffffffc0a61f43>] xfs_log_force+0x13/0x40 [xfs]
Oct 14 00:16:05 logserver [<ffffffffc0a61f93>] xfs_log_worker+0x23/0x50 [xfs]
Oct 14 00:16:05 logserver [<ffffffff8107ca16>] process_one_work+0x146/0x3d0
Oct 14 00:16:05 logserver [<ffffffff8107e043>] worker_thread+0x53/0x530
Oct 14 00:16:05 logserver [<ffffffff8107dff0>] ? rescuer_thread+0x3a0/0x3a0
Oct 14 00:16:05 logserver [<ffffffff81083273>] kthread+0xd3/0xf0
Oct 14 00:16:05 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 14 00:16:05 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 14 00:16:05 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 14 00:18:05 logserver INFO: task kthreadd:2 blocked for more than 120 seconds.
Oct 14 00:18:05 logserver Not tainted 4.1.4-hardened #1
Oct 14 00:18:05 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 14 00:18:05 logserver kthreadd D
Oct 14 00:18:05 logserver ffff8807fc5ff428
Oct 14 00:18:05 logserver 0 2 0 0x00000000
Oct 14 00:18:05 logserver ffff8807fc5ff428
Oct 14 00:18:05 logserver ffff8807fc5ff408
Oct 14 00:18:05 logserver ffff8807fc5f0930
Oct 14 00:18:05 logserver ffff88081fbd2c80
Oct 14 00:18:05 logserver
Oct 14 00:18:05 logserver ffff8807fc5f0ef0
Oct 14 00:18:05 logserver ffff8807fc5ff568
Oct 14 00:18:05 logserver 7fffffffffffffff
Oct 14 00:18:05 logserver ffff8807fc5f0930
Oct 14 00:18:05 logserver
Oct 14 00:18:05 logserver ffffffffffffffff
Oct 14 00:18:05 logserver ffff8807fc5ff448
Oct 14 00:18:05 logserver ffffffff815c3592
Oct 14 00:18:05 logserver ffff8807fc5ff468
Oct 14 00:18:05 logserver
Oct 14 00:18:05 logserver Call Trace:
Oct 14 00:18:05 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 14 00:18:05 logserver [<ffffffff815c6735>] schedule_timeout+0x155/0x1a0
Oct 14 00:18:05 logserver [<ffffffff8108e0b3>] ? try_to_wake_up+0x1d3/0x300
Oct 14 00:18:05 logserver [<ffffffff815c3ff3>] wait_for_completion+0x123/0x180
Oct 14 00:18:05 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 14 00:18:05 logserver [<ffffffff8107d888>] flush_work+0xf8/0x1e0
Oct 14 00:18:05 logserver [<ffffffff8107a9a0>] ? destroy_worker+0xa0/0xa0
Oct 14 00:18:05 logserver [<ffffffffc0a63858>] xlog_cil_force_lsn+0x88/0x210 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a62028>] _xfs_log_force_lsn+0x68/0x2f0 [xfs]
Oct 14 00:18:05 logserver [<ffffffff81169f80>] ? __slab_free+0x190/0x260
Oct 14 00:18:05 logserver [<ffffffffc0a622c3>] xfs_log_force_lsn+0x13/0x40 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a5140c>] __xfs_iunpin_wait+0x8c/0x110 [xfs]
Oct 14 00:18:05 logserver [<ffffffff810a3e10>] ? autoremove_wake_function+0x40/0x40
Oct 14 00:18:05 logserver [<ffffffffc0a54b69>] xfs_iunpin_wait+0x19/0x20 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a4842c>] xfs_reclaim_inode+0x7c/0x340 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a48923>] xfs_reclaim_inodes_ag+0x233/0x360 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a4929e>] xfs_reclaim_inodes_nr+0x2e/0x40 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a59394>] xfs_fs_free_cached_objects+0x14/0x20 [xfs]
Oct 14 00:18:05 logserver [<ffffffff8118095b>] super_cache_scan+0x19b/0x1a0
Oct 14 00:18:05 logserver [<ffffffff81127c72>] shrink_slab.part.48.constprop.60+0x1a2/0x270
Oct 14 00:18:05 logserver [<ffffffff8112a745>] shrink_zone+0x175/0x1b0
Oct 14 00:18:05 logserver [<ffffffff8112a93c>] do_try_to_free_pages+0x1bc/0x3b0
Oct 14 00:18:05 logserver [<ffffffff8112abce>] try_to_free_pages+0x9e/0xd0
Oct 14 00:18:05 logserver [<ffffffff81120c0b>] __alloc_pages_nodemask+0x53b/0x960
Oct 14 00:18:05 logserver [<ffffffff81121076>] alloc_kmem_pages_node+0x46/0x60
Oct 14 00:18:05 logserver [<ffffffff81060fe8>] copy_process.part.36+0x128/0x17a0
Oct 14 00:18:05 logserver [<ffffffff81017689>] ? sched_clock+0x9/0x10
Oct 14 00:18:05 logserver [<ffffffff810916ed>] ? sched_clock_cpu+0x9d/0xb0
Oct 14 00:18:05 logserver [<ffffffff81099f8c>] ? put_prev_entity+0x2c/0x4a0
Oct 14 00:18:05 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 14 00:18:05 logserver [<ffffffff8100c70f>] ? __switch_to+0x28f/0x610
Oct 14 00:18:05 logserver [<ffffffff81062817>] do_fork+0xd7/0x330
Oct 14 00:18:05 logserver [<ffffffff815c2fcc>] ? __schedule+0x29c/0x830
Oct 14 00:18:05 logserver [<ffffffff81062a91>] kernel_thread+0x21/0x30
Oct 14 00:18:05 logserver [<ffffffff81083be4>] kthreadd+0x144/0x1a0
Oct 14 00:18:05 logserver [<ffffffff81083aa0>] ? kthread_create_on_cpu+0x60/0x60
Oct 14 00:18:05 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 14 00:18:05 logserver [<ffffffff81083aa0>] ? kthread_create_on_cpu+0x60/0x60
Oct 14 00:18:05 logserver INFO: task xfsaild/sde4:3551 blocked for more than 120 seconds.
Oct 14 00:18:05 logserver Not tainted 4.1.4-hardened #1
Oct 14 00:18:05 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 14 00:18:05 logserver xfsaild/sde4 D
Oct 14 00:18:05 logserver ffff8807f894fb08
Oct 14 00:18:05 logserver 0 3551 2 0x00000000
Oct 14 00:18:05 logserver ffff8807f894fb08
Oct 14 00:18:05 logserver 0000000000000000
Oct 14 00:18:05 logserver ffff8807f72a3720
Oct 14 00:18:05 logserver 0000000000000000
Oct 14 00:18:05 logserver
Oct 14 00:18:05 logserver ffff8807f72a3ce0
Oct 14 00:18:05 logserver ffff8807f894fc48
Oct 14 00:18:05 logserver 7fffffffffffffff
Oct 14 00:18:05 logserver ffff8807f72a3720
Oct 14 00:18:05 logserver
Oct 14 00:18:05 logserver ffffffffffffffff
Oct 14 00:18:05 logserver ffff8807f894fb28
Oct 14 00:18:05 logserver ffffffff815c3592
Oct 14 00:18:05 logserver ffff8807f894fb28
Oct 14 00:18:05 logserver
Oct 14 00:18:05 logserver Call Trace:
Oct 14 00:18:05 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 14 00:18:05 logserver [<ffffffff815c6735>] schedule_timeout+0x155/0x1a0
Oct 14 00:18:05 logserver [<ffffffff8109cb7e>] ? load_balance+0x1be/0x8d0
Oct 14 00:18:05 logserver [<ffffffff815c3ff3>] wait_for_completion+0x123/0x180
Oct 14 00:18:05 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 14 00:18:05 logserver [<ffffffff8107d888>] flush_work+0xf8/0x1e0
Oct 14 00:18:05 logserver [<ffffffff8107a9a0>] ? destroy_worker+0xa0/0xa0
Oct 14 00:18:05 logserver [<ffffffff81090202>] ? cpuset_cpu_inactive+0xe2/0x120
Oct 14 00:18:05 logserver [<ffffffffc0a63858>] xlog_cil_force_lsn+0x88/0x210 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a61d1b>] _xfs_log_force+0x6b/0x280 [xfs]
Oct 14 00:18:05 logserver [<ffffffff810c3550>] ? migrate_timer_list+0x60/0x60
Oct 14 00:18:05 logserver [<ffffffffc0a61f43>] xfs_log_force+0x13/0x40 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a6bdcb>] xfsaild+0x13b/0x4c0 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a6bc90>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs]
Oct 14 00:18:05 logserver [<ffffffff81083273>] kthread+0xd3/0xf0
Oct 14 00:18:05 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 14 00:18:05 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 14 00:18:05 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 14 00:18:05 logserver INFO: task glusterfsd:4695 blocked for more than 120 seconds.
Oct 14 00:18:05 logserver Not tainted 4.1.4-hardened #1
Oct 14 00:18:05 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 14 00:18:05 logserver glusterfsd D
Oct 14 00:18:05 logserver ffff88068bd0b5b8
Oct 14 00:18:05 logserver 0 4695 1 0x00000000
Oct 14 00:18:05 logserver ffff88068bd0b5b8
Oct 14 00:18:05 logserver 0000000000000006
Oct 14 00:18:05 logserver ffff8807fbda52b0
Oct 14 00:18:05 logserver 0000000000000006
Oct 14 00:18:05 logserver
Oct 14 00:18:05 logserver ffff8807fbda5870
Oct 14 00:18:05 logserver 7fffffffffffffff
Oct 14 00:18:05 logserver ffff8807fbda52b0
Oct 14 00:18:05 logserver 0000000000000002
Oct 14 00:18:05 logserver
Oct 14 00:18:05 logserver ffff8807f93bb540
Oct 14 00:18:05 logserver ffff88068bd0b5d8
Oct 14 00:18:05 logserver ffffffff815c3592
Oct 14 00:18:05 logserver ffff88068bd0b5e8
Oct 14 00:18:05 logserver
Oct 14 00:18:05 logserver Call Trace:
Oct 14 00:18:05 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 14 00:18:05 logserver [<ffffffff815c6735>] schedule_timeout+0x155/0x1a0
Oct 14 00:18:05 logserver [<ffffffffc0a34371>] ? xfs_iext_bno_to_ext+0xb1/0x1d0 [xfs]
Oct 14 00:18:05 logserver [<ffffffff815c5777>] __down+0x77/0xe0
Oct 14 00:18:05 logserver [<ffffffff810a807c>] down+0x3c/0x50
Oct 14 00:18:05 logserver [<ffffffffc0a40a05>] xfs_buf_lock+0x25/0x60 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a40be5>] _xfs_buf_find+0x1a5/0x270 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a40cd5>] xfs_buf_get_map+0x25/0x130 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a34371>] ? xfs_iext_bno_to_ext+0xb1/0x1d0 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a78150>] ? exit_xfs_fs+0x102c/0x19fe4 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a41324>] xfs_buf_read_map+0x24/0xc0 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a6cb38>] xfs_trans_read_buf_map+0x98/0x1b0 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a32080>] xfs_read_agi+0x80/0xb0 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a78150>] ? exit_xfs_fs+0x102c/0x19fe4 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a320cb>] xfs_ialloc_read_agi+0x1b/0x80 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a32b61>] xfs_dialloc+0x171/0x280 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a52946>] xfs_ialloc+0x66/0x540 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a52e94>] xfs_dir_ialloc+0x74/0x290 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a6269e>] ? xfs_log_reserve+0x11e/0x130 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a5d2e5>] ? xfs_trans_reserve+0x1b5/0x1f0 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a5cfa5>] ? _xfs_trans_alloc+0x35/0xa0 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a53365>] xfs_create+0x265/0x680 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a4fdb7>] xfs_generic_create+0xd7/0x2e0 [xfs]
Oct 14 00:18:05 logserver [<ffffffff8119a68d>] ? __d_lookup+0x11d/0x160
Oct 14 00:18:05 logserver [<ffffffffc0a4ffef>] xfs_vn_mknod+0xf/0x20 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a5002e>] xfs_vn_create+0xe/0x10 [xfs]
Oct 14 00:18:05 logserver [<ffffffff81189e8d>] vfs_create+0xcd/0x140
Oct 14 00:18:05 logserver [<ffffffff8118b987>] do_last+0x1377/0x1490
Oct 14 00:18:05 logserver [<ffffffff8118e66d>] path_openat+0x8d/0x670
Oct 14 00:18:05 logserver [<ffffffff811907b9>] ? user_path_at_empty+0x69/0xd0
Oct 14 00:18:05 logserver [<ffffffff811908b4>] do_filp_open+0x44/0xc0
Oct 14 00:18:05 logserver [<ffffffff812eecda>] ? find_next_zero_bit+0x1a/0x30
Oct 14 00:18:05 logserver [<ffffffff8119ec0f>] ? __alloc_fd+0x8f/0x120
Oct 14 00:18:05 logserver [<ffffffff8117bb76>] do_sys_open+0x136/0x250
Oct 14 00:18:05 logserver [<ffffffff8117bca9>] SyS_open+0x19/0x20
Oct 14 00:18:05 logserver [<ffffffff815c7876>] system_call_fastpath+0x16/0x79
Oct 14 00:18:05 logserver INFO: task glusterfsd:6105 blocked for more than 120 seconds.
Oct 14 00:18:05 logserver Not tainted 4.1.4-hardened #1
Oct 14 00:18:05 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 14 00:18:05 logserver glusterfsd D
Oct 14 00:18:05 logserver ffff8805e529baa0
Oct 14 00:18:05 logserver 0 6105 1 0x00000000
Oct 14 00:18:05 logserver ffff8805e529baa0
Oct 14 00:18:05 logserver ffff8807f172cd80
Oct 14 00:18:05 logserver ffff8807fa150930
Oct 14 00:18:05 logserver ffffffff8153aaac
Oct 14 00:18:05 logserver
Oct 14 00:18:05 logserver ffff8807fa150ef0
Oct 14 00:18:05 logserver ffff880109279090
Oct 14 00:18:05 logserver ffff8801092790a8
Oct 14 00:18:05 logserver fffffffeffffffff
Oct 14 00:18:05 logserver
Oct 14 00:18:05 logserver ffff8807f7f19060
Oct 14 00:18:05 logserver ffff8805e529bac0
Oct 14 00:18:05 logserver ffffffff815c3592
Oct 14 00:18:05 logserver ffff880109279090
Oct 14 00:18:05 logserver
Oct 14 00:18:05 logserver Call Trace:
Oct 14 00:18:05 logserver [<ffffffff8153aaac>] ? ip_local_out_sk+0x2c/0x40
Oct 14 00:18:05 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 14 00:18:05 logserver [<ffffffff815c6535>] rwsem_down_read_failed+0xe5/0x170
Oct 14 00:18:05 logserver [<ffffffff8109861f>] ? dequeue_entity+0x11f/0x6e0
Oct 14 00:18:05 logserver [<ffffffff812e7964>] call_rwsem_down_read_failed+0x14/0x30
Oct 14 00:18:05 logserver [<ffffffff815c5b72>] ? down_read+0x12/0x20
Oct 14 00:18:05 logserver [<ffffffffc0a517f5>] xfs_ilock+0x45/0xc0 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a518e3>] xfs_ilock_attr_map_shared+0x33/0x40 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a0a424>] xfs_attr_get+0xd4/0x1b0 [xfs]
Oct 14 00:18:05 logserver [<ffffffffc0a5e367>] xfs_xattr_get+0x67/0xb0 [xfs]
Oct 14 00:18:05 logserver [<ffffffff811a5fe7>] generic_getxattr+0x47/0x70
Oct 14 00:18:05 logserver [<ffffffff811a6495>] vfs_getxattr+0x95/0xc0
Oct 14 00:18:05 logserver [<ffffffff811a6810>] getxattr+0xa0/0x210
Oct 14 00:18:05 logserver [<ffffffff8118f217>] ? putname+0x67/0x70
Oct 14 00:18:05 logserver [<ffffffff811907b9>] ? user_path_at_empty+0x69/0xd0
Oct 14 00:18:05 logserver [<ffffffff811a69dd>] path_getxattr+0x5d/0xa0
Oct 14 00:18:05 logserver [<ffffffff811a76dc>] SyS_lgetxattr+0xc/0x10
Oct 14 00:18:05 logserver [<ffffffff815c7876>] system_call_fastpath+0x16/0x79
Oct 19 18:23:05 logserver perf interrupt took too long (2512 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
Oct 20 06:09:00 logserver INFO: task glusterfsd:4538 blocked for more than 120 seconds.
Oct 20 06:09:00 logserver Not tainted 4.1.4-hardened #1
Oct 20 06:09:00 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 20 06:09:00 logserver glusterfsd D
Oct 20 06:09:00 logserver ffff8800c81ef788
Oct 20 06:09:00 logserver 0 4538 1 0x00000000
Oct 20 06:09:00 logserver ffff8800c81ef788
Oct 20 06:09:00 logserver ffff88081fddbd80
Oct 20 06:09:00 logserver ffff8807f857c980
Oct 20 06:09:00 logserver ffff8805eda15308
Oct 20 06:09:00 logserver
Oct 20 06:09:00 logserver ffff8807f857cf40
Oct 20 06:09:00 logserver 7fffffffffffffff
Oct 20 06:09:00 logserver ffff8807f857c980
Oct 20 06:09:00 logserver 0000000000000002
Oct 20 06:09:00 logserver
Oct 20 06:09:00 logserver ffff8807f83f4780
Oct 20 06:09:00 logserver ffff8800c81ef7a8
Oct 20 06:09:00 logserver ffffffff815c3592
Oct 20 06:09:00 logserver 0000000000000000
Oct 20 06:09:00 logserver
Oct 20 06:09:00 logserver Call Trace:
Oct 20 06:09:00 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 20 06:09:00 logserver [<ffffffff815c6735>] schedule_timeout+0x155/0x1a0
Oct 20 06:09:00 logserver [<ffffffff81169a12>] ? kmem_cache_alloc+0xb2/0xc0
Oct 20 06:09:00 logserver [<ffffffff815c5777>] __down+0x77/0xe0
Oct 20 06:09:00 logserver [<ffffffff810a807c>] down+0x3c/0x50
Oct 20 06:09:00 logserver [<ffffffffc0a40a05>] xfs_buf_lock+0x25/0x60 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a40be5>] _xfs_buf_find+0x1a5/0x270 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a40cd5>] xfs_buf_get_map+0x25/0x130 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a771e0>] ? exit_xfs_fs+0xbc/0x19fe4 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a41324>] xfs_buf_read_map+0x24/0xc0 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a1a0a7>] ? xfs_lookup_get_search_key+0x47/0x60 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a6cb38>] xfs_trans_read_buf_map+0x98/0x1b0 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a07e42>] xfs_read_agf+0x72/0xa0 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a771e0>] ? exit_xfs_fs+0xbc/0x19fe4 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a07e96>] xfs_alloc_read_agf+0x26/0xb0 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a083d4>] xfs_alloc_fix_freelist+0x454/0x4e0 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a5cadd>] ? xfs_trans_free_item_desc+0x2d/0x40 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a5db20>] ? xfs_trans_free_items+0x80/0xb0 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a636ba>] ? xfs_log_commit_cil+0x42a/0x540 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a089a6>] xfs_free_extent+0x96/0x130 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a3d51f>] xfs_bmap_finish+0x15f/0x1b0 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a53be4>] xfs_itruncate_extents+0x174/0x230 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a78ec0>] ? exit_xfs_fs+0x1d9c/0x19fe4 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a78ec0>] ? exit_xfs_fs+0x1d9c/0x19fe4 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a53d36>] xfs_inactive_truncate+0x96/0x120 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a54b28>] xfs_inactive+0x118/0x140 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a59c1d>] xfs_fs_evict_inode+0x5d/0x70 [xfs]
Oct 20 06:09:00 logserver [<ffffffff8119c696>] evict+0xa6/0x190
Oct 20 06:09:00 logserver [<ffffffff8119cf4a>] iput+0x14a/0x1b0
Oct 20 06:09:00 logserver [<ffffffff81197eb0>] __dentry_kill+0x180/0x1f0
Oct 20 06:09:00 logserver [<ffffffff81198066>] dput+0x146/0x240
Oct 20 06:09:00 logserver [<ffffffff8117ece2>] __fput+0x172/0x230
Oct 20 06:09:00 logserver [<ffffffff8117ede9>] ____fput+0x9/0x10
Oct 20 06:09:00 logserver [<ffffffff81081a37>] task_work_run+0xb7/0xf0
Oct 20 06:09:00 logserver [<ffffffff8100e168>] do_notify_resume+0x58/0x70
Oct 20 06:09:00 logserver [<ffffffff815c7a44>] int_signal+0x12/0x17
Oct 20 06:09:00 logserver INFO: task kworker/u16:1:29851 blocked for more than 120 seconds.
Oct 20 06:09:00 logserver Not tainted 4.1.4-hardened #1
Oct 20 06:09:00 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 20 06:09:00 logserver kworker/u16:1 D
Oct 20 06:09:00 logserver ffff88009f66f028
Oct 20 06:09:00 logserver 0 29851 2 0x00000000
Oct 20 06:09:00 logserver Workqueue: writeback bdi_writeback_workfn
Oct 20 06:09:00 logserver (flush-8:32)
Oct 20 06:09:00 logserver
Oct 20 06:09:00 logserver ffff88009f66f028
Oct 20 06:09:00 logserver ffff88009f66efe0
Oct 20 06:09:00 logserver ffff8807f9196e40
Oct 20 06:09:00 logserver ffff88071d84e000
Oct 20 06:09:00 logserver
Oct 20 06:09:00 logserver ffff8807f9197400
Oct 20 06:09:00 logserver 7fffffffffffffff
Oct 20 06:09:00 logserver ffff8807f9196e40
Oct 20 06:09:00 logserver 0000000000000002
Oct 20 06:09:00 logserver
Oct 20 06:09:00 logserver ffff8807f83f4780
Oct 20 06:09:00 logserver ffff88009f66f048
Oct 20 06:09:00 logserver ffffffff815c3592
Oct 20 06:09:00 logserver ffff88009f66f058
Oct 20 06:09:00 logserver
Oct 20 06:09:00 logserver Call Trace:
Oct 20 06:09:00 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 20 06:09:00 logserver [<ffffffff815c6735>] schedule_timeout+0x155/0x1a0
Oct 20 06:09:00 logserver [<ffffffff810a8029>] ? down_trylock+0x29/0x40
Oct 20 06:09:00 logserver [<ffffffff815c5777>] __down+0x77/0xe0
Oct 20 06:09:00 logserver [<ffffffff810a8000>] ? atomic_dec_and_mutex_lock+0x80/0x80
Oct 20 06:09:00 logserver [<ffffffff810a807c>] down+0x3c/0x50
Oct 20 06:09:00 logserver [<ffffffffc0a40a05>] xfs_buf_lock+0x25/0x60 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a40be5>] _xfs_buf_find+0x1a5/0x270 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a77290>] ? exit_xfs_fs+0x16c/0x19fe4 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a40cd5>] xfs_buf_get_map+0x25/0x130 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a77290>] ? exit_xfs_fs+0x16c/0x19fe4 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a41324>] xfs_buf_read_map+0x24/0xc0 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a6c924>] ? _xfs_trans_bjoin+0x44/0x60 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a6cb38>] xfs_trans_read_buf_map+0x98/0x1b0 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a1ad73>] xfs_btree_read_buf_block.constprop.27+0x73/0xc0 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a77290>] ? exit_xfs_fs+0x16c/0x19fe4 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a1ae3f>] xfs_btree_lookup_get_block+0x7f/0x100 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a19fe4>] ? xfs_btree_ptr_offset+0x44/0x80 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a1f3ef>] xfs_btree_lookup+0xbf/0x500 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a06cec>] xfs_alloc_ag_vextent_near+0x8c/0x880 [xfs]
Oct 20 06:09:00 logserver [<ffffffff812f8b21>] ? __list_del_entry+0x11/0x30
Oct 20 06:09:00 logserver [<ffffffffc0a07bc5>] xfs_alloc_ag_vextent+0xc5/0x110 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a08802>] xfs_alloc_vextent+0x3a2/0x4b0 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a16d5b>] xfs_bmap_btalloc+0x3ab/0x750 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a17109>] xfs_bmap_alloc+0x9/0x10 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a17b75>] xfs_bmapi_write+0x4c5/0xa90 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a4ec4b>] xfs_iomap_write_allocate+0x14b/0x360 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a39f2f>] xfs_map_blocks+0x15f/0x180 [xfs]
Oct 20 06:09:00 logserver [<ffffffffc0a3b0a5>] xfs_vm_writepage+0x1a5/0x5a0 [xfs]
Oct 20 06:09:00 logserver [<ffffffff81121922>] __writepage+0x12/0x40
Oct 20 06:09:00 logserver [<ffffffff811227b1>] write_cache_pages+0x201/0x430
Oct 20 06:09:00 logserver [<ffffffff81121910>] ? global_dirtyable_memory+0x50/0x50
Oct 20 06:09:00 logserver [<ffffffff81122a28>] generic_writepages+0x48/0x80
Oct 20 06:09:00 logserver [<ffffffffc0a3aa4e>] xfs_vm_writepages+0x3e/0x50 [xfs]
Oct 20 06:09:00 logserver [<ffffffff81123e19>] do_writepages+0x19/0x30
Oct 20 06:09:00 logserver [<ffffffff811aaa63>] __writeback_single_inode+0x33/0x190
Oct 20 06:09:00 logserver [<ffffffff811aae77>] writeback_sb_inodes+0x2b7/0x4e0
Oct 20 06:09:00 logserver [<ffffffff811ab13f>] __writeback_inodes_wb+0x9f/0xd0
Oct 20 06:09:00 logserver [<ffffffff811ab33b>] wb_writeback+0x1cb/0x210
Oct 20 06:09:00 logserver [<ffffffff811ab892>] bdi_writeback_workfn+0x172/0x390
Oct 20 06:09:00 logserver [<ffffffff8100c70f>] ? __switch_to+0x28f/0x610
Oct 20 06:09:00 logserver [<ffffffff8107ca16>] process_one_work+0x146/0x3d0
Oct 20 06:09:00 logserver [<ffffffff8107e043>] worker_thread+0x53/0x530
Oct 20 06:09:00 logserver [<ffffffff8107dff0>] ? rescuer_thread+0x3a0/0x3a0
Oct 20 06:09:00 logserver [<ffffffff81083273>] kthread+0xd3/0xf0
Oct 20 06:09:00 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 20 06:09:00 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 20 06:09:00 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 29 15:11:01 logserver INFO: task kthreadd:2 blocked for more than 120 seconds.
Oct 29 15:11:01 logserver Not tainted 4.1.4-hardened #1
Oct 29 15:11:01 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 29 15:11:01 logserver kthreadd D
Oct 29 15:11:01 logserver ffff8807fc5ff428
Oct 29 15:11:01 logserver 0 2 0 0x00000000
Oct 29 15:11:01 logserver ffff8807fc5ff428
Oct 29 15:11:01 logserver ffff8807fc5ff408
Oct 29 15:11:01 logserver ffff8807fc5f0930
Oct 29 15:11:01 logserver ffff88081fad2c80
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffff8807fc5f0ef0
Oct 29 15:11:01 logserver ffff8807fc5ff568
Oct 29 15:11:01 logserver 7fffffffffffffff
Oct 29 15:11:01 logserver ffff8807fc5f0930
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffffffffffffffff
Oct 29 15:11:01 logserver ffff8807fc5ff448
Oct 29 15:11:01 logserver ffffffff815c3592
Oct 29 15:11:01 logserver ffff8807fc5ff468
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver Call Trace:
Oct 29 15:11:01 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 29 15:11:01 logserver [<ffffffff815c6735>] schedule_timeout+0x155/0x1a0
Oct 29 15:11:01 logserver [<ffffffff8108e0b3>] ? try_to_wake_up+0x1d3/0x300
Oct 29 15:11:01 logserver [<ffffffff815c3ff3>] wait_for_completion+0x123/0x180
Oct 29 15:11:01 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 29 15:11:01 logserver [<ffffffff8107d888>] flush_work+0xf8/0x1e0
Oct 29 15:11:01 logserver [<ffffffff8107a9a0>] ? destroy_worker+0xa0/0xa0
Oct 29 15:11:01 logserver [<ffffffffc0a63858>] xlog_cil_force_lsn+0x88/0x210 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a62028>] _xfs_log_force_lsn+0x68/0x2f0 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a622c3>] xfs_log_force_lsn+0x13/0x40 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a5140c>] __xfs_iunpin_wait+0x8c/0x110 [xfs]
Oct 29 15:11:01 logserver [<ffffffff810a3e10>] ? autoremove_wake_function+0x40/0x40
Oct 29 15:11:01 logserver [<ffffffffc0a54b69>] xfs_iunpin_wait+0x19/0x20 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a4842c>] xfs_reclaim_inode+0x7c/0x340 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a48923>] xfs_reclaim_inodes_ag+0x233/0x360 [xfs]
Oct 29 15:11:01 logserver [<ffffffff8108e202>] ? wake_up_process+0x22/0x50
Oct 29 15:11:01 logserver [<ffffffffc0a4929e>] xfs_reclaim_inodes_nr+0x2e/0x40 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a59394>] xfs_fs_free_cached_objects+0x14/0x20 [xfs]
Oct 29 15:11:01 logserver [<ffffffff8118095b>] super_cache_scan+0x19b/0x1a0
Oct 29 15:11:01 logserver [<ffffffff81127c72>] shrink_slab.part.48.constprop.60+0x1a2/0x270
Oct 29 15:11:01 logserver [<ffffffff8112a745>] shrink_zone+0x175/0x1b0
Oct 29 15:11:01 logserver [<ffffffff8112a93c>] do_try_to_free_pages+0x1bc/0x3b0
Oct 29 15:11:01 logserver [<ffffffff8112abce>] try_to_free_pages+0x9e/0xd0
Oct 29 15:11:01 logserver [<ffffffff81120c0b>] __alloc_pages_nodemask+0x53b/0x960
Oct 29 15:11:01 logserver [<ffffffff81121076>] alloc_kmem_pages_node+0x46/0x60
Oct 29 15:11:01 logserver [<ffffffff81060fe8>] copy_process.part.36+0x128/0x17a0
Oct 29 15:11:01 logserver [<ffffffff81017689>] ? sched_clock+0x9/0x10
Oct 29 15:11:01 logserver [<ffffffff810916ed>] ? sched_clock_cpu+0x9d/0xb0
Oct 29 15:11:01 logserver [<ffffffff81099f8c>] ? put_prev_entity+0x2c/0x4a0
Oct 29 15:11:01 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 29 15:11:01 logserver [<ffffffff8100c70f>] ? __switch_to+0x28f/0x610
Oct 29 15:11:01 logserver [<ffffffff81062817>] do_fork+0xd7/0x330
Oct 29 15:11:01 logserver [<ffffffff815c2fcc>] ? __schedule+0x29c/0x830
Oct 29 15:11:01 logserver [<ffffffff81062a91>] kernel_thread+0x21/0x30
Oct 29 15:11:01 logserver [<ffffffff81083be4>] kthreadd+0x144/0x1a0
Oct 29 15:11:01 logserver [<ffffffff81083aa0>] ? kthread_create_on_cpu+0x60/0x60
Oct 29 15:11:01 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 29 15:11:01 logserver [<ffffffff81083aa0>] ? kthread_create_on_cpu+0x60/0x60
Oct 29 15:11:01 logserver INFO: task xfsaild/sda4:3534 blocked for more than 120 seconds.
Oct 29 15:11:01 logserver Not tainted 4.1.4-hardened #1
Oct 29 15:11:01 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 29 15:11:01 logserver xfsaild/sda4 D
Oct 29 15:11:01 logserver ffff8807f71dbb08
Oct 29 15:11:01 logserver 0 3534 2 0x00000000
Oct 29 15:11:01 logserver ffff8807f71dbb08
Oct 29 15:11:01 logserver ffff8807f71dbae8
Oct 29 15:11:01 logserver ffff8807f73cc980
Oct 29 15:11:01 logserver ffff88081fa92c80
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffff8807f73ccf40
Oct 29 15:11:01 logserver ffff8807f71dbc48
Oct 29 15:11:01 logserver 7fffffffffffffff
Oct 29 15:11:01 logserver ffff8807f73cc980
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffffffffffffffff
Oct 29 15:11:01 logserver ffff8807f71dbb28
Oct 29 15:11:01 logserver ffffffff815c3592
Oct 29 15:11:01 logserver ffff8807f71dbb48
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver Call Trace:
Oct 29 15:11:01 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 29 15:11:01 logserver [<ffffffff815c6735>] schedule_timeout+0x155/0x1a0
Oct 29 15:11:01 logserver [<ffffffff8108e0b3>] ? try_to_wake_up+0x1d3/0x300
Oct 29 15:11:01 logserver [<ffffffff815c3ff3>] wait_for_completion+0x123/0x180
Oct 29 15:11:01 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 29 15:11:01 logserver [<ffffffff8107d888>] flush_work+0xf8/0x1e0
Oct 29 15:11:01 logserver [<ffffffff8107a9a0>] ? destroy_worker+0xa0/0xa0
Oct 29 15:11:01 logserver [<ffffffffc0a63858>] xlog_cil_force_lsn+0x88/0x210 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a61d1b>] _xfs_log_force+0x6b/0x280 [xfs]
Oct 29 15:11:01 logserver [<ffffffff810c3550>] ? migrate_timer_list+0x60/0x60
Oct 29 15:11:01 logserver [<ffffffffc0a61f43>] xfs_log_force+0x13/0x40 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a6bdcb>] xfsaild+0x13b/0x4c0 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a6bc90>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs]
Oct 29 15:11:01 logserver [<ffffffff81083273>] kthread+0xd3/0xf0
Oct 29 15:11:01 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 29 15:11:01 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 29 15:11:01 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 29 15:11:01 logserver INFO: task xfsaild/sdc4:3544 blocked for more than 120 seconds.
Oct 29 15:11:01 logserver Not tainted 4.1.4-hardened #1
Oct 29 15:11:01 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 29 15:11:01 logserver xfsaild/sdc4 D
Oct 29 15:11:01 logserver ffff8807fa033d28
Oct 29 15:11:01 logserver 0 3544 2 0x00000000
Oct 29 15:11:01 logserver ffff8807fa033d28
Oct 29 15:11:01 logserver ffff8807fa033d18
Oct 29 15:11:01 logserver ffff8807f84dc050
Oct 29 15:11:01 logserver ffff8807faa53c00
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffff8807f84dc610
Oct 29 15:11:01 logserver ffff8807faa53c00
Oct 29 15:11:01 logserver 0000000000000000
Oct 29 15:11:01 logserver ffff8807faa53d28
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffff8807fbc96a00
Oct 29 15:11:01 logserver ffff8807fa033d48
Oct 29 15:11:01 logserver ffffffff815c3592
Oct 29 15:11:01 logserver ffff8807faa53c00
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver Call Trace:
Oct 29 15:11:01 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 29 15:11:01 logserver [<ffffffffc0a61dbb>] _xfs_log_force+0x10b/0x280 [xfs]
Oct 29 15:11:01 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 29 15:11:01 logserver [<ffffffff810c3550>] ? migrate_timer_list+0x60/0x60
Oct 29 15:11:01 logserver [<ffffffffc0a61f43>] xfs_log_force+0x13/0x40 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a6bdcb>] xfsaild+0x13b/0x4c0 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a6bc90>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs]
Oct 29 15:11:01 logserver [<ffffffff81083273>] kthread+0xd3/0xf0
Oct 29 15:11:01 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 29 15:11:01 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 29 15:11:01 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 29 15:11:01 logserver INFO: task xfsaild/sde4:3554 blocked for more than 120 seconds.
Oct 29 15:11:01 logserver Not tainted 4.1.4-hardened #1
Oct 29 15:11:01 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 29 15:11:01 logserver xfsaild/sde4 D
Oct 29 15:11:01 logserver ffff8807f931bb08
Oct 29 15:11:01 logserver 0 3554 2 0x00000000
Oct 29 15:11:01 logserver ffff8807f931bb08
Oct 29 15:11:01 logserver ffff8807f931bad8
Oct 29 15:11:01 logserver ffff8807f847ee40
Oct 29 15:11:01 logserver ffff8807f931bae8
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffff8807f847f400
Oct 29 15:11:01 logserver ffff8807f931bc48
Oct 29 15:11:01 logserver 7fffffffffffffff
Oct 29 15:11:01 logserver ffff8807f847ee40
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffffffffffffffff
Oct 29 15:11:01 logserver ffff8807f931bb28
Oct 29 15:11:01 logserver ffffffff815c3592
Oct 29 15:11:01 logserver ffff8807f931bb38
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver Call Trace:
Oct 29 15:11:01 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 29 15:11:01 logserver [<ffffffff815c6735>] schedule_timeout+0x155/0x1a0
Oct 29 15:11:01 logserver [<ffffffff8108e0b3>] ? try_to_wake_up+0x1d3/0x300
Oct 29 15:11:01 logserver [<ffffffff815c3ff3>] wait_for_completion+0x123/0x180
Oct 29 15:11:01 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 29 15:11:01 logserver [<ffffffff8107d888>] flush_work+0xf8/0x1e0
Oct 29 15:11:01 logserver [<ffffffff8107a9a0>] ? destroy_worker+0xa0/0xa0
Oct 29 15:11:01 logserver [<ffffffffc0a63858>] xlog_cil_force_lsn+0x88/0x210 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a61d1b>] _xfs_log_force+0x6b/0x280 [xfs]
Oct 29 15:11:01 logserver [<ffffffff810c3550>] ? migrate_timer_list+0x60/0x60
Oct 29 15:11:01 logserver [<ffffffffc0a61f43>] xfs_log_force+0x13/0x40 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a6bdcb>] xfsaild+0x13b/0x4c0 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a6bc90>] ? xfs_trans_ail_cursor_first+0xb0/0xb0 [xfs]
Oct 29 15:11:01 logserver [<ffffffff81083273>] kthread+0xd3/0xf0
Oct 29 15:11:01 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 29 15:11:01 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 29 15:11:01 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 29 15:11:01 logserver INFO: task xargs:28059 blocked for more than 120 seconds.
Oct 29 15:11:01 logserver Not tainted 4.1.4-hardened #1
Oct 29 15:11:01 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 29 15:11:01 logserver xargs D
Oct 29 15:11:01 logserver ffff880032247488
Oct 29 15:11:01 logserver 0 28059 28045 0x00000000
Oct 29 15:11:01 logserver ffff880032247488
Oct 29 15:11:01 logserver ffff88081fad2c80
Oct 29 15:11:01 logserver ffff8801116e8930
Oct 29 15:11:01 logserver ffff88081fad2c80
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffff8801116e8ef0
Oct 29 15:11:01 logserver ffff8800322475c8
Oct 29 15:11:01 logserver 7fffffffffffffff
Oct 29 15:11:01 logserver ffff8801116e8930
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffffffffffffffff
Oct 29 15:11:01 logserver ffff8800322474a8
Oct 29 15:11:01 logserver ffffffff815c3592
Oct 29 15:11:01 logserver ffff8800322474b8
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver Call Trace:
Oct 29 15:11:01 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 29 15:11:01 logserver [<ffffffff815c6735>] schedule_timeout+0x155/0x1a0
Oct 29 15:11:01 logserver [<ffffffff8108e0b3>] ? try_to_wake_up+0x1d3/0x300
Oct 29 15:11:01 logserver [<ffffffff815c3ff3>] wait_for_completion+0x123/0x180
Oct 29 15:11:01 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 29 15:11:01 logserver [<ffffffff8107d888>] flush_work+0xf8/0x1e0
Oct 29 15:11:01 logserver [<ffffffff8107a9a0>] ? destroy_worker+0xa0/0xa0
Oct 29 15:11:01 logserver [<ffffffffc0a63858>] xlog_cil_force_lsn+0x88/0x210 [xfs]
Oct 29 15:11:01 logserver [<ffffffff81169ea7>] ? __slab_free+0xb7/0x260
Oct 29 15:11:01 logserver [<ffffffffc0a62028>] _xfs_log_force_lsn+0x68/0x2f0 [xfs]
Oct 29 15:11:01 logserver [<ffffffff81169f80>] ? __slab_free+0x190/0x260
Oct 29 15:11:01 logserver [<ffffffffc0a622c3>] xfs_log_force_lsn+0x13/0x40 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a5140c>] __xfs_iunpin_wait+0x8c/0x110 [xfs]
Oct 29 15:11:01 logserver [<ffffffff810a3e10>] ? autoremove_wake_function+0x40/0x40
Oct 29 15:11:01 logserver [<ffffffffc0a54b69>] xfs_iunpin_wait+0x19/0x20 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a4842c>] xfs_reclaim_inode+0x7c/0x340 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a48923>] xfs_reclaim_inodes_ag+0x233/0x360 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a4929e>] xfs_reclaim_inodes_nr+0x2e/0x40 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a59394>] xfs_fs_free_cached_objects+0x14/0x20 [xfs]
Oct 29 15:11:01 logserver [<ffffffff8118095b>] super_cache_scan+0x19b/0x1a0
Oct 29 15:11:01 logserver [<ffffffff81127c72>] shrink_slab.part.48.constprop.60+0x1a2/0x270
Oct 29 15:11:01 logserver [<ffffffff8112a745>] shrink_zone+0x175/0x1b0
Oct 29 15:11:01 logserver [<ffffffff8112a93c>] do_try_to_free_pages+0x1bc/0x3b0
Oct 29 15:11:01 logserver [<ffffffff8112abce>] try_to_free_pages+0x9e/0xd0
Oct 29 15:11:01 logserver [<ffffffff81120c0b>] __alloc_pages_nodemask+0x53b/0x960
Oct 29 15:11:01 logserver [<ffffffff81121076>] alloc_kmem_pages_node+0x46/0x60
Oct 29 15:11:01 logserver [<ffffffff81060fe8>] copy_process.part.36+0x128/0x17a0
Oct 29 15:11:01 logserver [<ffffffff810a3baf>] ? remove_wait_queue+0x2f/0x40
Oct 29 15:11:01 logserver [<ffffffff81065eb3>] ? do_wait+0x123/0x220
Oct 29 15:11:01 logserver [<ffffffff81062817>] do_fork+0xd7/0x330
Oct 29 15:11:01 logserver [<ffffffff81062af1>] SyS_clone+0x11/0x20
Oct 29 15:11:01 logserver [<ffffffff815c7876>] system_call_fastpath+0x16/0x79
Oct 29 15:11:01 logserver INFO: task kworker/3:0H:22312 blocked for more than 120 seconds.
Oct 29 15:11:01 logserver Not tainted 4.1.4-hardened #1
Oct 29 15:11:01 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 29 15:11:01 logserver kworker/3:0H D
Oct 29 15:11:01 logserver ffff88009e27fae8
Oct 29 15:11:01 logserver 0 22312 2 0x00000000
Oct 29 15:11:01 logserver Workqueue: xfs-log/sde4 xfs_log_worker [xfs]
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffff88009e27fae8
Oct 29 15:11:01 logserver ffff88081fad2c80
Oct 29 15:11:01 logserver ffff8807fbf40000
Oct 29 15:11:01 logserver ffff8807f72a24c0
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffff8807fbf405c0
Oct 29 15:11:01 logserver ffff88009e27fc28
Oct 29 15:11:01 logserver 7fffffffffffffff
Oct 29 15:11:01 logserver ffff8807fbf40000
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffffffffffffffff
Oct 29 15:11:01 logserver ffff88009e27fb08
Oct 29 15:11:01 logserver ffffffff815c3592
Oct 29 15:11:01 logserver ffff88009e27fb18
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver Call Trace:
Oct 29 15:11:01 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 29 15:11:01 logserver [<ffffffff815c6735>] schedule_timeout+0x155/0x1a0
Oct 29 15:11:01 logserver [<ffffffff8108e0b3>] ? try_to_wake_up+0x1d3/0x300
Oct 29 15:11:01 logserver [<ffffffff815c3ff3>] wait_for_completion+0x123/0x180
Oct 29 15:11:01 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 29 15:11:01 logserver [<ffffffff8107d888>] flush_work+0xf8/0x1e0
Oct 29 15:11:01 logserver [<ffffffff8107a9a0>] ? destroy_worker+0xa0/0xa0
Oct 29 15:11:01 logserver [<ffffffff81090202>] ? cpuset_cpu_inactive+0xe2/0x120
Oct 29 15:11:01 logserver [<ffffffffc0a63858>] xlog_cil_force_lsn+0x88/0x210 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a61d1b>] _xfs_log_force+0x6b/0x280 [xfs]
Oct 29 15:11:01 logserver [<ffffffff8109d7f0>] ? pick_next_task_fair+0x560/0x980
Oct 29 15:11:01 logserver [<ffffffff8100c70f>] ? __switch_to+0x28f/0x610
Oct 29 15:11:01 logserver [<ffffffffc0a61f43>] xfs_log_force+0x13/0x40 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a61f93>] xfs_log_worker+0x23/0x50 [xfs]
Oct 29 15:11:01 logserver [<ffffffff8107ca16>] process_one_work+0x146/0x3d0
Oct 29 15:11:01 logserver [<ffffffff8107e043>] worker_thread+0x53/0x530
Oct 29 15:11:01 logserver [<ffffffff8107dff0>] ? rescuer_thread+0x3a0/0x3a0
Oct 29 15:11:01 logserver [<ffffffff81083273>] kthread+0xd3/0xf0
Oct 29 15:11:01 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 29 15:11:01 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 29 15:11:01 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 29 15:11:01 logserver INFO: task kworker/1:0:23324 blocked for more than 120 seconds.
Oct 29 15:11:01 logserver Not tainted 4.1.4-hardened #1
Oct 29 15:11:01 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 29 15:11:01 logserver kworker/1:0 D
Oct 29 15:11:01 logserver ffff880132e57b68
Oct 29 15:11:01 logserver 0 23324 2 0x00000000
Oct 29 15:11:01 logserver Workqueue: xfs-cil/sdf4 xlog_cil_push_work [xfs]
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffff880132e57b68
Oct 29 15:11:01 logserver ffff880132e57b58
Oct 29 15:11:01 logserver ffff8807f7d524c0
Oct 29 15:11:01 logserver 0000000000000000
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffff8807f7d52a80
Oct 29 15:11:01 logserver ffff8807fabd7d28
Oct 29 15:11:01 logserver ffff880428620170
Oct 29 15:11:01 logserver ffff8807f7d524c0
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffff8807fabd7d00
Oct 29 15:11:01 logserver ffff880132e57b88
Oct 29 15:11:01 logserver ffffffff815c3592
Oct 29 15:11:01 logserver ffff8807fabd7d28
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver Call Trace:
Oct 29 15:11:01 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 29 15:11:01 logserver [<ffffffffc0a6108b>] xlog_state_get_iclog_space+0xeb/0x300 [xfs]
Oct 29 15:11:01 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 29 15:11:01 logserver [<ffffffffc0a61581>] xlog_write+0x1d1/0x800 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a62f8f>] xlog_cil_push+0x23f/0x420 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a63180>] xlog_cil_push_work+0x10/0x20 [xfs]
Oct 29 15:11:01 logserver [<ffffffff8107ca16>] process_one_work+0x146/0x3d0
Oct 29 15:11:01 logserver [<ffffffff8107e1ac>] worker_thread+0x1bc/0x530
Oct 29 15:11:01 logserver [<ffffffff8107dff0>] ? rescuer_thread+0x3a0/0x3a0
Oct 29 15:11:01 logserver [<ffffffff81083273>] kthread+0xd3/0xf0
Oct 29 15:11:01 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 29 15:11:01 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 29 15:11:01 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 29 15:11:01 logserver INFO: task kworker/3:1:24406 blocked for more than 120 seconds.
Oct 29 15:11:01 logserver Not tainted 4.1.4-hardened #1
Oct 29 15:11:01 logserver "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 29 15:11:01 logserver kworker/3:1 D
Oct 29 15:11:01 logserver ffff880104e4fb68
Oct 29 15:11:01 logserver 0 24406 2 0x00000000
Oct 29 15:11:01 logserver Workqueue: xfs-cil/sde4 xlog_cil_push_work [xfs]
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffff880104e4fb68
Oct 29 15:11:01 logserver ffff880104e4fb58
Oct 29 15:11:01 logserver ffff8806b1080930
Oct 29 15:11:01 logserver 0000000000000000
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffff8806b1080ef0
Oct 29 15:11:01 logserver ffff8807fabd6d28
Oct 29 15:11:01 logserver ffff88010668d7e8
Oct 29 15:11:01 logserver ffff8806b1080930
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver ffff8807fabd6d00
Oct 29 15:11:01 logserver ffff880104e4fb88
Oct 29 15:11:01 logserver ffffffff815c3592
Oct 29 15:11:01 logserver ffff8807fabd6d28
Oct 29 15:11:01 logserver
Oct 29 15:11:01 logserver Call Trace:
Oct 29 15:11:01 logserver [<ffffffff815c3592>] schedule+0x32/0x80
Oct 29 15:11:01 logserver [<ffffffffc0a6108b>] xlog_state_get_iclog_space+0xeb/0x300 [xfs]
Oct 29 15:11:01 logserver [<ffffffff8108e240>] ? wake_up_state+0x10/0x10
Oct 29 15:11:01 logserver [<ffffffffc0a61581>] xlog_write+0x1d1/0x800 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a62f8f>] xlog_cil_push+0x23f/0x420 [xfs]
Oct 29 15:11:01 logserver [<ffffffffc0a63180>] xlog_cil_push_work+0x10/0x20 [xfs]
Oct 29 15:11:01 logserver [<ffffffff8107ca16>] process_one_work+0x146/0x3d0
Oct 29 15:11:01 logserver [<ffffffff8107e1ac>] worker_thread+0x1bc/0x530
Oct 29 15:11:01 logserver [<ffffffff8107dff0>] ? rescuer_thread+0x3a0/0x3a0
Oct 29 15:11:01 logserver [<ffffffff81083273>] kthread+0xd3/0xf0
Oct 29 15:11:01 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
Oct 29 15:11:01 logserver [<ffffffff815c7c9e>] ret_from_fork+0x3e/0x70
Oct 29 15:11:01 logserver [<ffffffff810831a0>] ? kthread_create_on_node+0x1a0/0x1a0
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
