On 25/01/14 16:41, Stuart Longland wrote:
> Hi Gregory,
> On 24/01/14 12:20, Gregory Farnum wrote:
>> Did the cluster actually detect the node as down? (You could check
>> this by looking at the ceph -w output or similar when running the
>> test.) If it was detected as down and the VM continued to block
>> (modulo maybe a little time for the client to decide its monitor was
>> down; I forget what the timeouts are there), that would be odd.
>
> I shall give that a command a try next time I get near the cluster
> (Tuesday). (I could do it today I guess, but I can't remotely power
> nodes back on, or hard-power them off from home.)
Okay, I did some further tests today. In addition to the Windows 2008R2
VM, I also started pummelling it with my own laptop (2.6GHz Core i5
3220M; 8GB RAM) which runs Gentoo Linux AMD64 and kernel 3.12.4.
ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) was
installed from Gentoo's repository.
I mapped a 20GB RBD using `rbd map`, formatted it XFS, then started
pummeling that with my gigabit link (which passes through a couple of
shared VLAN trunks), various disk stress testers and dd.
Whilst that was proceeding, I then wandered to the server rack and
started fiddling.
Before simulating outages, I was getting write speeds between the
74MB/sec and 145MB/sec according to dbench. dd was getting about
15.1MB/sec writing 1GB of random data.
With a bash script running dd in a loop, and also running bonnie++ to
really push things, I started playing with the nodes, rebooting some,
powering off others.
It seems there's a limit to how often you can power things off, even if
you wait for the cluster health to recover before proceeding.
Eventually the client (kernel or userspace) gets fed up, as seen in the
attached log.
At present, `ceph -s` reports:
> HEALTH_WARN clock skew detected on mon.2
> cluster b9b2ed48-e249-48ee-8e76-86493c2cc849
> health HEALTH_WARN clock skew detected on mon.2
> monmap e1: 3 mons at {0=10.87.160.224:6789/0,1=10.87.160.225:6789/0,2=10.87.160.226:6789/0}, election epoch 42, qu
> orum 0,1,2 0,1,2
> osdmap e174: 6 osds: 6 up, 6 in
> pgmap v45386: 800 pgs, 4 pools, 398 GB data, 102026 objects
> 1195 GB used, 15563 GB / 16758 GB avail
> 800 active+clean
>
and out of `ceph -w` I get:
> 6758 GB avail; 1130 B/s wr, 0 op/s
> 2014-01-28 14:49:20.812284 mon.0 [INF] pgmap v45379: 800 pgs: 800 active+clean; 398 GB data, 1195 GB used, 15563 GB / 16758 GB avail; 1126 B/s wr, 0 op/s
> 2014-01-28 14:49:34.225852 mon.0 [INF] pgmap v45380: 800 pgs: 800 active+clean; 398 GB data, 1195 GB used, 15563 GB / 16758 GB avail; 71 B/s wr, 0 op/s
> 2014-01-28 14:49:48.056665 mon.0 [INF] pgmap v45381: 800 pgs: 800 active+clean; 398 GB data, 1195 GB used, 15563 GB / 16758 GB avail
> 2014-01-28 14:49:49.065547 mon.0 [INF] pgmap v45382: 800 pgs: 800 active+clean; 398 GB data, 1195 GB used, 15563 GB / 16758 GB avail
> 2014-01-28 14:49:50.074878 mon.0 [INF] pgmap v45383: 800 pgs: 800 active+clean; 398 GB data, 1195 GB used, 15563 GB / 16758 GB avail; 16270 B/s wr, 0 op/s
> 2014-01-28 14:49:51.083527 mon.0 [INF] pgmap v45384: 800 pgs: 800 active+clean; 398 GB data, 1195 GB used, 15563 GB / 16758 GB avail; 16742 B/s wr, 0 op/s
> 2014-01-28 14:50:10.437994 mon.0 [WRN] mon.2 10.87.160.226:6789/0 clock skew 4.05188s > max 0.05s
> 2014-01-28 14:50:19.813536 mon.0 [INF] pgmap v45385: 800 pgs: 800 active+clean; 398 GB data, 1195 GB used, 15563 GB / 16758 GB avail; 1140 B/s wr, 0 op/s
> 2014-01-28 14:50:20.818168 mon.0 [INF] pgmap v45386: 800 pgs: 800 active+clean; 398 GB data, 1195 GB used, 15563 GB / 16758 GB avail; 1136 B/s wr, 0 op/s
> 2014-01-28 14:50:49.816479 mon.0 [INF] pgmap v45387: 800 pgs: 800 active+clean; 398 GB data, 1195 GB used, 15563 GB / 16758 GB avail; 1130 B/s wr, 0 op/s
> 2014-01-28 14:50:50.825369 mon.0 [INF] pgmap v45388: 800 pgs: 800 active+clean; 398 GB data, 1195 GB used, 15563 GB / 16758 GB avail; 1126 B/s wr, 0 op/s
> 2014-01-28 14:51:19.819779 mon.0 [INF] pgmap v45389: 800 pgs: 800 active+clean; 398 GB data, 1195 GB used, 15563 GB / 16758 GB avail; 1130 B/s wr, 0 op/s
I do note ntp doesn't seem to be doing its job, but that's a side issue.
I'm not sure if there's some magic way to tickle the kernel rbd driver
to get things moving. Short of rebooting that is.
Interestingly, today the Windows VM did not seize up despite SQLIOSim
giving the cluster its worst.
So it would appear this issue is intermittent in nature. I'm not sure
what else I can do to try and uncover what causes problems.
Maybe setting up a VM-based cluster with all the debugging turned on and
a script randomly calling virsh destroy / virsh start on VMs to try to
rattle out the bug, might help.
Regards,
--
Stuart Longland
Systems Engineer
_ ___
\ /|_) | T: +61 7 3535 9619
\/ | \ | 38b Douglas Street F: +61 7 3535 9699
SYSTEMS Milton QLD 4064 http://www.vrt.com.au
Jan 28 09:13:22 localhost kernel: [ 2039.971245] XFS (rbd1): Mounting Filesystem
Jan 28 09:13:22 localhost kernel: [ 2040.254667] XFS (rbd1): Ending clean mount
Jan 28 09:40:22 localhost kernel: [ 3661.195556] libceph: osd5 10.20.30.226:6803 socket closed (con state OPEN)
Jan 28 09:40:22 localhost kernel: [ 3661.207944] libceph: osd0 10.20.30.224:6804 socket closed (con state OPEN)
Jan 28 09:40:23 localhost kernel: [ 3661.348328] libceph: osd4 10.20.30.226:6800 socket closed (con state OPEN)
Jan 28 09:40:23 localhost kernel: [ 3661.405353] libceph: osd1 10.20.30.224:6803 socket closed (con state OPEN)
Jan 28 12:25:58 localhost kernel: [13603.478077] libceph: osd0 10.20.30.224:6804 socket closed (con state OPEN)
Jan 28 13:08:34 localhost kernel: [16161.004210] libceph: osd4 down
Jan 28 13:08:34 localhost kernel: [16161.004216] libceph: osd5 down
Jan 28 13:08:34 localhost kernel: [16161.139014] libceph: osd1 10.20.30.224:6803 socket closed (con state OPEN)
Jan 28 13:08:34 localhost kernel: [16161.174997] libceph: osd0 10.20.30.224:6804 socket closed (con state OPEN)
Jan 28 13:08:34 localhost kernel: [16161.231374] libceph: osd3 10.20.30.225:6800 socket closed (con state OPEN)
Jan 28 13:08:34 localhost kernel: [16161.251399] libceph: osd2 10.20.30.225:6803 socket closed (con state OPEN)
Jan 28 13:12:27 localhost kernel: [16394.591109] libceph: mon2 10.20.30.226:6789 socket closed (con state OPEN)
Jan 28 13:12:27 localhost kernel: [16394.591129] libceph: mon2 10.20.30.226:6789 session lost, hunting for new mon
Jan 28 13:12:27 localhost kernel: [16394.594618] libceph: osd0 down
Jan 28 13:12:27 localhost kernel: [16394.594624] libceph: osd1 down
Jan 28 13:12:27 localhost kernel: [16394.595511] libceph: osd4 up
Jan 28 13:12:27 localhost kernel: [16394.596331] libceph: osd5 up
Jan 28 13:12:27 localhost kernel: [16394.597188] libceph: mon1 10.20.30.225:6789 session established
Jan 28 13:15:56 localhost kernel: [16603.733951] libceph: osd0 up
Jan 28 13:16:07 localhost kernel: [16613.970769] libceph: osd1 up
Jan 28 13:16:07 localhost kernel: [16614.192961] libceph: osd4 10.20.30.226:6800 socket closed (con state OPEN)
Jan 28 13:16:07 localhost kernel: [16614.383263] libceph: osd3 10.20.30.225:6800 socket closed (con state OPEN)
Jan 28 13:21:34 localhost kernel: [16941.190329] libceph: osd3 10.20.30.225:6800 socket closed (con state OPEN)
Jan 28 13:21:37 localhost kernel: [16945.032781] libceph: osd2 10.20.30.225:6803 socket closed (con state OPEN)
Jan 28 13:21:39 localhost kernel: [16946.193433] libceph: mon1 10.20.30.225:6789 socket closed (con state OPEN)
Jan 28 13:21:39 localhost kernel: [16946.193453] libceph: mon1 10.20.30.225:6789 session lost, hunting for new mon
Jan 28 13:21:39 localhost kernel: [16946.196900] libceph: osd2 down
Jan 28 13:21:39 localhost kernel: [16946.196906] libceph: osd3 down
Jan 28 13:21:39 localhost kernel: [16946.196908] libceph: osd4 down
Jan 28 13:21:39 localhost kernel: [16946.196910] libceph: osd5 down
Jan 28 13:21:39 localhost kernel: [16946.197918] libceph: mon0 10.20.30.224:6789 session established
Jan 28 13:21:47 localhost kernel: [16954.218937] libceph: osd3 up
Jan 28 13:21:47 localhost kernel: [16954.460249] libceph: osd1 10.20.30.224:6803 socket closed (con state OPEN)
Jan 28 13:21:48 localhost kernel: [16955.284548] libceph: osd2 up
Jan 28 13:21:48 localhost kernel: [16955.420976] libceph: osd1 10.20.30.224:6803 socket closed (con state OPEN)
Jan 28 13:21:48 localhost kernel: [16955.902596] libceph: osd0 10.20.30.224:6800 socket closed (con state OPEN)
Jan 28 13:23:12 localhost kernel: [17040.034319] libceph: osd4 up
Jan 28 13:23:14 localhost kernel: [17041.194253] libceph: osd5 up
Jan 28 13:37:10 localhost kernel: [17877.679964] libceph: osd2 down
Jan 28 13:37:10 localhost kernel: [17877.679970] libceph: osd3 down
Jan 28 13:37:11 localhost kernel: [17878.943731] libceph: osd5 10.20.30.226:6803 socket closed (con state OPEN)
Jan 28 13:37:11 localhost kernel: [17879.102179] libceph: osd0 10.20.30.224:6800 socket closed (con state OPEN)
Jan 28 13:37:11 localhost kernel: [17879.107420] libceph: osd4 10.20.30.226:6800 socket closed (con state OPEN)
Jan 28 13:42:11 localhost kernel: [18179.471279] libceph: osd2 weight 0x0 (out)
Jan 28 13:42:11 localhost kernel: [18179.471284] libceph: osd3 weight 0x0 (out)
Jan 28 13:42:11 localhost kernel: [18179.644719] libceph: osd4 10.20.30.226:6800 socket closed (con state OPEN)
Jan 28 13:42:11 localhost kernel: [18179.690738] libceph: osd1 10.20.30.224:6803 socket closed (con state OPEN)
Jan 28 13:47:25 localhost kernel: [18493.299454] INFO: task kworker/u8:1:13833 blocked for more than 120 seconds.
Jan 28 13:47:25 localhost kernel: [18493.299461] Not tainted 3.12.4-rikishi #1
Jan 28 13:47:25 localhost kernel: [18493.299463] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 28 13:47:25 localhost kernel: [18493.299466] kworker/u8:1 D ffff88009c638498 0 13833 2 0x00000000
Jan 28 13:47:25 localhost kernel: [18493.299479] Workqueue: writeback bdi_writeback_workfn (flush-253:0)
Jan 28 13:47:25 localhost kernel: [18493.299483] ffff88009c638180 0000000000000046 ffff88022304d078 ffff880223be74f0
Jan 28 13:47:25 localhost kernel: [18493.299487] 0000000000011740 ffff8801e2ef3fd8 ffff8801e2ef3fd8 ffff88009c638180
Jan 28 13:47:25 localhost kernel: [18493.299491] ffff8802231fbc00 ffffffff81295883 ffff8800c4958bf0 ffff880220a689f8
Jan 28 13:47:25 localhost kernel: [18493.299495] Call Trace:
Jan 28 13:47:25 localhost kernel: [18493.299505] [<ffffffff81295883>] ? blk_fetch_request+0x9/0x25
Jan 28 13:47:25 localhost kernel: [18493.299514] [<ffffffffa06154f4>] ? rbd_request_fn+0x203/0x21e [rbd]
Jan 28 13:47:25 localhost kernel: [18493.299519] [<ffffffff812931bf>] ? __blk_run_queue+0x29/0x31
Jan 28 13:47:25 localhost kernel: [18493.299524] [<ffffffff812952ad>] ? queue_unplugged.isra.58+0x14/0x20
Jan 28 13:47:25 localhost kernel: [18493.299532] [<ffffffff814f3725>] ? io_schedule+0x86/0xc2
Jan 28 13:47:25 localhost kernel: [18493.299536] [<ffffffff8129509f>] ? get_request+0x4e8/0x560
Jan 28 13:47:25 localhost kernel: [18493.299542] [<ffffffff8104b63a>] ? abort_exclusive_wait+0x79/0x79
Jan 28 13:47:25 localhost kernel: [18493.299546] [<ffffffff81295d88>] ? blk_queue_bio+0x19a/0x2c2
Jan 28 13:47:25 localhost kernel: [18493.299551] [<ffffffff8129446a>] ? generic_make_request+0x96/0xd5
Jan 28 13:47:25 localhost kernel: [18493.299555] [<ffffffff8129458e>] ? submit_bio+0xe5/0x101
Jan 28 13:47:25 localhost kernel: [18493.299562] [<ffffffff81220627>] ? xfs_submit_ioend+0xaf/0xf6
Jan 28 13:47:25 localhost kernel: [18493.299567] [<ffffffff81220a1c>] ? xfs_vm_writepage+0x3ae/0x45c
Jan 28 13:47:25 localhost kernel: [18493.299573] [<ffffffff8109cff0>] ? __writepage+0xa/0x21
Jan 28 13:47:25 localhost kernel: [18493.299577] [<ffffffff8109d3dc>] ? write_cache_pages+0x1d5/0x2b4
Jan 28 13:47:25 localhost kernel: [18493.299582] [<ffffffff8109cfe6>] ? global_dirtyable_memory+0x30/0x30
Jan 28 13:47:25 localhost kernel: [18493.299587] [<ffffffff8109d4f3>] ? generic_writepages+0x38/0x54
Jan 28 13:47:25 localhost kernel: [18493.299591] [<ffffffff810e661b>] ? __writeback_single_inode+0x36/0xda
Jan 28 13:47:25 localhost kernel: [18493.299596] [<ffffffff810e73b6>] ? writeback_sb_inodes+0x1b7/0x2d5
Jan 28 13:47:25 localhost kernel: [18493.299600] [<ffffffff810e753d>] ? __writeback_inodes_wb+0x69/0xab
Jan 28 13:47:25 localhost kernel: [18493.299604] [<ffffffff810e7679>] ? wb_writeback+0xfa/0x18c
Jan 28 13:47:25 localhost kernel: [18493.299609] [<ffffffff810e790d>] ? bdi_writeback_workfn+0x14f/0x297
Jan 28 13:47:25 localhost kernel: [18493.299616] [<ffffffff81046074>] ? process_one_work+0x1cb/0x2ea
Jan 28 13:47:25 localhost kernel: [18493.299621] [<ffffffff810465e8>] ? worker_thread+0x1cd/0x2c8
Jan 28 13:47:25 localhost kernel: [18493.299626] [<ffffffff8104641b>] ? rescuer_thread+0x263/0x263
Jan 28 13:47:25 localhost kernel: [18493.299630] [<ffffffff8104ada0>] ? kthread+0xad/0xb5
Jan 28 13:47:25 localhost kernel: [18493.299635] [<ffffffff8104acf3>] ? kthread_freezable_should_stop+0x3b/0x3b
Jan 28 13:47:25 localhost kernel: [18493.299639] [<ffffffff814f510c>] ? ret_from_fork+0x7c/0xb0
Jan 28 13:47:25 localhost kernel: [18493.299643] [<ffffffff8104acf3>] ? kthread_freezable_should_stop+0x3b/0x3b
Jan 28 13:47:25 localhost kernel: [18493.299648] INFO: task kworker/0:1:626 blocked for more than 120 seconds.
Jan 28 13:47:25 localhost kernel: [18493.299650] Not tainted 3.12.4-rikishi #1
Jan 28 13:47:25 localhost kernel: [18493.299652] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 28 13:47:25 localhost kernel: [18493.299655] kworker/0:1 D ffff880223be7808 0 626 2 0x00000000
Jan 28 13:47:25 localhost kernel: [18493.299662] Workqueue: xfs-log/rbd1 xfs_log_worker
Jan 28 13:47:25 localhost kernel: [18493.299664] ffff880223be74f0 0000000000000046 0000000000000000 ffff88009f8a0f60
Jan 28 13:47:25 localhost kernel: [18493.299668] 0000000000011740 ffff88022326bfd8 ffff88022326bfd8 ffff880223be74f0
Jan 28 13:47:25 localhost kernel: [18493.299672] 00000000013f4ac0 0000000000000000 0000000800000008 0000000000001c31
Jan 28 13:47:25 localhost kernel: [18493.299676] Call Trace:
Jan 28 13:47:25 localhost kernel: [18493.299683] [<ffffffff8126796a>] ? xlog_bdstrat+0x34/0x38
Jan 28 13:47:25 localhost kernel: [18493.299686] [<ffffffff81268ebf>] ? xlog_sync+0x262/0x330
Jan 28 13:47:25 localhost kernel: [18493.299690] [<ffffffff81269df2>] ? _xfs_log_force_lsn+0x249/0x288
Jan 28 13:47:25 localhost kernel: [18493.299697] [<ffffffff8105428e>] ? try_to_wake_up+0x1ee/0x1ee
Jan 28 13:47:25 localhost kernel: [18493.299702] [<ffffffff812363fd>] ? xfs_trans_commit+0xd3/0x1cf
Jan 28 13:47:25 localhost kernel: [18493.299706] [<ffffffff81269b8b>] ? xfs_log_worker+0x19/0x37
Jan 28 13:47:25 localhost kernel: [18493.299711] [<ffffffff81046074>] ? process_one_work+0x1cb/0x2ea
Jan 28 13:47:25 localhost kernel: [18493.299716] [<ffffffff81044044>] ? pwq_activate_delayed_work+0x1e/0x28
Jan 28 13:47:25 localhost kernel: [18493.299720] [<ffffffff810465e8>] ? worker_thread+0x1cd/0x2c8
Jan 28 13:47:25 localhost kernel: [18493.299725] [<ffffffff8104641b>] ? rescuer_thread+0x263/0x263
Jan 28 13:47:25 localhost kernel: [18493.299729] [<ffffffff8104ada0>] ? kthread+0xad/0xb5
Jan 28 13:47:25 localhost kernel: [18493.299734] [<ffffffff8104acf3>] ? kthread_freezable_should_stop+0x3b/0x3b
Jan 28 13:47:25 localhost kernel: [18493.299737] [<ffffffff814f510c>] ? ret_from_fork+0x7c/0xb0
Jan 28 13:47:25 localhost kernel: [18493.299741] [<ffffffff8104acf3>] ? kthread_freezable_should_stop+0x3b/0x3b
Jan 28 13:48:39 localhost kernel: [18567.293612] libceph: osd0 down
Jan 28 13:48:39 localhost kernel: [18567.293617] libceph: osd1 down
Jan 28 13:48:39 localhost kernel: [18567.314205] libceph: osd2 up
Jan 28 13:48:39 localhost kernel: [18567.314209] libceph: osd3 up
Jan 28 13:48:39 localhost kernel: [18567.314213] libceph: osd2 weight 0x10000 (in)
Jan 28 13:48:39 localhost kernel: [18567.314215] libceph: osd3 weight 0x10000 (in)
Jan 28 13:48:39 localhost kernel: [18567.454114] libceph: osd4 10.20.30.226:6800 socket closed (con state OPEN)
Jan 28 13:48:41 localhost kernel: [18569.769291] libceph: osd5 10.20.30.226:6803 socket closed (con state OPEN)
Jan 28 13:50:31 localhost kernel: [18679.530417] libceph: osd0 up
Jan 28 13:50:31 localhost kernel: [18679.763596] libceph: osd2 10.20.30.225:6800 socket closed (con state OPEN)
Jan 28 13:50:31 localhost kernel: [18679.789581] libceph: osd4 10.20.30.226:6800 socket closed (con state OPEN)
Jan 28 13:50:31 localhost kernel: [18679.993391] libceph: osd0 10.20.30.224:6800 socket closed (con state OPEN)
Jan 28 13:50:37 localhost kernel: [18686.055568] libceph: osd1 up
Jan 28 13:50:38 localhost kernel: [18686.210263] libceph: osd4 10.20.30.226:6800 socket closed (con state OPEN)
Jan 28 13:51:40 localhost kernel: [18748.457248] libceph: mon0 10.20.30.224:6789 socket closed (con state OPEN)
Jan 28 13:51:40 localhost kernel: [18748.457268] libceph: mon0 10.20.30.224:6789 session lost, hunting for new mon
Jan 28 13:51:40 localhost kernel: [18748.460568] libceph: mon0 10.20.30.224:6789 session established
Jan 28 13:55:25 localhost kernel: [18973.614827] INFO: task dd:9798 blocked for more than 120 seconds.
Jan 28 13:55:25 localhost kernel: [18973.614833] Not tainted 3.12.4-rikishi #1
Jan 28 13:55:25 localhost kernel: [18973.614835] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 28 13:55:25 localhost kernel: [18973.614838] dd D ffff88009c639908 0 9798 3077 0x00000004
Jan 28 13:55:25 localhost kernel: [18973.614844] ffff88009c6395f0 0000000000000082 0000000000000000 ffff8802238f8e60
Jan 28 13:55:25 localhost kernel: [18973.614848] 0000000000011740 ffff8800c4b45fd8 ffff8800c4b45fd8 ffff88009c6395f0
Jan 28 13:55:25 localhost kernel: [18973.614864] 0000000000000000 0000000000000246 0000000000000041 0000000000000246
Jan 28 13:55:25 localhost kernel: [18973.614866] Call Trace:
Jan 28 13:55:25 localhost kernel: [18973.614874] [<ffffffff8104e78c>] ? down_trylock+0x20/0x29
Jan 28 13:55:25 localhost kernel: [18973.614879] [<ffffffff812244fd>] ? xfs_buf_trylock+0xa/0x14
Jan 28 13:55:25 localhost kernel: [18973.614883] [<ffffffff81224610>] ? _xfs_buf_find+0xd8/0x20c
Jan 28 13:55:25 localhost kernel: [18973.614888] [<ffffffff814f170f>] ? schedule_timeout+0x1f/0x176
Jan 28 13:55:25 localhost kernel: [18973.614892] [<ffffffff81236b32>] ? kmem_zone_zalloc+0x9/0x23
Jan 28 13:55:25 localhost kernel: [18973.614895] [<ffffffff81236072>] ? xfs_trans_add_item+0x1b/0x4f
Jan 28 13:55:25 localhost kernel: [18973.614898] [<ffffffff814f299c>] ? __down+0x69/0x96
Jan 28 13:55:25 localhost kernel: [18973.614901] [<ffffffff8104e7ba>] ? down+0x25/0x34
Jan 28 13:55:25 localhost kernel: [18973.614904] [<ffffffff81224536>] ? xfs_buf_lock+0x2f/0x31
Jan 28 13:55:25 localhost kernel: [18973.614908] [<ffffffff812246d8>] ? _xfs_buf_find+0x1a0/0x20c
Jan 28 13:55:25 localhost kernel: [18973.614911] [<ffffffff812247e3>] ? xfs_buf_get_map+0x20/0xfe
Jan 28 13:55:25 localhost kernel: [18973.614914] [<ffffffff81224d6d>] ? xfs_buf_read_map+0x1c/0x8f
Jan 28 13:55:25 localhost kernel: [18973.614919] [<ffffffff8126d8b9>] ? xfs_trans_read_buf_map+0x18c/0x24f
Jan 28 13:55:25 localhost kernel: [18973.614922] [<ffffffff81260689>] ? xfs_imap_to_bp+0x59/0xb7
Jan 28 13:55:25 localhost kernel: [18973.614925] [<ffffffff81260b3b>] ? xfs_iread+0xe5/0x302
Jan 28 13:55:25 localhost kernel: [18973.614928] [<ffffffff81236adf>] ? kmem_zone_alloc+0x5a/0xa4
Jan 28 13:55:25 localhost kernel: [18973.614931] [<ffffffff8122a950>] ? xfs_iget+0x2d4/0x473
Jan 28 13:55:25 localhost kernel: [18973.614934] [<ffffffff8125c151>] ? xfs_ialloc+0xa5/0x5ba
Jan 28 13:55:25 localhost kernel: [18973.614937] [<ffffffff81236b32>] ? kmem_zone_zalloc+0x9/0x23
Jan 28 13:55:25 localhost kernel: [18973.614940] [<ffffffff8125c6cb>] ? xfs_dir_ialloc+0x65/0x23e
Jan 28 13:55:25 localhost kernel: [18973.614942] [<ffffffff8125cc0e>] ? xfs_create+0x30f/0x525
Jan 28 13:55:25 localhost kernel: [18973.614946] [<ffffffff8122e9e8>] ? xfs_vn_mknod+0xc9/0x164
Jan 28 13:55:25 localhost kernel: [18973.614951] [<ffffffff810d2d27>] ? vfs_create+0x5d/0x94
Jan 28 13:55:25 localhost kernel: [18973.614954] [<ffffffff810d5776>] ? do_last.isra.59+0x566/0xa1e
Jan 28 13:55:25 localhost kernel: [18973.614957] [<ffffffff810d2ee5>] ? link_path_walk+0x60/0x7b8
Jan 28 13:55:25 localhost kernel: [18973.614960] [<ffffffff810d5e4b>] ? path_openat+0x21d/0x543
Jan 28 13:55:25 localhost kernel: [18973.614963] [<ffffffff810d643f>] ? do_filp_open+0x2b/0x6f
Jan 28 13:55:25 localhost kernel: [18973.614966] [<ffffffff810df4f6>] ? __alloc_fd+0x58/0xe0
Jan 28 13:55:25 localhost kernel: [18973.614970] [<ffffffff810c99fa>] ? do_sys_open+0x14b/0x1cf
Jan 28 13:55:25 localhost kernel: [18973.614974] [<ffffffff814f51bd>] ? system_call_fastpath+0x1a/0x1f
Jan 28 13:57:25 localhost kernel: [19093.693696] INFO: task dd:9798 blocked for more than 120 seconds.
Jan 28 13:57:25 localhost kernel: [19093.693699] Not tainted 3.12.4-rikishi #1
Jan 28 13:57:25 localhost kernel: [19093.693700] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 28 13:57:25 localhost kernel: [19093.693701] dd D ffff88009c639908 0 9798 3077 0x00000004
Jan 28 13:57:25 localhost kernel: [19093.693704] ffff88009c6395f0 0000000000000082 0000000000000000 ffff8802238f8e60
Jan 28 13:57:25 localhost kernel: [19093.693706] 0000000000011740 ffff8800c4b45fd8 ffff8800c4b45fd8 ffff88009c6395f0
Jan 28 13:57:25 localhost kernel: [19093.693707] 0000000000000000 0000000000000246 0000000000000041 0000000000000246
Jan 28 13:57:25 localhost kernel: [19093.693709] Call Trace:
Jan 28 13:57:25 localhost kernel: [19093.693715] [<ffffffff8104e78c>] ? down_trylock+0x20/0x29
Jan 28 13:57:25 localhost kernel: [19093.693719] [<ffffffff812244fd>] ? xfs_buf_trylock+0xa/0x14
Jan 28 13:57:25 localhost kernel: [19093.693721] [<ffffffff81224610>] ? _xfs_buf_find+0xd8/0x20c
Jan 28 13:57:25 localhost kernel: [19093.693725] [<ffffffff814f170f>] ? schedule_timeout+0x1f/0x176
Jan 28 13:57:25 localhost kernel: [19093.693727] [<ffffffff81236b32>] ? kmem_zone_zalloc+0x9/0x23
Jan 28 13:57:25 localhost kernel: [19093.693729] [<ffffffff81236072>] ? xfs_trans_add_item+0x1b/0x4f
Jan 28 13:57:25 localhost kernel: [19093.693731] [<ffffffff814f299c>] ? __down+0x69/0x96
Jan 28 13:57:25 localhost kernel: [19093.693733] [<ffffffff8104e7ba>] ? down+0x25/0x34
Jan 28 13:57:25 localhost kernel: [19093.693735] [<ffffffff81224536>] ? xfs_buf_lock+0x2f/0x31
Jan 28 13:57:25 localhost kernel: [19093.693737] [<ffffffff812246d8>] ? _xfs_buf_find+0x1a0/0x20c
Jan 28 13:57:25 localhost kernel: [19093.693739] [<ffffffff812247e3>] ? xfs_buf_get_map+0x20/0xfe
Jan 28 13:57:25 localhost kernel: [19093.693742] [<ffffffff81224d6d>] ? xfs_buf_read_map+0x1c/0x8f
Jan 28 13:57:25 localhost kernel: [19093.693745] [<ffffffff8126d8b9>] ? xfs_trans_read_buf_map+0x18c/0x24f
Jan 28 13:57:25 localhost kernel: [19093.693747] [<ffffffff81260689>] ? xfs_imap_to_bp+0x59/0xb7
Jan 28 13:57:25 localhost kernel: [19093.693749] [<ffffffff81260b3b>] ? xfs_iread+0xe5/0x302
Jan 28 13:57:25 localhost kernel: [19093.693751] [<ffffffff81236adf>] ? kmem_zone_alloc+0x5a/0xa4
Jan 28 13:57:25 localhost kernel: [19093.693752] [<ffffffff8122a950>] ? xfs_iget+0x2d4/0x473
Jan 28 13:57:25 localhost kernel: [19093.693755] [<ffffffff8125c151>] ? xfs_ialloc+0xa5/0x5ba
Jan 28 13:57:25 localhost kernel: [19093.693756] [<ffffffff81236b32>] ? kmem_zone_zalloc+0x9/0x23
Jan 28 13:57:25 localhost kernel: [19093.693758] [<ffffffff8125c6cb>] ? xfs_dir_ialloc+0x65/0x23e
Jan 28 13:57:25 localhost kernel: [19093.693759] [<ffffffff8125cc0e>] ? xfs_create+0x30f/0x525
Jan 28 13:57:25 localhost kernel: [19093.693762] [<ffffffff8122e9e8>] ? xfs_vn_mknod+0xc9/0x164
Jan 28 13:57:25 localhost kernel: [19093.693765] [<ffffffff810d2d27>] ? vfs_create+0x5d/0x94
Jan 28 13:57:25 localhost kernel: [19093.693766] [<ffffffff810d5776>] ? do_last.isra.59+0x566/0xa1e
Jan 28 13:57:25 localhost kernel: [19093.693768] [<ffffffff810d2ee5>] ? link_path_walk+0x60/0x7b8
Jan 28 13:57:25 localhost kernel: [19093.693770] [<ffffffff810d5e4b>] ? path_openat+0x21d/0x543
Jan 28 13:57:25 localhost kernel: [19093.693772] [<ffffffff810d643f>] ? do_filp_open+0x2b/0x6f
Jan 28 13:57:25 localhost kernel: [19093.693774] [<ffffffff810df4f6>] ? __alloc_fd+0x58/0xe0
Jan 28 13:57:25 localhost kernel: [19093.693777] [<ffffffff810c99fa>] ? do_sys_open+0x14b/0x1cf
Jan 28 13:57:25 localhost kernel: [19093.693779] [<ffffffff814f51bd>] ? system_call_fastpath+0x1a/0x1f
Jan 28 13:59:25 localhost kernel: [19213.772624] INFO: task dd:9798 blocked for more than 120 seconds.
Jan 28 13:59:25 localhost kernel: [19213.772627] Not tainted 3.12.4-rikishi #1
Jan 28 13:59:25 localhost kernel: [19213.772627] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 28 13:59:25 localhost kernel: [19213.772629] dd D ffff88009c639908 0 9798 3077 0x00000004
Jan 28 13:59:25 localhost kernel: [19213.772632] ffff88009c6395f0 0000000000000082 0000000000000000 ffff8802238f8e60
Jan 28 13:59:25 localhost kernel: [19213.772634] 0000000000011740 ffff8800c4b45fd8 ffff8800c4b45fd8 ffff88009c6395f0
Jan 28 13:59:25 localhost kernel: [19213.772636] 0000000000000000 0000000000000246 0000000000000041 0000000000000246
Jan 28 13:59:25 localhost kernel: [19213.772637] Call Trace:
Jan 28 13:59:25 localhost kernel: [19213.772661] [<ffffffff8104e78c>] ? down_trylock+0x20/0x29
Jan 28 13:59:25 localhost kernel: [19213.772667] [<ffffffff812244fd>] ? xfs_buf_trylock+0xa/0x14
Jan 28 13:59:25 localhost kernel: [19213.772672] [<ffffffff81224610>] ? _xfs_buf_find+0xd8/0x20c
Jan 28 13:59:25 localhost kernel: [19213.772678] [<ffffffff814f170f>] ? schedule_timeout+0x1f/0x176
Jan 28 13:59:25 localhost kernel: [19213.772682] [<ffffffff81236b32>] ? kmem_zone_zalloc+0x9/0x23
Jan 28 13:59:25 localhost kernel: [19213.772685] [<ffffffff81236072>] ? xfs_trans_add_item+0x1b/0x4f
Jan 28 13:59:25 localhost kernel: [19213.772690] [<ffffffff814f299c>] ? __down+0x69/0x96
Jan 28 13:59:25 localhost kernel: [19213.772694] [<ffffffff8104e7ba>] ? down+0x25/0x34
Jan 28 13:59:25 localhost kernel: [19213.772698] [<ffffffff81224536>] ? xfs_buf_lock+0x2f/0x31
Jan 28 13:59:25 localhost kernel: [19213.772702] [<ffffffff812246d8>] ? _xfs_buf_find+0x1a0/0x20c
Jan 28 13:59:25 localhost kernel: [19213.772706] [<ffffffff812247e3>] ? xfs_buf_get_map+0x20/0xfe
Jan 28 13:59:25 localhost kernel: [19213.772711] [<ffffffff81224d6d>] ? xfs_buf_read_map+0x1c/0x8f
Jan 28 13:59:25 localhost kernel: [19213.772716] [<ffffffff8126d8b9>] ? xfs_trans_read_buf_map+0x18c/0x24f
Jan 28 13:59:25 localhost kernel: [19213.772721] [<ffffffff81260689>] ? xfs_imap_to_bp+0x59/0xb7
Jan 28 13:59:25 localhost kernel: [19213.772724] [<ffffffff81260b3b>] ? xfs_iread+0xe5/0x302
Jan 28 13:59:25 localhost kernel: [19213.772727] [<ffffffff81236adf>] ? kmem_zone_alloc+0x5a/0xa4
Jan 28 13:59:25 localhost kernel: [19213.772731] [<ffffffff8122a950>] ? xfs_iget+0x2d4/0x473
Jan 28 13:59:25 localhost kernel: [19213.772735] [<ffffffff8125c151>] ? xfs_ialloc+0xa5/0x5ba
Jan 28 13:59:25 localhost kernel: [19213.772738] [<ffffffff81236b32>] ? kmem_zone_zalloc+0x9/0x23
Jan 28 13:59:25 localhost kernel: [19213.772742] [<ffffffff8125c6cb>] ? xfs_dir_ialloc+0x65/0x23e
Jan 28 13:59:25 localhost kernel: [19213.772745] [<ffffffff8125cc0e>] ? xfs_create+0x30f/0x525
Jan 28 13:59:25 localhost kernel: [19213.772750] [<ffffffff8122e9e8>] ? xfs_vn_mknod+0xc9/0x164
Jan 28 13:59:25 localhost kernel: [19213.772755] [<ffffffff810d2d27>] ? vfs_create+0x5d/0x94
Jan 28 13:59:25 localhost kernel: [19213.772759] [<ffffffff810d5776>] ? do_last.isra.59+0x566/0xa1e
Jan 28 13:59:25 localhost kernel: [19213.772763] [<ffffffff810d2ee5>] ? link_path_walk+0x60/0x7b8
Jan 28 13:59:25 localhost kernel: [19213.772767] [<ffffffff810d5e4b>] ? path_openat+0x21d/0x543
Jan 28 13:59:25 localhost kernel: [19213.772770] [<ffffffff810d643f>] ? do_filp_open+0x2b/0x6f
Jan 28 13:59:25 localhost kernel: [19213.772774] [<ffffffff810df4f6>] ? __alloc_fd+0x58/0xe0
Jan 28 13:59:25 localhost kernel: [19213.772780] [<ffffffff810c99fa>] ? do_sys_open+0x14b/0x1cf
Jan 28 13:59:25 localhost kernel: [19213.772784] [<ffffffff814f51bd>] ? system_call_fastpath+0x1a/0x1f
Jan 28 14:01:25 localhost kernel: [19333.851495] INFO: task dd:9798 blocked for more than 120 seconds.
Jan 28 14:01:25 localhost kernel: [19333.851503] Not tainted 3.12.4-rikishi #1
Jan 28 14:01:25 localhost kernel: [19333.851505] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 28 14:01:25 localhost kernel: [19333.851508] dd D ffff88009c639908 0 9798 3077 0x00000004
Jan 28 14:01:25 localhost kernel: [19333.851515] ffff88009c6395f0 0000000000000082 0000000000000000 ffff8802238f8e60
Jan 28 14:01:25 localhost kernel: [19333.851520] 0000000000011740 ffff8800c4b45fd8 ffff8800c4b45fd8 ffff88009c6395f0
Jan 28 14:01:25 localhost kernel: [19333.851524] 0000000000000000 0000000000000246 0000000000000041 0000000000000246
Jan 28 14:01:25 localhost kernel: [19333.851529] Call Trace:
Jan 28 14:01:25 localhost kernel: [19333.851541] [<ffffffff8104e78c>] ? down_trylock+0x20/0x29
Jan 28 14:01:25 localhost kernel: [19333.851549] [<ffffffff812244fd>] ? xfs_buf_trylock+0xa/0x14
Jan 28 14:01:25 localhost kernel: [19333.851554] [<ffffffff81224610>] ? _xfs_buf_find+0xd8/0x20c
Jan 28 14:01:25 localhost kernel: [19333.851562] [<ffffffff814f170f>] ? schedule_timeout+0x1f/0x176
Jan 28 14:01:25 localhost kernel: [19333.851567] [<ffffffff81236b32>] ? kmem_zone_zalloc+0x9/0x23
Jan 28 14:01:25 localhost kernel: [19333.851572] [<ffffffff81236072>] ? xfs_trans_add_item+0x1b/0x4f
Jan 28 14:01:25 localhost kernel: [19333.851577] [<ffffffff814f299c>] ? __down+0x69/0x96
Jan 28 14:01:25 localhost kernel: [19333.851583] [<ffffffff8104e7ba>] ? down+0x25/0x34
Jan 28 14:01:25 localhost kernel: [19333.851588] [<ffffffff81224536>] ? xfs_buf_lock+0x2f/0x31
Jan 28 14:01:25 localhost kernel: [19333.851593] [<ffffffff812246d8>] ? _xfs_buf_find+0x1a0/0x20c
Jan 28 14:01:25 localhost kernel: [19333.851599] [<ffffffff812247e3>] ? xfs_buf_get_map+0x20/0xfe
Jan 28 14:01:25 localhost kernel: [19333.851605] [<ffffffff81224d6d>] ? xfs_buf_read_map+0x1c/0x8f
Jan 28 14:01:25 localhost kernel: [19333.851612] [<ffffffff8126d8b9>] ? xfs_trans_read_buf_map+0x18c/0x24f
Jan 28 14:01:25 localhost kernel: [19333.851617] [<ffffffff81260689>] ? xfs_imap_to_bp+0x59/0xb7
Jan 28 14:01:25 localhost kernel: [19333.851622] [<ffffffff81260b3b>] ? xfs_iread+0xe5/0x302
Jan 28 14:01:25 localhost kernel: [19333.851626] [<ffffffff81236adf>] ? kmem_zone_alloc+0x5a/0xa4
Jan 28 14:01:25 localhost kernel: [19333.851630] [<ffffffff8122a950>] ? xfs_iget+0x2d4/0x473
Jan 28 14:01:25 localhost kernel: [19333.851635] [<ffffffff8125c151>] ? xfs_ialloc+0xa5/0x5ba
Jan 28 14:01:25 localhost kernel: [19333.851639] [<ffffffff81236b32>] ? kmem_zone_zalloc+0x9/0x23
Jan 28 14:01:25 localhost kernel: [19333.851643] [<ffffffff8125c6cb>] ? xfs_dir_ialloc+0x65/0x23e
Jan 28 14:01:25 localhost kernel: [19333.851648] [<ffffffff8125cc0e>] ? xfs_create+0x30f/0x525
Jan 28 14:01:25 localhost kernel: [19333.851654] [<ffffffff8122e9e8>] ? xfs_vn_mknod+0xc9/0x164
Jan 28 14:01:25 localhost kernel: [19333.851661] [<ffffffff810d2d27>] ? vfs_create+0x5d/0x94
Jan 28 14:01:25 localhost kernel: [19333.851665] [<ffffffff810d5776>] ? do_last.isra.59+0x566/0xa1e
Jan 28 14:01:25 localhost kernel: [19333.851670] [<ffffffff810d2ee5>] ? link_path_walk+0x60/0x7b8
Jan 28 14:01:25 localhost kernel: [19333.851675] [<ffffffff810d5e4b>] ? path_openat+0x21d/0x543
Jan 28 14:01:25 localhost kernel: [19333.851679] [<ffffffff810d643f>] ? do_filp_open+0x2b/0x6f
Jan 28 14:01:25 localhost kernel: [19333.851684] [<ffffffff810df4f6>] ? __alloc_fd+0x58/0xe0
Jan 28 14:01:25 localhost kernel: [19333.851691] [<ffffffff810c99fa>] ? do_sys_open+0x14b/0x1cf
Jan 28 14:01:25 localhost kernel: [19333.851696] [<ffffffff814f51bd>] ? system_call_fastpath+0x1a/0x1f
Jan 28 14:03:25 localhost kernel: [19453.930380] INFO: task dd:9798 blocked for more than 120 seconds.
Jan 28 14:03:25 localhost kernel: [19453.930387] Not tainted 3.12.4-rikishi #1
Jan 28 14:03:25 localhost kernel: [19453.930389] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 28 14:03:25 localhost kernel: [19453.930392] dd D ffff88009c639908 0 9798 3077 0x00000004
Jan 28 14:03:25 localhost kernel: [19453.930399] ffff88009c6395f0 0000000000000082 0000000000000000 ffff8802238f8e60
Jan 28 14:03:25 localhost kernel: [19453.930403] 0000000000011740 ffff8800c4b45fd8 ffff8800c4b45fd8 ffff88009c6395f0
Jan 28 14:03:25 localhost kernel: [19453.930407] 0000000000000000 0000000000000246 0000000000000041 0000000000000246
Jan 28 14:03:25 localhost kernel: [19453.930412] Call Trace:
Jan 28 14:03:25 localhost kernel: [19453.930437] [<ffffffff8104e78c>] ? down_trylock+0x20/0x29
Jan 28 14:03:25 localhost kernel: [19453.930441] [<ffffffff812244fd>] ? xfs_buf_trylock+0xa/0x14
Jan 28 14:03:25 localhost kernel: [19453.930443] [<ffffffff81224610>] ? _xfs_buf_find+0xd8/0x20c
Jan 28 14:03:25 localhost kernel: [19453.930446] [<ffffffff814f170f>] ? schedule_timeout+0x1f/0x176
Jan 28 14:03:25 localhost kernel: [19453.930449] [<ffffffff81236b32>] ? kmem_zone_zalloc+0x9/0x23
Jan 28 14:03:25 localhost kernel: [19453.930450] [<ffffffff81236072>] ? xfs_trans_add_item+0x1b/0x4f
Jan 28 14:03:25 localhost kernel: [19453.930452] [<ffffffff814f299c>] ? __down+0x69/0x96
Jan 28 14:03:25 localhost kernel: [19453.930454] [<ffffffff8104e7ba>] ? down+0x25/0x34
Jan 28 14:03:25 localhost kernel: [19453.930456] [<ffffffff81224536>] ? xfs_buf_lock+0x2f/0x31
Jan 28 14:03:25 localhost kernel: [19453.930458] [<ffffffff812246d8>] ? _xfs_buf_find+0x1a0/0x20c
Jan 28 14:03:25 localhost kernel: [19453.930460] [<ffffffff812247e3>] ? xfs_buf_get_map+0x20/0xfe
Jan 28 14:03:25 localhost kernel: [19453.930463] [<ffffffff81224d6d>] ? xfs_buf_read_map+0x1c/0x8f
Jan 28 14:03:25 localhost kernel: [19453.930466] [<ffffffff8126d8b9>] ? xfs_trans_read_buf_map+0x18c/0x24f
Jan 28 14:03:25 localhost kernel: [19453.930468] [<ffffffff81260689>] ? xfs_imap_to_bp+0x59/0xb7
Jan 28 14:03:25 localhost kernel: [19453.930470] [<ffffffff81260b3b>] ? xfs_iread+0xe5/0x302
Jan 28 14:03:25 localhost kernel: [19453.930471] [<ffffffff81236adf>] ? kmem_zone_alloc+0x5a/0xa4
Jan 28 14:03:25 localhost kernel: [19453.930473] [<ffffffff8122a950>] ? xfs_iget+0x2d4/0x473
Jan 28 14:03:25 localhost kernel: [19453.930475] [<ffffffff8125c151>] ? xfs_ialloc+0xa5/0x5ba
Jan 28 14:03:25 localhost kernel: [19453.930476] [<ffffffff81236b32>] ? kmem_zone_zalloc+0x9/0x23
Jan 28 14:03:25 localhost kernel: [19453.930478] [<ffffffff8125c6cb>] ? xfs_dir_ialloc+0x65/0x23e
Jan 28 14:03:25 localhost kernel: [19453.930479] [<ffffffff8125cc0e>] ? xfs_create+0x30f/0x525
Jan 28 14:03:25 localhost kernel: [19453.930481] [<ffffffff8122e9e8>] ? xfs_vn_mknod+0xc9/0x164
Jan 28 14:03:25 localhost kernel: [19453.930484] [<ffffffff810d2d27>] ? vfs_create+0x5d/0x94
Jan 28 14:03:25 localhost kernel: [19453.930486] [<ffffffff810d5776>] ? do_last.isra.59+0x566/0xa1e
Jan 28 14:03:25 localhost kernel: [19453.930488] [<ffffffff810d2ee5>] ? link_path_walk+0x60/0x7b8
Jan 28 14:03:25 localhost kernel: [19453.930490] [<ffffffff810d5e4b>] ? path_openat+0x21d/0x543
Jan 28 14:03:25 localhost kernel: [19453.930491] [<ffffffff810d643f>] ? do_filp_open+0x2b/0x6f
Jan 28 14:03:25 localhost kernel: [19453.930493] [<ffffffff810df4f6>] ? __alloc_fd+0x58/0xe0
Jan 28 14:03:25 localhost kernel: [19453.930497] [<ffffffff810c99fa>] ? do_sys_open+0x14b/0x1cf
Jan 28 14:03:25 localhost kernel: [19453.930498] [<ffffffff814f51bd>] ? system_call_fastpath+0x1a/0x1f
Jan 28 14:05:25 localhost kernel: [19574.009221] INFO: task dd:9798 blocked for more than 120 seconds.
Jan 28 14:05:25 localhost kernel: [19574.009228] Not tainted 3.12.4-rikishi #1
Jan 28 14:05:25 localhost kernel: [19574.009231] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 28 14:05:25 localhost kernel: [19574.009233] dd D ffff88009c639908 0 9798 3077 0x00000004
Jan 28 14:05:25 localhost kernel: [19574.009240] ffff88009c6395f0 0000000000000082 0000000000000000 ffff8802238f8e60
Jan 28 14:05:25 localhost kernel: [19574.009245] 0000000000011740 ffff8800c4b45fd8 ffff8800c4b45fd8 ffff88009c6395f0
Jan 28 14:05:25 localhost kernel: [19574.009249] 0000000000000000 0000000000000246 0000000000000041 0000000000000246
Jan 28 14:05:25 localhost kernel: [19574.009253] Call Trace:
Jan 28 14:05:25 localhost kernel: [19574.009266] [<ffffffff8104e78c>] ? down_trylock+0x20/0x29
Jan 28 14:05:25 localhost kernel: [19574.009273] [<ffffffff812244fd>] ? xfs_buf_trylock+0xa/0x14
Jan 28 14:05:25 localhost kernel: [19574.009279] [<ffffffff81224610>] ? _xfs_buf_find+0xd8/0x20c
Jan 28 14:05:25 localhost kernel: [19574.009286] [<ffffffff814f170f>] ? schedule_timeout+0x1f/0x176
Jan 28 14:05:25 localhost kernel: [19574.009291] [<ffffffff81236b32>] ? kmem_zone_zalloc+0x9/0x23
Jan 28 14:05:25 localhost kernel: [19574.009296] [<ffffffff81236072>] ? xfs_trans_add_item+0x1b/0x4f
Jan 28 14:05:25 localhost kernel: [19574.009301] [<ffffffff814f299c>] ? __down+0x69/0x96
Jan 28 14:05:25 localhost kernel: [19574.009306] [<ffffffff8104e7ba>] ? down+0x25/0x34
Jan 28 14:05:25 localhost kernel: [19574.009311] [<ffffffff81224536>] ? xfs_buf_lock+0x2f/0x31
Jan 28 14:05:25 localhost kernel: [19574.009316] [<ffffffff812246d8>] ? _xfs_buf_find+0x1a0/0x20c
Jan 28 14:05:25 localhost kernel: [19574.009322] [<ffffffff812247e3>] ? xfs_buf_get_map+0x20/0xfe
Jan 28 14:05:25 localhost kernel: [19574.009328] [<ffffffff81224d6d>] ? xfs_buf_read_map+0x1c/0x8f
Jan 28 14:05:25 localhost kernel: [19574.009334] [<ffffffff8126d8b9>] ? xfs_trans_read_buf_map+0x18c/0x24f
Jan 28 14:05:25 localhost kernel: [19574.009340] [<ffffffff81260689>] ? xfs_imap_to_bp+0x59/0xb7
Jan 28 14:05:25 localhost kernel: [19574.009344] [<ffffffff81260b3b>] ? xfs_iread+0xe5/0x302
Jan 28 14:05:25 localhost kernel: [19574.009348] [<ffffffff81236adf>] ? kmem_zone_alloc+0x5a/0xa4
Jan 28 14:05:25 localhost kernel: [19574.009353] [<ffffffff8122a950>] ? xfs_iget+0x2d4/0x473
Jan 28 14:05:25 localhost kernel: [19574.009358] [<ffffffff8125c151>] ? xfs_ialloc+0xa5/0x5ba
Jan 28 14:05:25 localhost kernel: [19574.009362] [<ffffffff81236b32>] ? kmem_zone_zalloc+0x9/0x23
Jan 28 14:05:25 localhost kernel: [19574.009366] [<ffffffff8125c6cb>] ? xfs_dir_ialloc+0x65/0x23e
Jan 28 14:05:25 localhost kernel: [19574.009370] [<ffffffff8125cc0e>] ? xfs_create+0x30f/0x525
Jan 28 14:05:25 localhost kernel: [19574.009376] [<ffffffff8122e9e8>] ? xfs_vn_mknod+0xc9/0x164
Jan 28 14:05:25 localhost kernel: [19574.009382] [<ffffffff810d2d27>] ? vfs_create+0x5d/0x94
Jan 28 14:05:25 localhost kernel: [19574.009386] [<ffffffff810d5776>] ? do_last.isra.59+0x566/0xa1e
Jan 28 14:05:25 localhost kernel: [19574.009392] [<ffffffff810d2ee5>] ? link_path_walk+0x60/0x7b8
Jan 28 14:05:25 localhost kernel: [19574.009396] [<ffffffff810d5e4b>] ? path_openat+0x21d/0x543
Jan 28 14:05:25 localhost kernel: [19574.009400] [<ffffffff810d643f>] ? do_filp_open+0x2b/0x6f
Jan 28 14:05:25 localhost kernel: [19574.009405] [<ffffffff810df4f6>] ? __alloc_fd+0x58/0xe0
Jan 28 14:05:25 localhost kernel: [19574.009412] [<ffffffff810c99fa>] ? do_sys_open+0x14b/0x1cf
Jan 28 14:05:25 localhost kernel: [19574.009416] [<ffffffff814f51bd>] ? system_call_fastpath+0x1a/0x1f
Jan 28 14:05:25 localhost kernel: [19574.009425] INFO: task kio_trash:22958 blocked for more than 120 seconds.
Jan 28 14:05:25 localhost kernel: [19574.009428] Not tainted 3.12.4-rikishi #1
Jan 28 14:05:25 localhost kernel: [19574.009430] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 28 14:05:25 localhost kernel: [19574.009432] kio_trash D ffff8801dbf20c68 0 22958 2882 0x00000000
Jan 28 14:05:25 localhost kernel: [19574.009436] ffff8801dbf20950 0000000000000082 ffff880222f3f000 ffffffff81812450
Jan 28 14:05:25 localhost kernel: [19574.009440] 0000000000011740 ffff8801f5ceffd8 ffff8801f5ceffd8 ffff8801dbf20950
Jan 28 14:05:25 localhost kernel: [19574.009444] 00000000a4efdcd8 ffff880100000000 ffff880000000001 0000000000000000
Jan 28 14:05:25 localhost kernel: [19574.009448] Call Trace:
Jan 28 14:05:25 localhost kernel: [19574.009453] [<ffffffff810d9bf9>] ? __d_instantiate+0x16/0xbf
Jan 28 14:05:25 localhost kernel: [19574.009459] [<ffffffff810d2680>] ? lookup_fast+0xe2/0x21b
Jan 28 14:05:25 localhost kernel: [19574.009464] [<ffffffff814f38bc>] ? schedule_preempt_disabled+0x6/0x8
Jan 28 14:05:25 localhost kernel: [19574.009469] [<ffffffff814f2650>] ? __mutex_lock_slowpath+0x12c/0x17a
Jan 28 14:05:25 localhost kernel: [19574.009474] [<ffffffff814f26ac>] ? mutex_lock+0xe/0x1d
Jan 28 14:05:25 localhost kernel: [19574.009479] [<ffffffff810d27e6>] ? lookup_slow+0x2d/0xa2
Jan 28 14:05:25 localhost kernel: [19574.009484] [<ffffffff810d3c98>] ? path_lookupat+0xfe/0x69a
Jan 28 14:05:25 localhost kernel: [19574.009490] [<ffffffff810d4252>] ? filename_lookup.isra.48+0x1e/0x5e
Jan 28 14:05:25 localhost kernel: [19574.009493] [<ffffffff810d6386>] ? user_path_at_empty+0x48/0x7d
Jan 28 14:05:25 localhost kernel: [19574.009499] [<ffffffff8102b433>] ? __do_page_fault+0x373/0x404
Jan 28 14:05:25 localhost kernel: [19574.009504] [<ffffffff810ce022>] ? vfs_fstatat+0x3e/0x8d
Jan 28 14:05:25 localhost kernel: [19574.009508] [<ffffffff810ce1af>] ? SyS_newlstat+0x12/0x2d
Jan 28 14:05:25 localhost kernel: [19574.009514] [<ffffffff810ca721>] ? vfs_write+0x11d/0x162
Jan 28 14:05:25 localhost kernel: [19574.009519] [<ffffffff814f4c88>] ? page_fault+0x28/0x30
Jan 28 14:05:25 localhost kernel: [19574.009526] [<ffffffff81086dc7>] ? from_kuid_munged+0x5/0x10
Jan 28 14:05:25 localhost kernel: [19574.009531] [<ffffffff810421fe>] ? sys_getuid+0x1d/0x22
Jan 28 14:05:25 localhost kernel: [19574.009535] [<ffffffff814f51bd>] ? system_call_fastpath+0x1a/0x1f
Jan 28 14:05:38 localhost kernel: [19587.729713] libceph: osd2 10.20.30.225:6800 socket closed (con state OPEN)
Jan 28 14:07:25 localhost kernel: [19694.088116] INFO: task dd:9798 blocked for more than 120 seconds.
Jan 28 14:07:25 localhost kernel: [19694.088122] Not tainted 3.12.4-rikishi #1
Jan 28 14:07:25 localhost kernel: [19694.088125] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 28 14:07:25 localhost kernel: [19694.088128] dd D ffff88009c639908 0 9798 3077 0x00000004
Jan 28 14:07:25 localhost kernel: [19694.088135] ffff88009c6395f0 0000000000000082 0000000000000000 ffff8802238f8e60
Jan 28 14:07:25 localhost kernel: [19694.088140] 0000000000011740 ffff8800c4b45fd8 ffff8800c4b45fd8 ffff88009c6395f0
Jan 28 14:07:25 localhost kernel: [19694.088144] 0000000000000000 0000000000000246 0000000000000041 0000000000000246
Jan 28 14:07:25 localhost kernel: [19694.088148] Call Trace:
Jan 28 14:07:25 localhost kernel: [19694.088161] [<ffffffff8104e78c>] ? down_trylock+0x20/0x29
Jan 28 14:07:25 localhost kernel: [19694.088169] [<ffffffff812244fd>] ? xfs_buf_trylock+0xa/0x14
Jan 28 14:07:25 localhost kernel: [19694.088174] [<ffffffff81224610>] ? _xfs_buf_find+0xd8/0x20c
Jan 28 14:07:25 localhost kernel: [19694.088182] [<ffffffff814f170f>] ? schedule_timeout+0x1f/0x176
Jan 28 14:07:25 localhost kernel: [19694.088187] [<ffffffff81236b32>] ? kmem_zone_zalloc+0x9/0x23
Jan 28 14:07:25 localhost kernel: [19694.088191] [<ffffffff81236072>] ? xfs_trans_add_item+0x1b/0x4f
Jan 28 14:07:25 localhost kernel: [19694.088196] [<ffffffff814f299c>] ? __down+0x69/0x96
Jan 28 14:07:25 localhost kernel: [19694.088202] [<ffffffff8104e7ba>] ? down+0x25/0x34
Jan 28 14:07:25 localhost kernel: [19694.088207] [<ffffffff81224536>] ? xfs_buf_lock+0x2f/0x31
Jan 28 14:07:25 localhost kernel: [19694.088212] [<ffffffff812246d8>] ? _xfs_buf_find+0x1a0/0x20c
Jan 28 14:07:25 localhost kernel: [19694.088217] [<ffffffff812247e3>] ? xfs_buf_get_map+0x20/0xfe
Jan 28 14:07:25 localhost kernel: [19694.088223] [<ffffffff81224d6d>] ? xfs_buf_read_map+0x1c/0x8f
Jan 28 14:07:25 localhost kernel: [19694.088230] [<ffffffff8126d8b9>] ? xfs_trans_read_buf_map+0x18c/0x24f
Jan 28 14:07:25 localhost kernel: [19694.088235] [<ffffffff81260689>] ? xfs_imap_to_bp+0x59/0xb7
Jan 28 14:07:25 localhost kernel: [19694.088240] [<ffffffff81260b3b>] ? xfs_iread+0xe5/0x302
Jan 28 14:07:25 localhost kernel: [19694.088244] [<ffffffff81236adf>] ? kmem_zone_alloc+0x5a/0xa4
Jan 28 14:07:25 localhost kernel: [19694.088248] [<ffffffff8122a950>] ? xfs_iget+0x2d4/0x473
Jan 28 14:07:25 localhost kernel: [19694.088253] [<ffffffff8125c151>] ? xfs_ialloc+0xa5/0x5ba
Jan 28 14:07:25 localhost kernel: [19694.088257] [<ffffffff81236b32>] ? kmem_zone_zalloc+0x9/0x23
Jan 28 14:07:25 localhost kernel: [19694.088261] [<ffffffff8125c6cb>] ? xfs_dir_ialloc+0x65/0x23e
Jan 28 14:07:25 localhost kernel: [19694.088265] [<ffffffff8125cc0e>] ? xfs_create+0x30f/0x525
Jan 28 14:07:25 localhost kernel: [19694.088271] [<ffffffff8122e9e8>] ? xfs_vn_mknod+0xc9/0x164
Jan 28 14:07:25 localhost kernel: [19694.088278] [<ffffffff810d2d27>] ? vfs_create+0x5d/0x94
Jan 28 14:07:25 localhost kernel: [19694.088282] [<ffffffff810d5776>] ? do_last.isra.59+0x566/0xa1e
Jan 28 14:07:25 localhost kernel: [19694.088287] [<ffffffff810d2ee5>] ? link_path_walk+0x60/0x7b8
Jan 28 14:07:25 localhost kernel: [19694.088292] [<ffffffff810d5e4b>] ? path_openat+0x21d/0x543
Jan 28 14:07:25 localhost kernel: [19694.088296] [<ffffffff810d643f>] ? do_filp_open+0x2b/0x6f
Jan 28 14:07:25 localhost kernel: [19694.088301] [<ffffffff810df4f6>] ? __alloc_fd+0x58/0xe0
Jan 28 14:07:25 localhost kernel: [19694.088307] [<ffffffff810c99fa>] ? do_sys_open+0x14b/0x1cf
Jan 28 14:07:25 localhost kernel: [19694.088312] [<ffffffff814f51bd>] ? system_call_fastpath+0x1a/0x1f
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
