Had some controller issues but believe we had those ironed out before this
recent breakage. I had reformatted both 48TB LUNs on this test box with -f
and defaults, and fired up the test app again. Throughput was fantastic
with no dropped application buffers for ~30 minutes, and IO times were 1.x
ms max, suggesting all the writes were acked by controller cache. The app
was running perfectly. Then it hanged and spit out an internal error msg:
Failed to reopen file /mnt/VOL1/sg-04/str-0015/f-0000000005
I have many call traces in dmesg, most are XFS, pasted below. I found no
SCSI I/O errors in dmesg. Application is submitting writes via libaio and
using O_DIRECT, writing to preallocated files. I don't see any errors in
the storage controller log and everything seems to check out there. Using
noop elevator, mpt2sas, no multipathing. The application process is hung
in d state and kill -9 won't get rid of it. I can't unmount the hosed up
filesystem. Any ideas?
# ls -la /mnt/VOL1
ls: cannot access /mnt/VOL1: Input/output error
# dd if=/mnt/VOL1 of=/dev/null bs=1M count=16
dd: opening `/mnt/VOL1': Input/output error
# dd if=/dev/sdd of=/dev/null bs=1M count=16
16+0 records in
16+0 records out
16777216 bytes (17 MB) copied, 1.10989 s, 15.1 MB/s
# ls -la /mnt/
ls: cannot access /mnt/VOL1: Input/output error
total 8
drwxrwxrwt 6 root root 120 Aug 25 17:59 .
drwxr-xr-x 21 root root 4096 Jul 15 09:39 ..
drwxrwxrwt 3 root root 80 Aug 25 16:52 initramfs
drwxr-xr-x 3 root root 4096 Apr 24 04:57 scratch
drwxrwxrwx 5 root root 58 Aug 25 22:06 VOL0
d????????? ? ? ? ? ? VOL1
# umount /mnt/VOL1
umount: /mnt/VOL1: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
Kernel 3.4.26
xfs_repair version 3.1.4
2 socket, 20 core Xeon(R) CPU E5-2650 v2 @ 2.60GHz
MemTotal: 264573184 kB
MemFree: 246519624 kB
Buffers: 16820 kB
Cached: 78120 kB
SwapCached: 0 kB
Active: 13130040 kB
Inactive: 75352 kB
Active(anon): 13110512 kB
Inactive(anon): 276 kB
Active(file): 19528 kB
Inactive(file): 75076 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 13126968 kB
Mapped: 11848 kB
Shmem: 340 kB
Slab: 208476 kB
SReclaimable: 118240 kB
SUnreclaim: 90236 kB
KernelStack: 3976 kB
PageTables: 32840 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 132286592 kB
Committed_AS: 25044716 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 1050608 kB
VmallocChunk: 34358680400 kB
AnonHugePages: 13078528 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 5056 kB
DirectMap2M: 2045952 kB
DirectMap1G: 266338304 kB
/dev/sdc /mnt/VOL0 xfs
rw,noatime,nodiratime,attr2,nobarrier,inode64,noquota 0 0
/dev/sdd /mnt/VOL1 xfs
rw,noatime,nodiratime,attr2,nobarrier,inode64,noquota 0 0
major minor #blocks name
8 32 46837141504 sdc
8 48 46837141504 sdd
2x hardware RAID5 LUNs - 64KB su, 768KB sw
2 controllers, one per LUN, 3GB cache each, write back, FPGA RAID engine
No LVM. LUNs directly formatted with XFS
26 Seagate SAS 3.5" 7.2K drives, 13 per RAID5, drive caches disabled
My counterpart had what I'd guess is this same problem on the full test
rig,
which has 16 of these LUNS, 208 drives total. He was also testing with
libaio.
I have not dug into the logs on that host as of yet. Its config is
identical
to this box but for the number of LUNs, drives.
[22635.102013] INFO: task kworker/7:0:45 blocked for more than 120
seconds.
[22635.102016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[22635.102018] kworker/7:0 D ffff8840666c0b08 0 45 2
0x00000000
[22635.102021] ffff8840666e7bd0 0000000000000046 ffff883f7c02d000
ffff8840666f5180
[22635.102026] ffff8840666e7b80 0000000000000206 00000000000122c0
00000000000122c0
[22635.102030] ffff8840666e7fd8 ffff8840666e6000 00000000000122c0
ffff8840666e6000
[22635.102033] Call Trace:
[22635.102041] [<ffffffff814f5fd7>] schedule+0x64/0x66
[22635.102044] [<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
[22635.102047] [<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
[22635.102051] [<ffffffff81261913>]
call_rwsem_down_write_failed+0x13/0x20
[22635.102055] [<ffffffff814f6a92>] ?
_raw_spin_unlock_irqrestore+0x30/0x3d
[22635.102058] [<ffffffff814f5458>] ? down_write+0x25/0x27
[22635.102070] [<ffffffffa01b35e4>] xfs_ilock+0x4f/0xb4 [xfs]
[22635.102077] [<ffffffffa01b5ab0>] xfs_iomap_write_unwritten+0x1b3/0x32e
[xfs]
[22635.102080] [<ffffffff814f6a92>] ?
_raw_spin_unlock_irqrestore+0x30/0x3d
[22635.102084] [<ffffffffa01ab3d2>] ? xfs_setfilesize+0x128/0x128 [xfs]
[22635.102088] [<ffffffff810bc602>] ? mempool_free+0x73/0x78
[22635.102093] [<ffffffffa01ab3d2>] ? xfs_setfilesize+0x128/0x128 [xfs]
[22635.102098] [<ffffffffa01ab45b>] xfs_end_io+0x89/0xb4 [xfs]
[22635.102102] [<ffffffff81047321>] process_one_work+0x204/0x327
[22635.102105] [<ffffffff8104757f>] worker_thread+0x13b/0x25a
[22635.102108] [<ffffffff81047444>] ? process_one_work+0x327/0x327
[22635.102111] [<ffffffff8104af6f>] kthread+0x89/0x91
[22635.102115] [<ffffffff814fdbd4>] kernel_thread_helper+0x4/0x10
[22635.102118] [<ffffffff8104aee6>] ? __init_kthread_worker+0x3c/0x3c
[22635.102120] [<ffffffff814fdbd0>] ? gs_change+0xb/0xb
[22635.102131] INFO: task streamRT-sa:5891 blocked for more than 120
seconds.
[22635.102132] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[22635.102134] streamRT-sa D ffff883f52850348 0 5891 5114
0x00000004
[22635.102136] ffff884063af5b28 0000000000000082 000000000000029c
ffff884066679100
[22635.102140] 0000000000000006 0000000000000001 00000000000122c0
00000000000122c0
[22635.102143] ffff884063af5fd8 ffff884063af4000 00000000000122c0
ffff884063af4000
[22635.102146] Call Trace:
[22635.102149] [<ffffffff814f5fd7>] schedule+0x64/0x66
[22635.102151] [<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
[22635.102154] [<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
[22635.102156] [<ffffffff81261913>]
call_rwsem_down_write_failed+0x13/0x20
[22635.102158] [<ffffffff814f5458>] ? down_write+0x25/0x27
[22635.102164] [<ffffffffa01b35e4>] xfs_ilock+0x4f/0xb4 [xfs]
[22635.102170] [<ffffffffa01b00e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
[22635.102172] [<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
[22635.102178] [<ffffffffa01b0519>] xfs_file_aio_write_checks+0x41/0xfe
[xfs]
[22635.102184] [<ffffffffa01b06ff>] xfs_file_dio_aio_write+0x103/0x1fc
[xfs]
[22635.102189] [<ffffffffa01b0ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
[22635.102195] [<ffffffffa01b0971>] ?
xfs_file_buffered_aio_write+0x179/0x179 [xfs]
[22635.102199] [<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
[22635.102202] [<ffffffff8113360f>] ? aio_fsync+0x29/0x29
[22635.102204] [<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
[22635.102207] [<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
[22635.102209] [<ffffffff8113513d>] do_io_submit+0xfa/0x271
[22635.102212] [<ffffffff811352c4>] sys_io_submit+0x10/0x12
[22635.102215] [<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
[22635.102217] INFO: task streamRT-sa:5895 blocked for more than 120
seconds.
[22635.102218] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[22635.102219] streamRT-sa D ffff883f537fa3c8 0 5895 5114
0x00000004
[22635.102222] ffff883f7d057b28 0000000000000082 0000000000000000
ffff884066227100
[22635.102225] ffff883f7d057ad8 0000000000000002 00000000000122c0
00000000000122c0
[22635.102228] ffff883f7d057fd8 ffff883f7d056000 00000000000122c0
ffff883f7d056000
[22635.102232] Call Trace:
[22635.102234] [<ffffffff814f5fd7>] schedule+0x64/0x66
[22635.102236] [<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
[22635.102239] [<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
[22635.102241] [<ffffffff81261913>]
call_rwsem_down_write_failed+0x13/0x20
[22635.102243] [<ffffffff814f5458>] ? down_write+0x25/0x27
[22635.102249] [<ffffffffa01b35e4>] xfs_ilock+0x4f/0xb4 [xfs]
[22635.102254] [<ffffffffa01b00e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
[22635.102257] [<ffffffff814f6a92>] ?
_raw_spin_unlock_irqrestore+0x30/0x3d
[22635.102262] [<ffffffffa01b0519>] xfs_file_aio_write_checks+0x41/0xfe
[xfs]
[22635.102268] [<ffffffffa01b06ff>] xfs_file_dio_aio_write+0x103/0x1fc
[xfs]
[22635.102274] [<ffffffffa01b0ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
[22635.102279] [<ffffffffa01b0971>] ?
xfs_file_buffered_aio_write+0x179/0x179 [xfs]
[22635.102282] [<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
[22635.102284] [<ffffffff8113360f>] ? aio_fsync+0x29/0x29
[22635.102286] [<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
[22635.102288] [<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
[22635.102291] [<ffffffff8113513d>] do_io_submit+0xfa/0x271
[22635.102294] [<ffffffff8103ed3f>] ? sys_rt_sigprocmask+0x69/0xa1
[22635.102297] [<ffffffff811352c4>] sys_io_submit+0x10/0x12
[22635.102299] [<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
[22635.102301] INFO: task streamRT-sa:5900 blocked for more than 120
seconds.
[22635.102302] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[22635.102303] streamRT-sa D ffff883f53003b88 0 5900 5114
0x00000004
[22635.102305] ffff883f7d109b28 0000000000000082 0000000000000000
ffff884066679100
[22635.102309] ffff883f7d109ad8 0000000000000002 00000000000122c0
00000000000122c0
[22635.102312] ffff883f7d109fd8 ffff883f7d108000 00000000000122c0
ffff883f7d108000
[22635.102315] Call Trace:
[22635.102318] [<ffffffff814f5fd7>] schedule+0x64/0x66
[22635.102320] [<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
[22635.102322] [<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
[22635.102324] [<ffffffff81261913>]
call_rwsem_down_write_failed+0x13/0x20
[22635.102327] [<ffffffff814f5458>] ? down_write+0x25/0x27
[22635.102332] [<ffffffffa01b35e4>] xfs_ilock+0x4f/0xb4 [xfs]
[22635.102338] [<ffffffffa01b00e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
[22635.102340] [<ffffffff814f6a92>] ?
_raw_spin_unlock_irqrestore+0x30/0x3d
[22635.102346] [<ffffffffa01b0519>] xfs_file_aio_write_checks+0x41/0xfe
[xfs]
[22635.102351] [<ffffffffa01b06ff>] xfs_file_dio_aio_write+0x103/0x1fc
[xfs]
[22635.102357] [<ffffffffa01b0ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
[22635.102362] [<ffffffff8105539b>] ? resched_task+0x3e/0x75
[22635.102367] [<ffffffffa01b0971>] ?
xfs_file_buffered_aio_write+0x179/0x179 [xfs]
[22635.102369] [<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
[22635.102372] [<ffffffff8113360f>] ? aio_fsync+0x29/0x29
[22635.102374] [<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
[22635.102376] [<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
[22635.102379] [<ffffffff8113513d>] do_io_submit+0xfa/0x271
[22635.102382] [<ffffffff811352c4>] sys_io_submit+0x10/0x12
[22635.102384] [<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
[22635.102386] INFO: task streamRT-sa:5904 blocked for more than 120
seconds.
[22635.102387] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[22635.102388] streamRT-sa D ffff88405fd50a48 0 5904 5114
0x00000004
[22635.102390] ffff883f51c2bb28 0000000000000082 0000000000000000
ffff8840662d0080
[22635.102394] ffff883f51c2bad8 0000000000000002 00000000000122c0
00000000000122c0
[22635.102397] ffff883f51c2bfd8 ffff883f51c2a000 00000000000122c0
ffff883f51c2a000
[22635.102400] Call Trace:
[22635.102402] [<ffffffff814f5fd7>] schedule+0x64/0x66
[22635.102405] [<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
[22635.102407] [<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
[22635.102409] [<ffffffff81261913>]
call_rwsem_down_write_failed+0x13/0x20
[22635.102412] [<ffffffff814f5458>] ? down_write+0x25/0x27
[22635.102417] [<ffffffffa01b35e4>] xfs_ilock+0x4f/0xb4 [xfs]
[22635.102423] [<ffffffffa01b00e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
[22635.102425] [<ffffffff814f6a92>] ?
_raw_spin_unlock_irqrestore+0x30/0x3d
[22635.102430] [<ffffffffa01b0519>] xfs_file_aio_write_checks+0x41/0xfe
[xfs]
[22635.102436] [<ffffffffa01b06ff>] xfs_file_dio_aio_write+0x103/0x1fc
[xfs]
[22635.102442] [<ffffffffa01b0ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
[22635.102447] [<ffffffffa01b0971>] ?
xfs_file_buffered_aio_write+0x179/0x179 [xfs]
[22635.102450] [<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
[22635.102452] [<ffffffff8113360f>] ? aio_fsync+0x29/0x29
[22635.102454] [<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
[22635.102457] [<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
[22635.102459] [<ffffffff8113513d>] do_io_submit+0xfa/0x271
[22635.102461] [<ffffffff8103ed3f>] ? sys_rt_sigprocmask+0x69/0xa1
[22635.102464] [<ffffffff811352c4>] sys_io_submit+0x10/0x12
[22635.102466] [<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
[22635.102468] INFO: task streamRT-sa:5906 blocked for more than 120
seconds.
[22635.102469] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[22635.102470] streamRT-sa D ffff88405fe2ca88 0 5906 5114
0x00000004
[22635.102473] ffff883f51c2fb28 0000000000000082 0000000000000000
ffff8840667f00c0
[22635.102476] 0000000000000000 0000000000000000 00000000000122c0
00000000000122c0
[22635.102479] ffff883f51c2ffd8 ffff883f51c2e000 00000000000122c0
ffff883f51c2e000
[22635.102482] Call Trace:
[22635.102485] [<ffffffff814f5fd7>] schedule+0x64/0x66
[22635.102487] [<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
[22635.102489] [<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
[22635.102491] [<ffffffff81261913>]
call_rwsem_down_write_failed+0x13/0x20
[22635.102494] [<ffffffff814f5458>] ? down_write+0x25/0x27
[22635.102499] [<ffffffffa01b35e4>] xfs_ilock+0x4f/0xb4 [xfs]
[22635.102505] [<ffffffffa01b00e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
[22635.102507] [<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
[22635.102512] [<ffffffffa01b0519>] xfs_file_aio_write_checks+0x41/0xfe
[xfs]
[22635.102518] [<ffffffffa01b06ff>] xfs_file_dio_aio_write+0x103/0x1fc
[xfs]
[22635.102524] [<ffffffffa01b0ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
[22635.102529] [<ffffffffa01b0971>] ?
xfs_file_buffered_aio_write+0x179/0x179 [xfs]
[22635.102532] [<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
[22635.102534] [<ffffffff8113360f>] ? aio_fsync+0x29/0x29
[22635.102536] [<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
[22635.102538] [<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
[22635.102541] [<ffffffff8113513d>] do_io_submit+0xfa/0x271
[22635.102543] [<ffffffff8103ed3f>] ? sys_rt_sigprocmask+0x69/0xa1
[22635.102545] [<ffffffff811352c4>] sys_io_submit+0x10/0x12
[22635.102548] [<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
[22635.102550] INFO: task streamRT-sa:5908 blocked for more than 120
seconds.
[22635.102551] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[22635.102552] streamRT-sa D ffff88405fe2fac8 0 5908 5114
0x00000004
[22635.102554] ffff883f518b3b28 0000000000000082 000000000000029c
ffff884066770040
[22635.102558] 0000000000000006 0000000000000001 00000000000122c0
00000000000122c0
[22635.102561] ffff883f518b3fd8 ffff883f518b2000 00000000000122c0
ffff883f518b2000
[22635.102564] Call Trace:
[22635.102566] [<ffffffff814f5fd7>] schedule+0x64/0x66
[22635.102569] [<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
[22635.102571] [<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
[22635.102573] [<ffffffff81261913>]
call_rwsem_down_write_failed+0x13/0x20
[22635.102575] [<ffffffff814f5458>] ? down_write+0x25/0x27
[22635.102581] [<ffffffffa01b35e4>] xfs_ilock+0x4f/0xb4 [xfs]
[22635.102586] [<ffffffffa01b00e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
[22635.102589] [<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
[22635.102594] [<ffffffffa01b0519>] xfs_file_aio_write_checks+0x41/0xfe
[xfs]
[22635.102600] [<ffffffffa01b06ff>] xfs_file_dio_aio_write+0x103/0x1fc
[xfs]
[22635.102620] [<ffffffffa01b0ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
[22635.102627] [<ffffffffa01b0971>] ?
xfs_file_buffered_aio_write+0x179/0x179 [xfs]
[22635.102630] [<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
[22635.102637] [<ffffffff8113360f>] ? aio_fsync+0x29/0x29
[22635.102641] [<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
[22635.102651] [<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
[22635.102655] [<ffffffff8113513d>] do_io_submit+0xfa/0x271
[22635.102661] [<ffffffff811352c4>] sys_io_submit+0x10/0x12
[22635.102664] [<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
[22635.102669] INFO: task streamRT-sa:5909 blocked for more than 120
seconds.
[22635.102675] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[22635.102677] streamRT-sa D ffff88405fe2f3c8 0 5909 5114
0x00000004
[22635.102683] ffff883f518b5b28 0000000000000082 00000000000000de
ffff88406663a040
[22635.102694] 0000000000000002 0000000000000001 00000000000122c0
00000000000122c0
[22635.102705] ffff883f518b5fd8 ffff883f518b4000 00000000000122c0
ffff883f518b4000
[22635.102719] Call Trace:
[22635.102723] [<ffffffff814f5fd7>] schedule+0x64/0x66
[22635.102729] [<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
[22635.102734] [<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
[22635.102739] [<ffffffff81261913>]
call_rwsem_down_write_failed+0x13/0x20
[22635.102742] [<ffffffff814f5458>] ? down_write+0x25/0x27
[22635.102751] [<ffffffffa01b35e4>] xfs_ilock+0x4f/0xb4 [xfs]
[22635.102761] [<ffffffffa01b00e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
[22635.102765] [<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
[22635.102773] [<ffffffffa01b0519>] xfs_file_aio_write_checks+0x41/0xfe
[xfs]
[22635.102781] [<ffffffffa01b06ff>] xfs_file_dio_aio_write+0x103/0x1fc
[xfs]
[22635.102788] [<ffffffffa01b0ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
[22635.102798] [<ffffffffa01b0971>] ?
xfs_file_buffered_aio_write+0x179/0x179 [xfs]
[22635.102801] [<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
[22635.102805] [<ffffffff8113360f>] ? aio_fsync+0x29/0x29
[22635.102809] [<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
[22635.102812] [<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
[22635.102815] [<ffffffff8113513d>] do_io_submit+0xfa/0x271
[22635.102818] [<ffffffff8103ed3f>] ? sys_rt_sigprocmask+0x69/0xa1
[22635.102824] [<ffffffff811352c4>] sys_io_submit+0x10/0x12
[22635.102827] [<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
[22635.102834] INFO: task streamRT-sa:5911 blocked for more than 120
seconds.
[22635.102836] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[22635.102838] streamRT-sa D ffff883f5372c408 0 5911 5114
0x00000004
[22635.102844] ffff883f5208bb28 0000000000000082 0000000000000000
ffff884066001180
[22635.102850] 0000000000000000 0000000000000000 00000000000122c0
00000000000122c0
[22635.102861] ffff883f5208bfd8 ffff883f5208a000 00000000000122c0
ffff883f5208a000
[22635.102877] Call Trace:
[22635.102883] [<ffffffff814f5fd7>] schedule+0x64/0x66
[22635.102886] [<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
[22635.102890] [<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
[22635.102895] [<ffffffff81261913>]
call_rwsem_down_write_failed+0x13/0x20
[22635.102902] [<ffffffff814f5458>] ? down_write+0x25/0x27
[22635.102911] [<ffffffffa01b35e4>] xfs_ilock+0x4f/0xb4 [xfs]
[22635.102918] [<ffffffffa01b00e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
[22635.102924] [<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
[22635.102931] [<ffffffffa01b0519>] xfs_file_aio_write_checks+0x41/0xfe
[xfs]
[22635.102939] [<ffffffffa01b06ff>] xfs_file_dio_aio_write+0x103/0x1fc
[xfs]
[22635.102947] [<ffffffffa01b0ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
[22635.102954] [<ffffffffa01b0971>] ?
xfs_file_buffered_aio_write+0x179/0x179 [xfs]
[22635.102959] [<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
[22635.102962] [<ffffffff8113360f>] ? aio_fsync+0x29/0x29
[22635.102968] [<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
[22635.102973] [<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
[22635.102976] [<ffffffff8113513d>] do_io_submit+0xfa/0x271
[22635.102980] [<ffffffff8103ed3f>] ? sys_rt_sigprocmask+0x69/0xa1
[22635.102985] [<ffffffff811352c4>] sys_io_submit+0x10/0x12
[22635.102989] [<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
[22635.102995] INFO: task streamRT-sa:5918 blocked for more than 120
seconds.
[22635.102997] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[22635.103001] streamRT-sa D ffff883f53733a48 0 5918 5114
0x00000004
[22635.103007] ffff884063d99b28 0000000000000082 0000000000000000
ffff88406668b1c0
[22635.103017] 0000000000000000 0000000000000000 00000000000122c0
00000000000122c0
[22635.103029] ffff884063d99fd8 ffff884063d98000 00000000000122c0
ffff884063d98000
[22635.103037] Call Trace:
[22635.103042] [<ffffffff814f5fd7>] schedule+0x64/0x66
[22635.103047] [<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
[22635.103050] [<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
[22635.103054] [<ffffffff81261913>]
call_rwsem_down_write_failed+0x13/0x20
[22635.103058] [<ffffffff814f5458>] ? down_write+0x25/0x27
[22635.103065] [<ffffffffa01b35e4>] xfs_ilock+0x4f/0xb4 [xfs]
[22635.103072] [<ffffffffa01b00e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
[22635.103076] [<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
[22635.103086] [<ffffffffa01b0519>] xfs_file_aio_write_checks+0x41/0xfe
[xfs]
[22635.103092] [<ffffffffa01b06ff>] xfs_file_dio_aio_write+0x103/0x1fc
[xfs]
[22635.103099] [<ffffffffa01b0ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
[22635.103109] [<ffffffffa01b0971>] ?
xfs_file_buffered_aio_write+0x179/0x179 [xfs]
[22635.103113] [<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
[22635.103120] [<ffffffff8113360f>] ? aio_fsync+0x29/0x29
[22635.103126] [<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
[22635.103129] [<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
[22635.103134] [<ffffffff8113513d>] do_io_submit+0xfa/0x271
[22635.103139] [<ffffffff8103ed3f>] ? sys_rt_sigprocmask+0x69/0xa1
[22635.103142] [<ffffffff811352c4>] sys_io_submit+0x10/0x12
[22635.103146] [<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Aug25 ? 00:00:05 init [3]
root 2 0 0 Aug25 ? 00:00:00 [kthreadd]
root 3 2 0 Aug25 ? 00:00:00 [ksoftirqd/0]
root 6 2 0 Aug25 ? 00:00:00 [migration/0]
root 7 2 0 Aug25 ? 00:00:00 [rcuc/0]
root 8 2 0 Aug25 ? 00:00:00 [rcun/0]
root 9 2 0 Aug25 ? 00:00:00 [rcub/0]
root 10 2 0 Aug25 ? 00:00:00 [rcun/1]
root 11 2 0 Aug25 ? 00:00:00 [rcub/1]
root 12 2 0 Aug25 ? 00:00:00 [watchdog/0]
root 13 2 0 Aug25 ? 00:00:00 [migration/1]
root 15 2 0 Aug25 ? 00:00:00 [rcuc/1]
root 16 2 0 Aug25 ? 00:00:00 [ksoftirqd/1]
root 18 2 0 Aug25 ? 00:00:00 [watchdog/1]
root 19 2 0 Aug25 ? 00:00:00 [migration/2]
root 20 2 0 Aug25 ? 00:00:00 [kworker/2:0]
root 21 2 0 Aug25 ? 00:00:00 [rcuc/2]
root 22 2 0 Aug25 ? 00:00:00 [ksoftirqd/2]
root 23 2 0 Aug25 ? 00:00:00 [watchdog/2]
root 24 2 0 Aug25 ? 00:00:00 [migration/3]
root 26 2 0 Aug25 ? 00:00:00 [rcuc/3]
root 27 2 0 Aug25 ? 00:00:00 [ksoftirqd/3]
root 28 2 0 Aug25 ? 00:00:00 [watchdog/3]
root 29 2 0 Aug25 ? 00:00:00 [migration/4]
root 30 2 0 Aug25 ? 00:00:00 [kworker/4:0]
root 31 2 0 Aug25 ? 00:00:00 [rcuc/4]
root 32 2 0 Aug25 ? 00:00:00 [ksoftirqd/4]
root 33 2 0 Aug25 ? 00:00:00 [watchdog/4]
root 34 2 0 Aug25 ? 00:00:00 [migration/5]
root 36 2 0 Aug25 ? 00:00:00 [rcuc/5]
root 37 2 0 Aug25 ? 00:00:00 [ksoftirqd/5]
root 38 2 0 Aug25 ? 00:00:00 [watchdog/5]
root 39 2 0 Aug25 ? 00:00:00 [migration/6]
root 40 2 0 Aug25 ? 00:00:00 [kworker/6:0]
root 41 2 0 Aug25 ? 00:00:00 [rcuc/6]
root 42 2 0 Aug25 ? 00:00:00 [ksoftirqd/6]
root 43 2 0 Aug25 ? 00:00:00 [watchdog/6]
root 44 2 0 Aug25 ? 00:00:00 [migration/7]
root 45 2 0 Aug25 ? 00:00:00 [kworker/7:0]
root 46 2 0 Aug25 ? 00:00:00 [rcuc/7]
root 47 2 0 Aug25 ? 00:00:00 [ksoftirqd/7]
root 48 2 0 Aug25 ? 00:00:00 [watchdog/7]
root 49 2 0 Aug25 ? 00:00:00 [migration/8]
root 50 2 0 Aug25 ? 00:00:00 [kworker/8:0]
root 51 2 0 Aug25 ? 00:00:00 [rcuc/8]
root 52 2 0 Aug25 ? 00:00:00 [ksoftirqd/8]
root 53 2 0 Aug25 ? 00:00:00 [watchdog/8]
root 54 2 0 Aug25 ? 00:00:00 [migration/9]
root 56 2 0 Aug25 ? 00:00:00 [rcuc/9]
root 57 2 0 Aug25 ? 00:00:00 [ksoftirqd/9]
root 58 2 0 Aug25 ? 00:00:00 [watchdog/9]
root 59 2 0 Aug25 ? 00:00:00 [migration/10]
root 60 2 0 Aug25 ? 00:00:00 [kworker/10:0]
root 61 2 0 Aug25 ? 00:00:00 [rcuc/10]
root 62 2 0 Aug25 ? 00:00:00 [ksoftirqd/10]
root 63 2 0 Aug25 ? 00:00:00 [watchdog/10]
root 64 2 0 Aug25 ? 00:00:00 [migration/11]
root 66 2 0 Aug25 ? 00:00:00 [rcuc/11]
root 67 2 0 Aug25 ? 00:00:00 [ksoftirqd/11]
root 68 2 0 Aug25 ? 00:00:00 [watchdog/11]
root 69 2 0 Aug25 ? 00:00:00 [migration/12]
root 70 2 0 Aug25 ? 00:00:00 [kworker/12:0]
root 71 2 0 Aug25 ? 00:00:00 [rcuc/12]
root 72 2 0 Aug25 ? 00:00:00 [ksoftirqd/12]
root 73 2 0 Aug25 ? 00:00:00 [watchdog/12]
root 74 2 0 Aug25 ? 00:00:00 [migration/13]
root 76 2 0 Aug25 ? 00:00:00 [rcuc/13]
root 77 2 0 Aug25 ? 00:00:00 [ksoftirqd/13]
root 78 2 0 Aug25 ? 00:00:00 [watchdog/13]
root 79 2 0 Aug25 ? 00:00:00 [migration/14]
root 80 2 0 Aug25 ? 00:00:00 [kworker/14:0]
root 81 2 0 Aug25 ? 00:00:00 [rcuc/14]
root 82 2 0 Aug25 ? 00:00:00 [ksoftirqd/14]
root 83 2 0 Aug25 ? 00:00:00 [watchdog/14]
root 84 2 0 Aug25 ? 00:00:00 [migration/15]
root 86 2 0 Aug25 ? 00:00:00 [rcuc/15]
root 87 2 0 Aug25 ? 00:00:00 [ksoftirqd/15]
root 88 2 0 Aug25 ? 00:00:00 [watchdog/15]
root 89 2 0 Aug25 ? 00:00:00 [migration/16]
root 90 2 0 Aug25 ? 00:00:00 [kworker/16:0]
root 91 2 0 Aug25 ? 00:00:00 [rcuc/16]
root 92 2 0 Aug25 ? 00:00:00 [rcun/2]
root 93 2 0 Aug25 ? 00:00:00 [rcub/2]
root 94 2 0 Aug25 ? 00:00:00 [ksoftirqd/16]
root 95 2 0 Aug25 ? 00:00:00 [watchdog/16]
root 96 2 0 Aug25 ? 00:00:00 [migration/17]
root 98 2 0 Aug25 ? 00:00:00 [rcuc/17]
root 99 2 0 Aug25 ? 00:00:00 [ksoftirqd/17]
root 100 2 0 Aug25 ? 00:00:00 [watchdog/17]
root 101 2 0 Aug25 ? 00:00:00 [migration/18]
root 102 2 0 Aug25 ? 00:00:00 [kworker/18:0]
root 103 2 0 Aug25 ? 00:00:00 [rcuc/18]
root 104 2 0 Aug25 ? 00:00:00 [ksoftirqd/18]
root 105 2 0 Aug25 ? 00:00:00 [watchdog/18]
root 106 2 0 Aug25 ? 00:00:00 [migration/19]
root 107 2 0 Aug25 ? 00:00:00 [kworker/19:0]
root 108 2 0 Aug25 ? 00:00:00 [rcuc/19]
root 109 2 0 Aug25 ? 00:00:00 [ksoftirqd/19]
root 110 2 0 Aug25 ? 00:00:00 [watchdog/19]
root 111 2 0 Aug25 ? 00:00:00 [migration/20]
root 112 2 0 Aug25 ? 00:00:00 [kworker/20:0]
root 113 2 0 Aug25 ? 00:00:00 [rcuc/20]
root 114 2 0 Aug25 ? 00:00:00 [ksoftirqd/20]
root 115 2 0 Aug25 ? 00:00:00 [watchdog/20]
root 116 2 0 Aug25 ? 00:00:00 [migration/21]
root 117 2 0 Aug25 ? 00:00:00 [kworker/21:0]
root 118 2 0 Aug25 ? 00:00:00 [rcuc/21]
root 119 2 0 Aug25 ? 00:00:00 [ksoftirqd/21]
root 120 2 0 Aug25 ? 00:00:00 [watchdog/21]
root 121 2 0 Aug25 ? 00:00:00 [migration/22]
root 122 2 0 Aug25 ? 00:00:00 [kworker/22:0]
root 123 2 0 Aug25 ? 00:00:00 [rcuc/22]
root 124 2 0 Aug25 ? 00:00:00 [ksoftirqd/22]
root 125 2 0 Aug25 ? 00:00:00 [watchdog/22]
root 126 2 0 Aug25 ? 00:00:00 [migration/23]
root 128 2 0 Aug25 ? 00:00:00 [rcuc/23]
root 129 2 0 Aug25 ? 00:00:00 [ksoftirqd/23]
root 130 2 0 Aug25 ? 00:00:00 [watchdog/23]
root 131 2 0 Aug25 ? 00:00:00 [migration/24]
root 132 2 0 Aug25 ? 00:00:00 [kworker/24:0]
root 133 2 0 Aug25 ? 00:00:00 [rcuc/24]
root 134 2 0 Aug25 ? 00:00:00 [ksoftirqd/24]
root 135 2 0 Aug25 ? 00:00:00 [watchdog/24]
root 136 2 0 Aug25 ? 00:00:00 [migration/25]
root 138 2 0 Aug25 ? 00:00:00 [rcuc/25]
root 139 2 0 Aug25 ? 00:00:00 [ksoftirqd/25]
root 140 2 0 Aug25 ? 00:00:00 [watchdog/25]
root 141 2 0 Aug25 ? 00:00:00 [migration/26]
root 142 2 0 Aug25 ? 00:00:00 [kworker/26:0]
root 143 2 0 Aug25 ? 00:00:00 [rcuc/26]
root 144 2 0 Aug25 ? 00:00:00 [ksoftirqd/26]
root 145 2 0 Aug25 ? 00:00:00 [watchdog/26]
root 146 2 0 Aug25 ? 00:00:00 [migration/27]
root 147 2 0 Aug25 ? 00:00:00 [kworker/27:0]
root 148 2 0 Aug25 ? 00:00:00 [rcuc/27]
root 149 2 0 Aug25 ? 00:00:00 [ksoftirqd/27]
root 150 2 0 Aug25 ? 00:00:00 [watchdog/27]
root 151 2 0 Aug25 ? 00:00:00 [migration/28]
root 152 2 0 Aug25 ? 00:00:00 [kworker/28:0]
root 153 2 0 Aug25 ? 00:00:00 [rcuc/28]
root 154 2 0 Aug25 ? 00:00:00 [ksoftirqd/28]
root 155 2 0 Aug25 ? 00:00:00 [watchdog/28]
root 156 2 0 Aug25 ? 00:00:00 [migration/29]
root 157 2 0 Aug25 ? 00:00:00 [kworker/29:0]
root 158 2 0 Aug25 ? 00:00:00 [rcuc/29]
root 159 2 0 Aug25 ? 00:00:00 [ksoftirqd/29]
root 160 2 0 Aug25 ? 00:00:00 [watchdog/29]
root 161 2 0 Aug25 ? 00:00:00 [migration/30]
root 162 2 0 Aug25 ? 00:00:00 [kworker/30:0]
root 163 2 0 Aug25 ? 00:00:00 [rcuc/30]
root 164 2 0 Aug25 ? 00:00:00 [ksoftirqd/30]
root 165 2 0 Aug25 ? 00:00:00 [watchdog/30]
root 166 2 0 Aug25 ? 00:00:00 [migration/31]
root 167 2 0 Aug25 ? 00:00:00 [kworker/31:0]
root 168 2 0 Aug25 ? 00:00:00 [rcuc/31]
root 169 2 0 Aug25 ? 00:00:00 [ksoftirqd/31]
root 170 2 0 Aug25 ? 00:00:00 [watchdog/31]
root 171 2 0 Aug25 ? 00:00:00 [cpuset]
root 172 2 0 Aug25 ? 00:00:00 [khelper]
root 173 2 0 Aug25 ? 00:00:00 [netns]
root 418 2 0 Aug25 ? 00:00:00 [sync_supers]
root 420 2 0 Aug25 ? 00:00:00 [bdi-default]
root 422 2 0 Aug25 ? 00:00:00 [kblockd]
root 714 2 0 Aug25 ? 00:00:00 [ata_sff]
root 724 2 0 Aug25 ? 00:00:00 [khubd]
root 733 2 0 Aug25 ? 00:00:00 [kworker/30:1]
root 735 2 0 Aug25 ? 00:00:00 [kworker/28:1]
root 737 2 0 Aug25 ? 00:00:00 [kworker/26:1]
root 739 2 0 Aug25 ? 00:00:00 [kworker/24:1]
root 741 2 0 Aug25 ? 00:00:00 [kworker/22:1]
root 743 2 0 Aug25 ? 00:00:00 [kworker/20:1]
root 745 2 0 Aug25 ? 00:00:00 [kworker/18:1]
root 747 2 0 Aug25 ? 00:00:00 [kworker/16:1]
root 749 2 0 Aug25 ? 00:00:00 [kworker/14:1]
root 751 2 0 Aug25 ? 00:00:00 [kworker/12:1]
root 757 2 0 Aug25 ? 00:00:00 [kworker/10:1]
root 759 2 0 Aug25 ? 00:00:00 [kworker/8:1]
root 761 2 0 Aug25 ? 00:00:00 [kworker/6:1]
root 763 2 0 Aug25 ? 00:00:00 [kworker/4:1]
root 866 2 0 Aug25 ? 00:00:00 [rpciod]
root 868 2 0 Aug25 ? 00:00:00 [kworker/2:1]
root 975 2 0 Aug25 ? 00:00:00 [khungtaskd]
root 981 2 0 Aug25 ? 00:00:00 [kswapd0]
root 982 2 0 Aug25 ? 00:00:00 [khugepaged]
root 983 2 0 Aug25 ? 00:00:00 [fsnotify_mark]
root 984 2 0 Aug25 ? 00:00:00 [nfsiod]
root 986 2 0 Aug25 ? 00:00:00 [crypto]
root 1151 2 0 Aug25 ? 00:00:00 [kpsmoused]
root 1167 2 0 Aug25 ? 00:00:00 [deferwq]
root 1913 2 0 Aug25 ? 00:00:00 [bond0]
root 2144 1 0 Aug25 ? 00:00:00 udevd --daemon
root 2525 2 0 Aug25 ? 00:00:00 [scsi_eh_0]
root 2537 2 0 Aug25 ? 00:00:00 [mlx4]
root 2540 2 0 Aug25 ? 00:00:00 [scsi_eh_1]
root 2541 2 0 Aug25 ? 00:00:00 [fw_event0]
root 2595 2 0 Aug25 ? 00:00:00 [scsi_eh_2]
root 2596 2 0 Aug25 ? 00:00:00 [scsi_eh_3]
root 2597 2 0 Aug25 ? 00:00:00 [scsi_eh_4]
root 2598 2 0 Aug25 ? 00:00:00 [scsi_eh_5]
root 2599 2 0 Aug25 ? 00:00:00 [scsi_eh_6]
root 2600 2 0 Aug25 ? 00:00:00 [scsi_eh_7]
root 2667 2 0 Aug25 ? 00:00:00 [mlx4_en]
root 2797 2144 0 Aug25 ? 00:00:00 udevd --daemon
root 2798 2144 0 Aug25 ? 00:00:00 udevd --daemon
root 2800 2 0 Aug25 ? 00:00:00 [mlx4_en]
root 2817 2 0 Aug25 ? 00:00:00 [mlx4_en]
root 2843 2 0 Aug25 ? 00:00:00 [poll_0_status]
root 2844 2 0 Aug25 ? 00:00:00 [scsi_eh_8]
root 2846 2 0 Aug25 ? 00:00:00 [fw_event1]
root 2860 2 0 Aug25 ? 00:00:00 [poll_1_status]
root 2861 2 0 Aug25 ? 00:00:00 [scsi_eh_9]
root 2862 2 0 Aug25 ? 00:00:00 [fw_event2]
root 2864 2 0 Aug25 ? 00:00:00 [poll_2_status]
root 2865 2 0 Aug25 ? 00:00:00 [scsi_eh_10]
root 2866 2 0 Aug25 ? 00:00:00 [fw_event3]
root 2867 2 0 Aug25 ? 00:00:00 [poll_3_status]
root 4059 2 0 Aug25 ? 00:00:00 [bond1]
root 4464 2 0 Aug25 ? 00:00:00 [kdmflush]
root 4470 2 0 Aug25 ? 00:00:00 [kdmflush]
root 4476 2 0 Aug25 ? 00:00:00 [kdmflush]
root 4482 2 0 Aug25 ? 00:00:00 [kdmflush]
root 4488 2 0 Aug25 ? 00:00:00 [kdmflush]
root 4494 2 0 Aug25 ? 00:00:00 [kdmflush]
root 4500 2 0 Aug25 ? 00:00:00 [kdmflush]
root 4506 2 0 Aug25 ? 00:00:00 [kdmflush]
root 4512 2 0 Aug25 ? 00:00:00 [kdmflush]
root 4518 2 0 Aug25 ? 00:00:00 [kdmflush]
root 4527 2 0 Aug25 ? 00:00:00 [jbd2/dm-4-8]
root 4528 2 0 Aug25 ? 00:00:00 [ext4-dio-unwrit]
root 4593 1 0 Aug25 ? 00:00:00 /usr/sbin/rsyslogd -c5
root 4627 1 0 Aug25 ? 00:00:00 /usr/sbin/sshd
root 4645 1 0 Aug25 ? 00:00:00 /usr/sbin/acpid
root 4683 1 0 Aug25 ? 00:00:00 /usr/sbin/cron
ntp 4713 1 0 Aug25 ? 00:00:00 /usr/sbin/ntpd -g -c
/etc/ntp.conf -p /var/run/ntpd.pid -u ntp:ntp
daemon 4747 1 0 Aug25 ? 00:00:00 /sbin/portmap
root 4773 1 0 Aug25 ? 00:00:00 /sbin/rpc.statd
root 4796 1 0 Aug25 ttyS0 00:00:00 /sbin/getty -L 115200
ttyS0 vt100
root 4797 1 0 Aug25 ? 00:00:22 initBC
root 4908 2 0 Aug25 ? 00:00:00 [xfsalloc]
root 4909 2 0 Aug25 ? 00:00:00 [xfs_mru_cache]
root 4910 2 0 Aug25 ? 00:00:00 [xfslogd]
root 4930 2 0 Aug25 ? 00:00:00 [xfsbufd/sdc]
root 4931 2 0 Aug25 ? 00:00:00 [xfs-data/sdc]
root 4932 2 0 Aug25 ? 00:00:00 [xfs-conv/sdc]
root 4933 2 0 Aug25 ? 00:00:01 [xfsaild/sdc]
root 4935 2 0 Aug25 ? 00:00:00 [xfsbufd/sdd]
root 4936 2 0 Aug25 ? 00:00:00 [xfs-data/sdd]
root 4937 2 0 Aug25 ? 00:00:00 [xfs-conv/sdd]
root 4938 2 0 Aug25 ? 00:00:01 [xfsaild/sdd]
root 5115 1 6 Aug25 ? 00:11:28 [streamRT-sa] <defunct>
root 6582 2 0 Aug25 ? 00:00:00 [kworker/1:2]
root 6633 2 0 Aug25 ? 00:00:00 [kworker/23:1]
root 6636 2 0 Aug25 ? 00:00:00 [kworker/15:0]
root 6640 2 0 Aug25 ? 00:00:00 [kworker/29:2]
root 6647 2 0 Aug25 ? 00:00:00 [kworker/3:1]
root 6649 2 0 Aug25 ? 00:00:00 [kworker/25:2]
root 6654 2 0 Aug25 ? 00:00:00 [kworker/1:0]
root 6658 2 0 Aug25 ? 00:00:00 [kworker/19:1]
root 6659 2 0 Aug25 ? 00:00:00 [kworker/21:1]
root 6661 2 0 Aug25 ? 00:00:00 [kworker/27:2]
root 6662 2 0 Aug25 ? 00:00:00 [kworker/15:2]
root 6663 2 0 Aug25 ? 00:00:00 [kworker/5:2]
root 6666 2 0 Aug25 ? 00:00:00 [kworker/31:2]
root 6671 2 0 Aug25 ? 00:00:00 [kworker/0:4]
root 6673 2 0 Aug25 ? 00:00:00 [kworker/7:3]
root 6677 2 0 Aug25 ? 00:00:00 [kworker/0:6]
root 6678 2 0 Aug25 ? 00:00:00 [kworker/0:7]
root 6681 2 0 Aug25 ? 00:00:00 [kworker/0:10]
root 6684 2 0 Aug25 ? 00:00:00 [kworker/13:5]
root 6690 2 0 Aug25 ? 00:00:00 [kworker/9:4]
root 6695 2 0 Aug25 ? 00:00:00 [kworker/7:5]
root 6696 2 0 Aug25 ? 00:00:00 [kworker/13:7]
root 6697 2 0 Aug25 ? 00:00:00 [kworker/9:6]
root 6702 2 0 Aug25 ? 00:00:00 [kworker/0:11]
root 6705 2 0 Aug25 ? 00:00:00 [kworker/0:14]
root 6707 2 0 Aug25 ? 00:00:00 [kworker/0:16]
root 6719 2 0 Aug25 ? 00:00:00 [kworker/0:24]
root 6721 2 0 Aug25 ? 00:00:00 [kworker/0:26]
root 6723 2 0 Aug25 ? 00:00:00 [kworker/9:8]
root 6726 2 0 Aug25 ? 00:00:00 [kworker/31:5]
root 6731 2 0 Aug25 ? 00:00:00 [kworker/1:7]
root 6734 2 0 Aug25 ? 00:00:00 [kworker/11:6]
root 6736 2 0 Aug25 ? 00:00:00 [kworker/13:10]
root 6740 2 0 Aug25 ? 00:00:00 [kworker/0:30]
root 6741 2 0 Aug25 ? 00:00:00 [kworker/0:31]
root 6745 2 0 Aug25 ? 00:00:00 [kworker/9:9]
root 6748 2 0 Aug25 ? 00:00:00 [kworker/0:36]
root 6749 2 0 Aug25 ? 00:00:00 [kworker/19:3]
root 6753 2 0 Aug25 ? 00:00:00 [kworker/23:3]
root 6757 2 0 Aug25 ? 00:00:00 [kworker/13:12]
root 6758 2 0 Aug25 ? 00:00:00 [kworker/7:6]
root 6761 2 0 Aug25 ? 00:00:00 [kworker/0:41]
root 6767 2 0 Aug25 ? 00:00:00 [kworker/0:44]
root 6768 2 0 Aug25 ? 00:00:00 [kworker/5:7]
root 6769 2 0 Aug25 ? 00:00:00 [kworker/1:9]
root 6770 2 0 Aug25 ? 00:00:00 [kworker/0:45]
root 6774 2 0 Aug25 ? 00:00:00 [kworker/17:7]
root 6776 2 0 Aug25 ? 00:00:00 [kworker/3:4]
root 6779 2 0 Aug25 ? 00:00:00 [kworker/11:9]
root 6781 2 0 Aug25 ? 00:00:00 [kworker/0:50]
root 6787 2 0 Aug25 ? 00:00:00 [kworker/0:53]
root 6788 2 0 Aug25 ? 00:00:00 [kworker/25:1]
root 6793 2 0 Aug25 ? 00:00:00 [kworker/1:10]
root 6795 2 0 Aug25 ? 00:00:00 [kworker/27:4]
root 6797 2 0 Aug25 ? 00:00:00 [kworker/0:57]
root 6798 2 0 Aug25 ? 00:00:00 [kworker/0:58]
root 6806 2 0 Aug25 ? 00:00:00 [kworker/15:5]
root 6807 2 0 Aug25 ? 00:00:00 [kworker/0:62]
root 6808 2 0 Aug25 ? 00:00:00 [kworker/0:63]
root 6809 2 0 Aug25 ? 00:00:00 [kworker/17:8]
root 6814 2 0 Aug25 ? 00:00:00 [kworker/0:66]
root 6815 2 0 Aug25 ? 00:00:01 [kworker/0:67]
root 6819 2 0 Aug25 ? 00:00:00 [kworker/0:69]
root 6824 2 0 Aug25 ? 00:00:00 [kworker/25:3]
root 6895 2 0 00:20 ? 00:00:00 [kworker/u:0]
root 6900 2 0 00:26 ? 00:00:00 [kworker/u:1]
root 6904 2 0 00:57 ? 00:00:00 [flush-0:20]
root 6933 2 0 01:00 ? 00:00:00 [kworker/u:2]
root 6934 4627 0 01:02 ? 00:00:00 sshd: iris [priv]
iris 6936 6934 0 01:02 ? 00:00:00 sshd: iris@pts/1
iris 6937 6936 0 01:02 pts/1 00:00:00 -bash
root 6967 6937 0 01:02 pts/1 00:00:00 sudo su
root 6968 6967 0 01:02 pts/1 00:00:00 su
root 6969 6968 0 01:02 pts/1 00:00:00 bash
root 6976 6969 0 01:04 pts/1 00:00:00 ps -ef
# lsof|grep stream
lsof: WARNING: can't stat() xfs file system /mnt/VOL1
Output information may be incomplete.
bash 6937 iris cwd DIR 0,20 4096
2113547 /iris/home/adrian/streamRT (mbc:/iris)
sudo 6967 root cwd DIR 0,20 4096
2113547 /iris/home/adrian/streamRT (mbc:/iris)
su 6968 root cwd DIR 0,20 4096
2113547 /iris/home/adrian/streamRT (mbc:/iris)
bash 6969 root cwd DIR 0,20 4096
2113547 /iris/home/adrian/streamRT (mbc:/iris)
--
Stan
_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs
