Hi Tejun,
After getting the soft reset, I ran dd on the partition and found that
it is blocked. Here is the output
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(3, "\225\225\225\225\225\225\225\225\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
write(4, "\225\225\225\225\225\225\225\225\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(3, unfinished...
I enabled sysrq and attached is the log of sysrq trace output. But I
couldn't get anything constructive from that.
Thanks
Sagar
On Wed, Jul 16, 2008 at 2:27 PM, Sagar Borikar <sagar.borikar@xxxxxxxxx> wrote:
> On Wed, Jul 16, 2008 at 10:30 AM, Tejun Heo <htejun@xxxxxxxxx> wrote:
>> Sagar Borikar wrote:
>>>> Can you please post the log which leads to the softreset? The above
>>>> only shows EH's response to the problem not the cause itself. Also, if
>>>> possible, please try more recent kernel. Recent ones have better error
>>>> reporting and will help debugging the problem even if it can't be used
>>>> for production.
>>>
>>> Unfortunatey, we can't upgrade kernel as this is a production system.
>>> And these are the only logs
>>> which I get on the console and dmesg output, nothing else comes from
>>> sata controller. So I am
>>> clueless from where the soft reset is triggered.
>>
>> Eh... That shouldn't happen but ISTR fixing paths where diagnostics
>> weren't reported correctly. :-( You can make it more verbose by
>> editing drivers/ata/libata-eh.c::ata_eh_report(). There are several
>> conditions which make it veil without reporting. Remove them so that it
>> always reports.
> Sure. I'll update you with the results after this and couple of other
> debug information changes. The failure comes anywhere between 10 hours
> to 48 hours. I would pray that it comes
> as early as possible this time :)
>
>>>> SATA IO errors are much more common than PATA ones probably due to its
>>>> high signal rate. Even on a otherwise perfectly healthy system, SATA IO
>>>> errors occur occasionally (say, once in several months). However, if
>>>> such problems are frequent and regular, it does indicate a problem. One
>>>> of not-so-rare causes for such problems is power or interference
>>>> problem. Using different powre supply and hooking up the harddrive to a
>>>> separate power supply is the easiest way to rule this out.
>>> Understood but the point is we don't see soft reset on other platforms
>>> with different sata controller. So I would guess that it could be a
>>> combination of power suppy
>>> and the sata controller. I'll update you with the results of separate
>>> power supply to drive and system test.
>>
>> Hmmm.. Combination of PSU + SATA controller isn't very likely. :-(
>> Well, please rule it out anyway as the other possibility (silent data
>> corruption) is scarier and much more difficult to track down.
>
> Ok. But then any other clue for soft resets?
>>>> This momentary power loss (short voltage drop will do the job) is not so
>>>> rare. A few months ago, I tracked down a fs corruption problem on a
>>>> server from a major vendor to this problem and it wasn't a single
>>>> machine. The whole line or production batch was problematic.
>>>
>>>> You can often hear the head doing an emergency unload and then spinning
>>>> back up shortly after. This also increments emergency unload and/or
>>>> start stop count in the smart output, so if those counters increase
>>>> after such IO errors, it's likely that you're experiencing this problem.
>>>>
>>>> Hope it helped.
>>>
>>> Thanks a ton for this detail information.
>>
>> Can you please report full boot log + smartctl -a output before and
>> after an error?
> log attached. smartctl output is before soft reset and corruption
>> --
>> tejun
>>
>
SysRq : Show State
sibling
task PC pid father child younger older
init S 80126598 0 1 0 2 (NOTLB)
Stack : 7fdbed30 00000000 00000000 00000000 fffffe00 81137950 81137950 00000000
81137898 00000004 00000001 00000000 7fdbee7c 80126598 8013d688 8013d65c
8011dfc8 00100100 00200200 81139f30 00000000 81137898 8011dfc8 81135de8
81135de8 399349d0 00000000 81137898 8011dfc8 00000000 00000000 00000000
7fdbee7c 00000000 7fdbee7c 00000001 00408140 00000000 00000000 00000000
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80126598>] do_wait+0x2f0/0x42c
[<801267b0>] sys_wait4+0x3c/0x48
[<8010c180>] stack_done+0x20/0x3c
ksoftirqd/0 S 80128C0C 0 2 1 3 (L-TLB)
Stack : 00000000 8011ee28 9000a003 00000000 81139ef0 fffffffc 00000000 80128bb8
00000000 00000000 00000000 00000000 00000000 80128c0c 00000000 00000000
00000000 00000000 801398bc 801398a4 00000000 00000000 00000000 00000000
ffffffff ffffffff ffffffff ffffffff 00000001 00000000 00000000 00000000
00000000 00000000 80104c38 00000000 90008003 00000000 00000000 00000000
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80128c0c>] ksoftirqd+0x54/0xc4
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
watchdog/0 S 80145540 0 3 1 4 2 (L-TLB)
Stack : 00000000 811a6b94 8011e898 8011e810 81139ef8 fffffffc 00000000 801454f0
00000000 00000000 00000000 00000000 00000000 80145540 00000000 801454f0
00000000 00000000 00000063 00000000 801398bc 801398a4 00000000 00000000
00000000 00000000 ffffffff ffffffff ffffffff ffffffff 00000001 00000000
00000000 00000000 00000000 00000000 80104c38 00000000 90008003 00000000
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80145540>] watchdog+0x50/0x70
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
events/0 S 80135728 0 4 1 5 3 (L-TLB)
Stack : 00000000 811a6b94 00000000 8113ed00 8113ed10 8113ed08 8113ed00 80135620
00000000 00000000 00000000 00000000 00000000 80135728 0000007d 00000000
80102ef0 08602040 00000000 81136c68 8011dfc8 8113ed10 8113ed10 81136c68
ffffffff ffffffff ffffffff ffffffff 00000000 00000000 00000000 00000001
00020000 00000000 00000000 00000000 81139eb0 fffffffc 8113ed00 801398bc
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80135728>] worker_thread+0x108/0x180
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
khelper S 80135728 0 5 1 6 4 (L-TLB)
Stack : 00000000 2ac3ddf0 00000000 8113ed80 8113ed90 8113ed88 8113ed80 80135620
00000000 00000000 00000000 00000000 00000000 80135728 0000007d 00000000
80102ef0 fff2fff2 00000000 81136858 8011dfc8 8113ed90 8113ed90 81136858
ffffffff ffffffff ffffffff ffffffff 00000000 00000000 00000000 00000001
00020000 00000000 00000000 00000000 81139eb0 fffffffc 8113ed80 801398bc
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80135728>] worker_thread+0x108/0x180
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
kthread S 80135728 0 6 1 7 115 5 (L-TLB)
Stack : fffffffb 8011e3cc 00000000 8113ee00 8113ee10 8113ee08 8113ee00 80135620
00000000 00000000 00000000 00000000 00000000 80135728 0000007d 00000000
80102ef0 fb60ff40 00000000 81136448 8011dfc8 8113ee10 8113ee10 81136448
ffffffff ffffffff ffffffff ffffffff 00000000 00000000 00000000 00000001
00020000 00000000 00000000 00000000 81139e60 fffffffc 8113ee00 801398bc
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80135728>] worker_thread+0x108/0x180
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
kblockd/0 S 80135728 0 7 6 10 (L-TLB)
Stack : 9000a003 000000a0 00000000 81126580 81126590 81126588 81126580 80135620
00000000 00000000 00000000 00000000 00000000 80135728 003d08ba 2d854d00
805bccc8 80102fe8 00000000 81136038 8011dfc8 81126590 81126590 81136038
ffffffff ffffffff ffffffff ffffffff 00000000 00000000 00000000 00000001
00020000 00000000 00000000 00000000 81139e40 fffffffc 81126580 801398bc
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80135728>] worker_thread+0x108/0x180
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
khubd S 80396F18 0 10 6 65 7 (L-TLB)
Stack : 00000001 00000001 81139ea4 81139ea4 8052a8d0 fffffffc 00000000 80396e40
00000000 00000000 00000000 00000000 00000000 80396f18 00000001 81191098
81139ea0 8048130c 00000000 81191098 80139e08 8052a8d8 8052a8d8 fffffffc
00000000 81191098 80139e08 8119ff5c 8119ff5c 00000000 81139ea0 801398bc
00000000 00000000 00000000 00000000 ffffffff ffffffff ffffffff ffffffff
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80396f18>] hub_thread+0xd8/0x128
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
kswapd0 D 80483240 0 65 6 66 10 (L-TLB)
Stack : 8118b600 00003000 813c429c 813c4ccc 8120b548 836fc6e0 00000008 0610b000
836fc7a8 00000001 00000000 00000004 00000000 80483240 80518af8 81211ee8
00000001 81211db0 845e3880 87f23880 8120b548 00000001 836fc770 8013d860
8118b960 00002002 0000afc8 80364a20 802cad44 00000000 00000000 805c3a40
813ba128 813ba100 00000008 00047000 00000000 802cac88 80136924 8012cabc
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80483240>] __down_read+0xc4/0xec
[<8013d860>] down_read+0x10/0x1c
[<802cad44>] xfs_ilock+0x8c/0xa4
[<802cac88>] xfs_ilock_map_shared+0x38/0x4c
[<802d27f8>] xfs_iomap+0xd8/0x4dc
[<802fe90c>] xfs_bmap+0x30/0x3c
[<802f3cfc>] xfs_map_blocks+0x50/0x84
[<802f52a4>] xfs_page_state_convert+0x56c/0x840
[<802f565c>] xfs_vm_writepage+0xe4/0x140
[<80153cf4>] pageout+0x150/0x1e8
[<80154144>] shrink_page_list+0x2b8/0x504
[<8015455c>] shrink_inactive_list+0xc0/0x304
[<80154da8>] shrink_zone+0x100/0x148
[<80155298>] balance_pgdat+0x1fc/0x350
[<801554e0>] kswapd+0xf4/0xfc
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
aio/0 S 80135728 0 66 6 67 65 (L-TLB)
Stack : 8011d29c 7b00ff42 f700ff00 ff00fd0c 811c5890 811c5888 811c5880 80135620
00000000 00000000 00000000 00000000 00000000 80135728 8113ee10 00000000
80102ef0 fb60ff40 00000000 8120b138 8011dfc8 811c5890 811c5890 8120b138
ffffffff ffffffff ffffffff ffffffff 00000000 00000000 00000000 00000001
00020000 00000000 00000000 00000000 81139e58 fffffffc 811c5880 801398bc
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80135728>] worker_thread+0x108/0x180
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
cifsoplockd S 80482588 0 67 6 68 66 (L-TLB)
Stack : 0000008c 87a54000 00002000 8012c9f0 01946984 00002616 81365f18 8023cd2c
00000000 00000000 00000000 00000000 00000000 80482588 00000000 2b151170
81139eb4 81139eb4 805c2b18 805c2b18 01946984 8012df90 8120ad28 805c21c8
805c77c8 fffffffc 00000000 8023cec8 00000001 8120ad28 81139eb0 8048130c
00000000 8023cd2c 00000000 00000000 81139eb0 fffffffc 00000000 8023cd2c
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482588>] schedule_timeout+0x9c/0xd4
[<8023cec8>] cifs_oplock_thread+0x19c/0x1cc
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
cifsdnotifyd S 80482588 0 68 6 69 67 (L-TLB)
Stack : 00010000 857da000 82c4df9f 00008000 019460cf 00000ea6 81367f40 8023cef8
00000000 00000000 00000000 00000000 00000000 80482588 00000000 2b151170
00000000 00000000 805c2ad0 8610c2fc 019460cf 8012df90 8120a918 805c21c8
805c77b4 805c77b4 00000000 8023cf20 00000000 8023cef8 00000000 00000000
81139eb0 fffffffc 801398bc 801398a4 00000000 00000000 00000000 00000000
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482588>] schedule_timeout+0x9c/0xd4
[<8023cf20>] cifs_dnotify_thread+0x28/0xb8
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
xfslogd/0 S 80135728 0 69 6 70 68 (L-TLB)
Stack : 9000a003 000000a0 00000000 00ff00ff 813b5a10 813b5a08 813b5a00 80135620
00000000 00000000 00000000 00000000 00000000 80135728 8113ee10 00000000
80102ef0 00f200f2 00000000 8120a508 8011dfc8 813b5a10 813b5a10 8120a508
ffffffff ffffffff ffffffff ffffffff 00000000 00000000 00000000 00000001
00020000 00000000 00000000 00000000 81139e00 fffffffc 813b5a00 801398bc
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80135728>] worker_thread+0x108/0x180
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
xfsdatad/0 S 80135728 0 70 6 75 69 (L-TLB)
Stack : 00000000 811a6b94 00000000 40ff19bf 813b5990 813b5988 813b5980 80135620
00000000 00000000 00000000 00000000 00000000 80135728 8113ee10 00000000
80102ef0 21f760fa 00000000 8120a0f8 8011dfc8 813b5990 813b5990 8120a0f8
ffffffff ffffffff ffffffff ffffffff 00000000 00000000 00000000 00000001
00020000 00000000 00000000 00000000 81139e00 fffffffc 813b5980 801398bc
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80135728>] worker_thread+0x108/0x180
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
gpio_blink S 80481DD4 0 75 6 96 70 (L-TLB)
Stack : 00000000 00000000 00000000 00000000 805250b0 00000000 00000000 8033fcb4
00000000 00000000 00000000 00000000 00000000 80481dd4 00000000 00000000
ffff44e1 00000002 00000001 811918b8 8011dfc8 805250b4 805250b4 8033fcb4
00000000 811918b8 8011dfc8 00000000 00000000 8033fcb4 00000000 00000000
8033fe18 8033fe04 81139eb0 811918b8 90008000 ffff00fe 00000000 00000000
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80481dd4>] wait_for_completion_interruptible+0xf0/0x168
[<8033fe18>] sequoia_gpio_blinkthread+0x164/0x188
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
ata/0 S 80135728 0 96 6 97 75 (L-TLB)
Stack : 8011d29c 0000005d 39330040 801203c8 813ba510 813ba508 813ba500 80135620
00000000 00000000 00000000 00000000 00000000 80135728 8113ee10 00000000
80102ef0 80124028 00000000 81190468 8011dfc8 813ba510 813ba510 81190468
ffffffff ffffffff ffffffff ffffffff 00000000 00000000 00000000 00000001
00020000 00000000 00000000 00000000 81139e60 fffffffc 813ba500 801398bc
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80135728>] worker_thread+0x108/0x180
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
ata_aux S 80135728 0 97 6 98 96 (L-TLB)
Stack : 8011d29c 0000005c 00000000 813ba480 813ba490 813ba488 813ba480 80135620
00000000 00000000 00000000 00000000 00000000 80135728 8113ee10 00000000
80102ef0 80124028 00000000 81190058 8011dfc8 813ba490 813ba490 81190058
ffffffff ffffffff ffffffff ffffffff 00000000 00000000 00000000 00000001
00020000 00000000 00000000 00000000 81139e60 fffffffc 813ba480 801398bc
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80135728>] worker_thread+0x108/0x180
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
ataSpinup/0 S 80135728 0 98 6 100 97 (L-TLB)
Stack : 813ba408 80135620 00000000 00000000 813ba410 813ba408 813ba400 80135620
00000000 00000000 00000000 00000000 00000000 80135728 8113ee10 00000000
80102ef0 80124028 00000000 811d60b8 8011dfc8 813ba410 813ba410 811d60b8
ffffffff ffffffff ffffffff ffffffff 00000000 00000000 00000000 00000001
00020000 00000000 00000000 00000000 81139e60 fffffffc 813ba400 801398bc
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80135728>] worker_thread+0x108/0x180
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
scsi_eh_0 S 80362A54 0 100 6 101 98 (L-TLB)
Stack : 90008001 813c4000 90008001 813c4000 813c4000 fffffffc 813c4000 803629cc
00000000 00000000 00000000 00000000 00000000 80362a54 00000000 00000000
00000000 00000000 81139ca0 801398bc 00000000 00000000 00000000 00000000
ffffffff ffffffff ffffffff ffffffff 00000001 00000000 00000000 00000000
00000000 00000000 80104c38 00000000 90008003 00000000 00000000 00000000
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80362a54>] scsi_error_handler+0x88/0x164
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
scsi_eh_1 S 80362A54 0 101 6 157 100 (L-TLB)
Stack : 9000a001 811c8000 9000a001 811c8000 811c8000 fffffffc 811c8000 803629cc
00000000 00000000 00000000 00000000 00000000 80362a54 00000000 00000000
00000000 00000000 81139ca0 801398bc 00000000 00000000 00000000 00000000
ffffffff ffffffff ffffffff ffffffff 00000001 00000000 00000000 00000000
00000000 00000000 80104c38 00000000 90008003 00000000 00000000 00000000
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80362a54>] scsi_error_handler+0x88/0x164
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
mtdblockd S 80388668 0 115 1 299 6 (L-TLB)
Stack : 00000001 00000157 811a7284 00000001 00000001 805f6b10 00000000 8052a3a0
811a6424 00000000 00000000 00000000 00000000 80388668 00000000 804d59a4
00000000 00000000 00000000 811fb118 8011dfc8 81201690 81201690 00000000
00000000 811fb118 8011dfc8 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 80104c38 90008003 00000000 00000000 00000000
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80388668>] mtd_blktrans_thread+0x12c/0x1e0
[<80104c38>] kernel_thread_helper+0x10/0x18
scsi_eh_2 S 80362A54 0 157 6 158 101 (L-TLB)
Stack : 00000000 00000000 00000001 811a8078 813fc400 fffffffc 813fc400 803629cc
00000000 00000000 00000000 00000000 00000000 80362a54 00000000 00000000
00000000 00000000 81139d58 801398bc 00000000 00000000 00000000 00000000
ffffffff ffffffff ffffffff ffffffff 00000001 00000000 00000000 00000000
00000000 00000000 80104c38 00000000 90008003 00000000 00000000 00000000
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80362a54>] scsi_error_handler+0x88/0x164
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
usb-storage S 80481080 0 158 6 192 157 (L-TLB)
Stack : 00000000 8118b4e0 813fc69c 803b0d64 00000000 813fc73c 811a98d8 813fc740
00000000 00000000 00000000 00000000 00000000 80481080 80360d1c 813fc400
8118b4e0 8035e07c 00000001 811a98d8 8011dfc8 813fc740 813fc740 803b01d4
813fc69c 813fc400 813fc69c 803b1bbc 803b1c10 803b1df0 00000001 811a98d8
81139d60 8048130c 813fc69c 803b1bbc 00000000 00000000 81139d60 fffffffc
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80481080>] __down_interruptible+0xdc/0x158
[<803b1c10>] usb_stor_control_thread+0x54/0x2d4
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
kcryptd/0 S 80135728 0 192 6 193 158 (L-TLB)
Stack : 8011d29c 000000bc 31383800 801203c8 80603890 80603888 80603880 80135620
00000000 00000000 00000000 00000000 00000000 80135728 8113ee10 00000000
80102ef0 80124028 00000000 811fb528 8011dfc8 80603890 80603890 811fb528
ffffffff ffffffff ffffffff ffffffff 00000000 00000000 00000000 00000001
00020000 00000000 00000000 00000000 81139e58 fffffffc 80603880 801398bc
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80135728>] worker_thread+0x108/0x180
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
kmirrord S 80135728 0 193 6 2145 192 (L-TLB)
Stack : 8011d29c 000000bb 31383700 801203c8 80603a90 80603a88 80603a80 80135620
00000000 00000000 00000000 00000000 00000000 80135728 8113ee10 00000000
80102ef0 80124028 00000000 811d6ce8 8011dfc8 80603a90 80603a90 811d6ce8
ffffffff ffffffff ffffffff ffffffff 00000000 00000000 00000000 00000001
00020000 00000000 00000000 00000000 81139e60 fffffffc 80603a80 801398bc
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80135728>] worker_thread+0x108/0x180
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
jffs2_gcd_mtd S 80288D5C 0 299 1 307 115 (L-TLB)
Stack : 9004a001 811914a8 8788df38 8012eed4 9004a001 805b2000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 80288d5c 80102ef0 00000001
00000000 00000000 00000001 00000080 00000000 00000000 00000000 00000000
00000000 00000000 805b2000 80288cd4 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80288d5c>] jffs2_garbage_collect_thread+0x88/0x16c
[<80104c38>] kernel_thread_helper+0x10/0x18
udevd S 80482518 0 307 1 1011 299 (NOTLB)
Stack : 8111aa80 ffffffff 811aa000 811aa280 87a71f00 7fffffff 00000100 00000000
00000008 879c3f20 879c3de8 879c3de4 879c3de0 80482518 804036a0 811aa000
8792d580 00000000 8019ecc4 879c3e80 87f45b80 803facb8 87a71f00 00000001
87a71f00 00000000 00000100 801844f0 00000000 805c1fa0 80128638 801285d0
80184038 00000000 00000000 00000004 87f459a0 00000000 811fad08 8011dfc8
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482518>] schedule_timeout+0x2c/0xd4
[<801844f0>] do_select+0x32c/0x3a8
[<80184798>] core_sys_select+0x22c/0x318
[<80184970>] sys_select+0xec/0x234
[<8010c180>] stack_done+0x20/0x3c
mdadm S 80482588 0 1011 1 1201 307 (NOTLB)
Stack : 87259db8 00000005 7f836668 00000005 0194627c 00001388 87259ad0 00000041
00000005 87259f20 87259de8 87259de4 87259de0 80482588 00000000 00000000
87259b18 80183ef8 805c2ae0 805c2ae0 0194627c 8012df90 874dc958 805c21c8
87bb4ac0 00000000 00000020 801844f0 8618d560 8618d560 00000014 803ffa3c
80184038 00000000 00000000 00000001 87bb4ac0 00000000 874dc958 8011dfc8
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482588>] schedule_timeout+0x9c/0xd4
[<801844f0>] do_select+0x32c/0x3a8
[<80184798>] core_sys_select+0x22c/0x318
[<80184970>] sys_select+0xec/0x234
[<8010c180>] stack_done+0x20/0x3c
ifplugd S 80482588 0 1201 1 1226 1011 (NOTLB)
Stack : 80000000 804233f4 00000000 811aa000 01945bd6 000000fa 874bfad0 00000000
00000006 874bff20 874bfde8 874bfde4 874bfde0 80482588 86d14d40 00000000
87569e00 87a716e0 805c2880 805c2880 01945bd6 8012df90 811a94c8 805c21c8
87a716e0 00000000 00000040 801844f0 20000000 86696024 c0a8035c effffffa
80184038 00000000 00000000 00000001 87a716e0 00000000 811a94c8 8011dfc8
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482588>] schedule_timeout+0x9c/0xd4
[<801844f0>] do_select+0x32c/0x3a8
[<80184798>] core_sys_select+0x22c/0x318
[<80184970>] sys_select+0xec/0x234
[<8010c180>] stack_done+0x20/0x3c
netdyn-eth0 S 80126598 0 1226 1 1246 1513 1201 (NOTLB)
Stack : 8013d8d0 8013d860 805af814 874dc138 fffffe00 874dc1f0 874dc1f0 00000000
874dc138 00000004 00000001 00000000 7fc14428 80126598 874dcd68 86cb5f30
00000000 00030002 7fc14340 00000012 00000000 874dc138 8011dfc8 87317e28
87317e28 00000012 00000000 874dc138 8011dfc8 00000000 00000000 86cb5f08
ffffffff 7fc14428 00000000 00000004 ffffffff 10013750 00000000 00000000
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80126598>] do_wait+0x2f0/0x42c
[<801267b0>] sys_wait4+0x3c/0x48
[<801267cc>] sys_waitpid+0x10/0x24
[<8010c180>] stack_done+0x20/0x3c
udhcpc S 80126598 0 1246 1226 12387 (NOTLB)
Stack : 8013d8d0 8013d860 875d60d4 874dcd68 fffffe00 874dce20 874dce20 00000000
874dcd68 00000004 00000001 00000000 00000000 80126598 875916a8 877b5f30
7fa969f8 00030002 7fa96880 00000012 00000000 874dcd68 8011dfc8 874a8e48
874a8e48 00000012 00000000 874dcd68 8011dfc8 00000000 00000000 00000001
00003063 00000000 00000000 7fa96d04 00000dac 00000000 39543ce2 000007d0
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80126598>] do_wait+0x2f0/0x42c
[<801267b0>] sys_wait4+0x3c/0x48
[<801267cc>] sys_waitpid+0x10/0x24
[<8010c180>] stack_done+0x20/0x3c
portmap S 80482518 0 1513 1 1523 1226 (NOTLB)
Stack : 8583fa40 805278c4 8583fa40 00000000 87739e08 7fffffff 00000000 00000000
87739f20 00000000 87739e08 00000000 87739ba8 80482518 8042dca4 01023d8d
81df026c 8583fa40 87739df0 87183aa0 87739df8 803facb8 82336dc4 81e1ade0
87739e08 87662ae0 00000000 80185044 80429564 80000000 805cd66c 8583fa40
00000000 804073b8 00000003 87739de8 00000000 00000018 fffffff4 10006620
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482518>] schedule_timeout+0x2c/0xd4
[<80185044>] do_poll+0x1d8/0x24c
[<80185200>] do_sys_poll+0x148/0x224
[<80185328>] sys_poll+0x4c/0x58
[<8010c180>] stack_done+0x20/0x3c
tcps-httpds S 80482518 0 1523 1 1553 1513 (NOTLB)
Stack : 00000000 00000000 00000000 00000000 00000002 7fffffff 7fffffff 7f909c70
7f909c80 100012dc 004112f4 10012fa8 00000440 80482518 86d13e2c 87537120
86d13e41 ffffffff 00000000 ffffffff 803fe804 00000000 00000000 00000000
00000002 86d155c0 7fffffff 8042cd48 86d13e40 87537120 86d13e40 80189c58
00000001 81190878 80139e08 86eb2838 86eb2838 80189e90 00000000 81190878
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482518>] schedule_timeout+0x2c/0xd4
[<8042cd48>] inet_csk_wait_for_connect+0x90/0x120
[<8042ce3c>] inet_csk_accept+0x64/0x15c
[<8044e3b0>] inet_accept+0x34/0xc8
[<803fb588>] sys_accept+0xe8/0x1a4
[<8010c180>] stack_done+0x20/0x3c
tcps-httpd S 80482518 0 1553 1 5138 1627 1523 (NOTLB)
Stack : 85e5bdf0 ffffffff 000006d8 85e5be00 00000002 7fffffff 7fffffff 7ff7eca0
7ff7ecb0 100012dc 004112f4 100130b0 00000440 80482518 85e5be2c 83786200
85e5be41 ffffffff 00000000 ffffffff 803fe804 870079f4 00000001 810ea660
00000002 86d15180 7fffffff 8042cd48 85e5be40 83786200 85e5be40 80189c58
00000001 8754b648 80139e08 861c1e58 861c1e58 80189e90 00000000 8754b648
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482518>] schedule_timeout+0x2c/0xd4
[<8042cd48>] inet_csk_wait_for_connect+0x90/0x120
[<8042ce3c>] inet_csk_accept+0x64/0x15c
[<8044e3b0>] inet_accept+0x34/0xc8
[<803fb588>] sys_accept+0xe8/0x1a4
[<8010c180>] stack_done+0x20/0x3c
and S 80482CF0 0 1627 1 1633 1553 (NOTLB)
Stack : 8012dbcc 8012dba8 80347cd4 00000000 8750beb8 00000000 7fa26b58 7fa26b60
7fa26be0 10011cc8 ffffffff 00000000 7fa26d10 80482cf0 0f27abcb 801091cc
2aac2000 8015ba6c 00000001 8750be90 8750bf20 00000001 8013d688 8013d65c
80128638 801285d0 00000010 805bc148 00000001 876010c8 822a7eb8 00000000
00019f55 347bf960 8013d5ec 80518550 874dd998 87f45ae0 8750bf30 80128704
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482cf0>] do_nanosleep+0x50/0x98
[<8013d688>] hrtimer_nanosleep+0x58/0x178
[<8013d824>] sys_nanosleep+0x7c/0x90
[<8010c180>] stack_done+0x20/0x3c
dagent S 80482518 0 1633 1 1761 1627 (NOTLB)
Stack : 81127ca0 8650fc20 80518af8 9004a001 00000000 7fffffff 8650fc90 8650fcc8
86db59e0 000005dc 00000000 00000000 00000008 80482518 00000000 00000000
00000000 80519030 000200d2 8014f064 8017ec7c 8017e87c 85d43518 2ab01978
00000000 86db59e0 8650fc90 80402cd4 803fe804 8755b668 2aac3000 80519030
00000001 8755b668 80139e08 86db69f8 86db69f8 810fdd00 00000000 8755b668
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482518>] schedule_timeout+0x2c/0xd4
[<80402cd4>] wait_for_packet+0x134/0x190
[<80402e20>] skb_recv_datagram+0xf0/0x124
[<80446c3c>] udp_recvmsg+0x84/0x26c
[<803fe95c>] sock_common_recvmsg+0x44/0x60
[<803fa390>] sock_recvmsg+0xb4/0xe4
[<803fbff4>] sys_recvmsg+0x120/0x1d4
[<8010c180>] stack_done+0x20/0x3c
_dmesglog S 80126598 0 1725 1 15125 1809 1761 (NOTLB)
Stack : 8013d8d0 8013d860 870aae94 8753ba38 fffffe00 8753baf0 8753baf0 00000000
8753ba38 00000004 00000001 00000000 7fbbba60 80126598 86156918 85831f30
00000000 00030002 7fbbb978 00000012 00000000 8753ba38 8011dfc8 873173a8
873173a8 00000012 00000000 8753ba38 8011dfc8 00000000 00000000 85831f08
ffffffff 7fbbba60 00000000 00000004 ffffffff 1000f018 00000000 00000000
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80126598>] do_wait+0x2f0/0x42c
[<801267b0>] sys_wait4+0x3c/0x48
[<801267cc>] sys_waitpid+0x10/0x24
[<8010c180>] stack_done+0x20/0x3c
gpio_poller S 80482CF0 0 1761 1 1725 1633 (NOTLB)
Stack : dc000000 00000002 8334e000 ffffff9c 86597eb8 00000000 00000000 00000002
00400e00 10010a48 ffffffff 00000000 7f91ebe8 80482cf0 8711c7a0 806573a0
80178f64 875c5634 00000001 10010a48 86597f20 00000001 8013d688 8013d65c
80128638 801285d0 00000000 00000000 822a7eb9 84331eb8 00000000 00000000
00019f36 0bbe68a0 8013d5ec 80518550 8753a1d8 8016cd30 86597f30 80128704
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482cf0>] do_nanosleep+0x50/0x98
[<8013d688>] hrtimer_nanosleep+0x58/0x178
[<8013d824>] sys_nanosleep+0x7c/0x90
[<8010c180>] stack_done+0x20/0x3c
syslogd S 80482518 0 1809 1 1810 1725 (NOTLB)
Stack : 80481630 00000004 7fbf2978 00000004 86c68ca0 7fffffff 00000010 00000104
00000004 85837f20 85837de8 85837de4 85837de0 80482518 804036a0 8011ee28
85837b18 80183ef8 85836000 85837af8 86c68ca0 803facb8 86c68ca0 00000000
86c68ca0 00000000 00000010 801844f0 85837ce0 a13ec180 803a7c00 803a7bc0
80184038 00000000 00000000 00000001 86c68ca0 00000000 811a8ca8 8011dfc8
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482518>] schedule_timeout+0x2c/0xd4
[<801844f0>] do_select+0x32c/0x3a8
[<80184798>] core_sys_select+0x22c/0x318
[<80184970>] sys_select+0xec/0x234
[<8010c180>] stack_done+0x20/0x3c
klogd R running 0 1810 1 1811 1809 (NOTLB)
sh S 80482518 0 1811 1 29735 2405 1810 (NOTLB)
Stack : 81193e30 80632000 00000001 810e3fe0 80632000 7fffffff 7fffffff 00000001
7f9eb308 00000000 00000000 8063212c 854c9260 80482518 80342bfc 80632124
80616400 8033bb88 16040a00 00000000 00000000 8034328c 0000001e 80616400
80632000 80632000 7fffffff 8033d7ec 9004a000 8063200c 00000000 0000001e
00000000 811fb938 8011dfc8 8063212c 8063212c 8013d860 00000000 811fb938
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482518>] schedule_timeout+0x2c/0xd4
[<8033d7ec>] read_chan+0x2e8/0x5e4
[<803383a4>] tty_read+0xa0/0xec
[<8016d720>] vfs_read+0xb8/0x1a0
[<8016db20>] sys_read+0x54/0x98
[<8010c180>] stack_done+0x20/0x3c
md0_linear S 80482518 0 2145 6 2303 193 (L-TLB)
Stack : 80626600 7fffffff 84bd4888 803e2eb8 84bd4880 7fffffff 84bd4888 803e2eb8
00000000 00000000 00000000 00000000 00000000 80482518 80626600 7fffffff
84bd4888 803e2eb8 00000000 00000000 803c37ac 8013978c 9000a003 000000a0
84bd4880 7fffffff 84bd4888 803e2fc0 00000000 00000000 00000001 8754ba58
00000000 8754ba58 80139e08 84bd4888 84bd4888 00000000 00000000 8754ba58
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482518>] schedule_timeout+0x2c/0xd4
[<803e2fc0>] md_thread+0x108/0x16c
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
xfsbufd S 80482588 0 2303 6 2304 2145 (L-TLB)
Stack : 00001000 81156600 85832980 8060f520 01945c81 0000012c 8417ff10 858324b0
00001194 00000000 00000000 00000000 00000000 80482588 85832480 858324b0
00001194 00000000 805c2ab0 811aa148 01945c81 8012df90 87572a58 805c21c8
00000003 85832494 85832480 804825d8 9000a003 00000000 00000000 00000000
802f8454 802f85a4 85832480 802f83e0 00000000 00000000 8417ff60 8417ff60
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482588>] schedule_timeout+0x9c/0xd4
[<804825d8>] schedule_timeout_interruptible+0x18/0x24
[<802f8454>] xfsbufd+0x74/0x230
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
xfssyncd S 80482588 0 2304 6 2419 2303 (L-TLB)
Stack : 849cdf18 849cdef0 9000a000 ffff00fe 0194664d 00002328 849cdf18 00002328
849cdf68 00000000 00000000 00000000 00000000 80482588 8118e500 00002328
849cdf68 00000000 8610cb7c 805c2b00 0194664d 8012df90 8755a628 805c21c8
849cdf68 849cdf68 8118e500 804825d8 9000a003 000000a0 0000008c 8048130c
802ff66c 802ff73c 00000000 00000000 84469c68 fffffffc 849cdf68 849cdf68
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<80482588>] schedule_timeout+0x9c/0xd4
[<804825d8>] schedule_timeout_interruptible+0x18/0x24
[<802ff66c>] xfssyncd+0x48/0x18c
[<801398bc>] kthread+0x98/0xe0
[<80104c38>] kernel_thread_helper+0x10/0x18
logger S 8017BEE4 0 2314 1 2530 2424 (NOTLB)
Stack : 00000000 00000038 858637e0 8017d5e8 874ea600 2aac2000 84d2bf18 874ea600
0000000a 7fff7bf8 00000074 00000070 00000000 8017bee4 7fff7768 00004000
858637e0 803fb910 00000000 811d70f8 80139e08 874ea600 874ea600 00000000
00000000 811d70f8 80139e08 84d2be3c 84d2be3c 00000000 856fa7fc 8017c5a4
00000000 00000000 00000000 00000b3b 00000000 00000000 00001000 856fa878
...
Call Trace:
[<8048190c>] schedule+0x810/0x97c
[<8017bee4>] pipe_wait+0x84/0xb4
[<8017c5a4>] pipe_readv+0x264/0x30c
[<8017c670>] pipe_read+0x24/0x30
