> Rolf Eike Beer <eike-kernel@sf-tec.de> wrote:
> I'm experiencing a very annoying system lockup for some days. The
> setup is as follows:
...
> Now I try to move the data from one pv to the other using pv. This
> prints out the current state (currently 10.9%) and then starts doing
> something. Two minutes later the kernel will complain:
>
> INFO: task jbd2/dm-2-8:3616 blocked for more than 120 seconds.
...
I was originally seeing this on 2.6.37,
Do you have a raid check running via cron, perhaps cron.weekly
or cron.monthly?
Yesterday I had the same problem on 2.6.18-194.32.1.el5.
Previously I saw it repeatedly on 2.6.32.26-175.fc12.x86_64.
In at least some cases mdadm was doing a rsync or repair.
I don't yes know if if that was true in all cases. There
is a recently fixed bug in the resync / repair code which
may be related. I don't know if the fix has been it into
any distros yet.
I haven't been able to reproduce the problem in a very useful
way. It's always occurred while there was significant LVM
activity (such as a pvmove) at the same time has heavy activity
on the filesystem, either an rsync or extracting a large tar,
and probably a raid resync all at the same time. I'm attempting
to reproduce it with less activity on the system but it hasn't
happened yet. I now suspect it WON'T occur without a combination
of those processes. I was running my pvmove with "-vv" yesterday
and got this output:
Updating volume group metadata
Found volume group "clones"
Found volume group "clones"
Suspending clones-c44_s1 (253:4) with device flush
Suspending clones-pvmove0 (253:7) with device flush
Found volume group "clones"
Found volume group "clones"
Suspending clones-kevinbowden (253:5) with device flush
It "froze up" after that output.
--
Ray Morris
support@bettercgi.com
Strongbox - The next generation in site security:
http://www.bettercgi.com/strongbox/
Throttlebox - Intelligent Bandwidth Control
http://www.bettercgi.com/throttlebox/
Strongbox / Throttlebox affiliate program:
http://www.bettercgi.com/affiliates/user/register.php
On Tue, 8 Mar 2011 10:38:38 +0100
Rolf Eike Beer <eike-kernel@sf-tec.de> wrote:
> Hi all,
>
> I'm experiencing a very annoying system lockup for some days. The
> setup is as follows:
>
> -two pairs of SATA disks that are bundled into a software raid 1 each
> -each of the raid devices is a physical volume
> -a volume group that includes both pv's
> -all mounted volumes (including root and swap) are in that vg
>
> The machine is a Xeon E5520 with 16G RAM that is otherwise idle, so
> swap shouldn't matter. And from what I read out of the documentation
> this all looks perfectly sane, but:
>
> Now I try to move the data from one pv to the other using pv. This
> prints out the current state (currently 10.9%) and then starts doing
> something. Two minutes later the kernel will complain:
>
> INFO: task jbd2/dm-2-8:3616 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
> message. jbd2/dm-2-8 D ffff88008c211200 0 3616 2
> 0x00000000 ffff88045d7f9bb0 0000000000000046 ffff88045debf000
> ffff88045d7f8010 0000000000004000 0000000000011200 ffff88045d7f9fd8
> 0000000000011200 ffff88045d870cc0 ffff88045d7f9fd8 ffff88045d7f9aa0
> 0000000000000202 Call Trace:
> [<ffffffff8126693a>] ? 0xffffffff8126693a
> [<ffffffff81181eac>] ? 0xffffffff81181eac
> [<ffffffff81267e6e>] ? 0xffffffff81267e6e
> [<ffffffff8104db29>] ? 0xffffffff8104db29
> [<ffffffff81306d38>] 0xffffffff81306d38
> [<ffffffff810ce96c>] 0xffffffff810ce96c
> [<ffffffff8130718a>] 0xffffffff8130718a
> [<ffffffff810ce931>] ? 0xffffffff810ce931
> [<ffffffff810ce931>] ? 0xffffffff810ce931
> [<ffffffff81307226>] 0xffffffff81307226
> [<ffffffff81047104>] ? 0xffffffff81047104
> [<ffffffff810ce8f6>] 0xffffffff810ce8f6
> [<ffffffff81152a70>] 0xffffffff81152a70
> [<ffffffff81306c0a>] ? 0xffffffff81306c0a
> [<ffffffff8103b24a>] ? 0xffffffff8103b24a
> [<ffffffff81155c7e>] 0xffffffff81155c7e
> [<ffffffff810470cc>] ? 0xffffffff810470cc
> [<ffffffff81155bb0>] ? 0xffffffff81155bb0
> [<ffffffff81046c8f>] 0xffffffff81046c8f
> [<ffffffff81003654>] 0xffffffff81003654
> [<ffffffff81046c0e>] ? 0xffffffff81046c0e
> [<ffffffff81003650>] ? 0xffffffff81003650
>
> For the records, pvmove --version gives
>
> LVM version: 2.02.73(2) (2010-09-18)
> Library version: 1.02.54 (2010-08-18)
> Driver version: 4.18.0
>
> When I restart the system now and start pvmove again the status
> number increased so it actually _did_ some work, but afterwards it
> hangs the machine. The system is still alive (I can ping, start new
> screens, do some simple commands) but for me it looks like the IO
> stuff is completely hanging. Last time I tried to move some files out
> of the way but that changes never made it to disk. I was originally
> seeing this on 2.6.37, upgrading to .2 didn't make a difference. The
> machine is connected to a serial console and I can easily send some
> sysrq requests from a still active ssh session so things are not
> completely dead. Here is some info I collected from sysrq:
>
> SysRq : Show Memory
> Mem-Info:
> Node 0 DMA per-cpu:
> CPU 0: hi: 0, btch: 1 usd: 0
> CPU 1: hi: 0, btch: 1 usd: 0
> CPU 2: hi: 0, btch: 1 usd: 0
> CPU 3: hi: 0, btch: 1 usd: 0
> CPU 4: hi: 0, btch: 1 usd: 0
> CPU 5: hi: 0, btch: 1 usd: 0
> CPU 6: hi: 0, btch: 1 usd: 0
> CPU 7: hi: 0, btch: 1 usd: 0
> Node 0 DMA32 per-cpu:
> CPU 0: hi: 186, btch: 31 usd: 104
> CPU 1: hi: 186, btch: 31 usd: 159
> CPU 2: hi: 186, btch: 31 usd: 167
> CPU 3: hi: 186, btch: 31 usd: 174
> CPU 4: hi: 186, btch: 31 usd: 160
> CPU 5: hi: 186, btch: 31 usd: 153
> CPU 6: hi: 186, btch: 31 usd: 142
> CPU 7: hi: 186, btch: 31 usd: 109
> Node 0 Normal per-cpu:
> CPU 0: hi: 186, btch: 31 usd: 165
> CPU 1: hi: 186, btch: 31 usd: 93
> CPU 2: hi: 186, btch: 31 usd: 113
> CPU 3: hi: 186, btch: 31 usd: 174
> CPU 4: hi: 186, btch: 31 usd: 59
> CPU 5: hi: 186, btch: 31 usd: 89
> CPU 6: hi: 186, btch: 31 usd: 40
> CPU 7: hi: 186, btch: 31 usd: 94
> active_anon:2891 inactive_anon:3 isolated_anon:0
> active_file:22394 inactive_file:1476 isolated_file:0
> unevictable:3514 dirty:5 writeback:1 unstable:0
> free:4055871 slab_reclaimable:2434 slab_unreclaimable:4352
> mapped:1775 shmem:80 pagetables:453 bounce:0
> Node 0 DMA free:15904kB min:12kB low:12kB high:16kB active_anon:0kB
> inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:15680kB mlocked:0kB
> dirty:0kB writeback:0kB ms
> lowmem_reserve[]: 0 2173 16061 16061
> Node 0 DMA32 free:2268600kB min:2192kB low:2740kB high:3288kB
> active_anon:1288kB inactive_anon:4kB active_file:868kB
> inactive_file:116kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:2225720kB mlocked:0kB dirty:o
> lowmem_reserve[]: 0 0 13887 13887
> Node 0 Normal free:13938980kB min:14016kB low:17520kB high:21024kB
> active_anon:10276kB inactive_anon:8kB active_file:88708kB
> inactive_file:5788kB unevictable:14056kB isolated(anon):0kB
> isolated(file):0kB present:14220800kB mloco
> lowmem_reserve[]: 0 0 0 0
> Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 0*512kB
> 1*1024kB 1*2048kB 3*4096kB = 15904kB
> Node 0 DMA32: 716*4kB 637*8kB 524*16kB 399*32kB 322*64kB 183*128kB
> 106*256kB 57*512kB 45*1024kB 44*2048kB 489*4096kB = 2268600kB
> Node 0 Normal: 1505*4kB 4432*8kB 3658*16kB 2488*32kB 1736*64kB
> 1207*128kB 836*256kB 1031*512kB 901*1024kB 314*2048kB 2731*4096kB =
> 13938980kB 24670 total pagecache pages
> 1 pages in swap cache
> Swap cache stats: add 51604, delete 51603, find 24440/27591
> Free swap = 33554424kB
> Total swap = 33554428kB
> 4194288 pages RAM
> 74658
>
> At this point the output simply was cut off. It continued with those
> when I issues the next sysrq command a while later:
>
> SysRq : Show backtrace of all active CPUs
> sending NMI to all CPUs:
> NMI backtrace for cpu 0
> CPU 0
> Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE
> iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
> nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables
> ipmi_devintf ipmi_poweroff ipmi_si ipi
>
> Pid: 10066, comm: bash Not tainted 2.6.37.2 #2 Intel Corporation
> S5500BC/S5500BC
> RIP: 0010:[<ffffffff81199a13>] [<ffffffff81199a13>]
> 0xffffffff81199a13 RSP: 0018:ffff88045d55bdc8 EFLAGS: 00000807
> RAX: 0000000072b54a00 RBX: 0000000000000000 RCX: 00000000001a36f0
> RDX: 0000000000036e4e RSI: 0000000000000001 RDI: 0000000000068dbc
> RBP: ffff88045d55bdc8 R08: 0000000000000010 R09: ffffffff8143d4f8
> R10: ffff8800000bcd80 R11: 0000000000000000 R12: 0000000000001000
> R13: 0000000000000002 R14: 0000000000000092 R15: 0000000000000002
> FS: 00007fb99d455700(0000) GS:ffff88008c200000(0000)
> knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0:
> 0000000080050033 CR2: 00007fb99f6e6368 CR3: 0000000456682000 CR4:
> 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400 Process bash (pid: 10066, threadinfo
> ffff88045d55a000, task ffff88045f59f320) S 000000000000000085aa>]
> 0xffffffff810185aa [<ff1fef86>] 0xffffffff811fef86
> [<fff810adda4>] 0xffffffff81 48 6b 94 02 98 00 00 00 3e 48 89 c8 f7
> e2 <48> 8d 7a 01 ff 15 f3 83 27 00 c9 c3 55 48 69 cf 1c 43 00 00 48
> Call Trace:
> [<ffffffff8101811a>] 0xffffffff8101811a
> [<ffffffff810185aa>] 0xffffffff810185aa
> [<ffffffff8101b0e0>] 0xffffffff8101b0e0
> [<ffffffff810186c6>] 0xffffffff810186c6
> [<ffffffff811feca9>] 0xffffffff811feca9
> [<ffffffff811fef86>] 0xffffffff811fef86
> [<ffffffff811ff054>] 0xffffffff811ff054
> [<ffffffff810f1d63>] 0xffffffff810f1d63
> [<ffffffff810adc39>] 0xffffffff810adc39
> [<ffffffff810adda4>] 0xffffffff810adda4
> [<ffffffff810028fb>] 0xffffffff810028fb
> Pid: 10066, comm: bash Not tainted 2.6.37.2 #2
> Call Trace:
> <NMI> [<ffffffff8100a51c>] ? 0xffffffff8100a51c
> [<ffffffff81018850>] 0xffffffff81018850
> [<ffffffff8100445c>] 0xffffffff8100445c
> [<ffffffff81308fba>] 0xffffffff81308fba
> [<ffffffff81199a13>] ? 0xffffffff81199a13
> <<EOE>> [<ffffffff8101811a>] 0xffffffff8101811a
> [<ffffffff810185aa>] 0xffffffff810185aa
> [<ffffffff8101b0e0>] 0xffffffff8101b0e0
> [<ffffffff810186c6>] 0xffffffff
> [<ffffffff810f1d63>] 0x028fb
> NMI backtrace for cpu 1
> onntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables
> ipmi_devintf ipmi_poweroff ipmi_si ipmi_watchdog ipmi_msghandler tun
> bridge stp llc kvm_intel kvm bonding ipv6 ioatdma dca rtc_cmos e1000e
> i2c_i801 pcspkr processoi
>
> Pid: 0, comm: kworker/0:0 Not tainted 2.6.37.2 #2 Intel Corporation
> S5500BC/S5500BC
> RIP: 0010:[<ffffffff811cc13a>] [<ffffffff811cc13a>]
> 0xffffffff811cc13a RSP: 0018:ffff88046e119e78 EFLAGS: 00000046
> RAX: 0000000000000020 RBX: 120905f0a66d1ce7 RCX: 0000000000000001
> RDX: 0000000000000000 RSI: ffff88046e118010 RDI: ffff88046e119fd8
> RBP: ffff88046e119ec8 R08: 0000000000000000 R09: 000000000000000b
> R10: 0000000100092889 R11: 0000000000000000 R12: 0000000000000020
> R13: 0000000000000008 R14: 0000000000000000 R15: 0000000000000000
> FS: 0000000000000000(0000) GS:ffff88008c220000(0000)
> knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0:
> 000000008005003b CR2: 00007f1a6d71f020 CR3: 00000000013f7000 CR4:
> 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400 Process kworker/0:0 (pid: 0, threadinfo
> ffff88046e118000, task ffff88046e117980)
> Stack:
> 0000000000000000 000000000000ea50 ffff88046e119ec8 ffffffff8126fe15
> 0000026151d50c00 000000016e118010 ffff88046e118010 ffff88008c2387a0
> ffff88008c238670 0000000000000000 ffff88046e119ee8 ffffffff8126f0c4
> Call Trace:
> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> Code: 8d b7 38 e0 ff ff a8 08 75 21 31 c9 48 89 f0 48 89 ca 0f 01 c8
> 0f ae f0 48 8b 87 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c96fe15
> [<ffffffff8126f0c4>] 0xtainted 2.6.37.2 #2
> Callc
> [<ffffffff813 0xffffffff8126f0c4
> [<able_filter ip6_tables ipt_MASQUERADE iptable_nat nf_nat
> nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT
> xt_tcpudp iptable_filter ip_tables x_tables ipmi_devintf
> ipmi_poweroff ipmi_si ipmi_watchdog ipmi_msgi
>
> Pid: 0, comm: kworker/0:1 Not tainted 2.6.37.2 #2 Intel Corporation
> S5500BC/S5500BC
> RIP: 0010:[<ffffffff811cc13a>] [<ffffffff811cc13a>]
> 0xffffffff811cc13a RSP: 0018:ffff88046e129e78 EFLAGS: 00000046
> RAX: 0000000000000020 RBX: 120905f097675bfd RCX: 0000000000000001
> RDX: 0000000000000000 RSI: ffff88046e128010 RDI: ffff88046e129fd8
> RBP: ffff88046e129ec8 R08: 0000000000000000 R09: 0000000000000001
> R10: 0000000100092886 R11: 0000000000000000 R12: 0000000000000020
> R13: 0000000000000008 R14: 0000000000000000 R15: 0000000000000000
> FS: 0000000000000000(0000) GS:ffff88008c240000(0000)
> knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0:
> 000000008005003b CR2: 00007fce81cb00c0 CR3: 00000000013f7000 CR4:
> 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400 Process kworker/0:1 (pid: 0, threadinfo
> ffff88046e128000, task ffff88046e116000)
> Stack:
> 0000000000000000 000000000000ea50 ffff88046e129ec8 ffffffff8126fe15
> 000002616fa27100 000000026e128010 ffff88046e128010 ffff88008c2587a0
> ffff88008c258670 0000000000000000 ffff88046e129ee8 ffffffff8126f0c4
> Call Trace:
> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> Code: 8d b7 38 e0 ff ff a8 08 75 21 31 c9 48 89 f0 48 89 ca 0f 01 c8
> 0f ae f0 48 8b 87 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9
> <e8> 79 1b e8 ff 48 89 c7 48 29 df e8 c5 9b e6 ff 48 89 c3 49 89
> Call Trace:
> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> Pid: 0, comm: kworker/0:1 Not tainted 2.6.37.2 #2
> Call Trace:
> <NMI> [<ffffffff8100a51c>] ? 0xffffffff8100a51c
> [<ffffffff81018850>] 0xffffffff81018850
> [<ffffffff8100445c>] 0xffffffff8100445c
> [<ffffffff81308fba>] 0xffffffff81308fba
> [<ffffffff811cc13a>] ? 0xffffffff811cc13a
> <<EOE>> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> NMI backtrace for cpu 3
> CPU 3
> Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE
> iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
> nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables
> ipmi_devintf ipmi_poweroff ipmi_si ipi
>
> Pid: 0, comm: kworker/0:1 Not tainted 2.6.37.2 #2 Intel Corporation
> S5500BC/S5500BC
> RIP: 0010:[<ffffffff811cc13a>] [<ffffffff811cc13a>]
> 0xffffffff811cc13a RSP: 0018:ffff88046e157e78 EFLAGS: 00000046
> RAX: 0000000000000020 RBX: 120905f0b5353da5 RCX: 0000000000000001
> RDX: 0000000000000000 RSI: ffff88046e156010 RDI: ffff88046e157fd8
> RBP: ffff88046e157ec8 R08: 0000000000000000 R09: 0000000000000001
> R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000020
> R13: 0000000000000008 R14: 0000000000000000 R15: 0000000000000000
> FS: 0000000000000000(0000) GS:ffff88008c260000(0000)
> knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0:
> 000000008005003b CR2: 00007f6da87f79d0 CR3: 00000000013f7000 CR4:
> 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400 Process kworker/0:1 (pid: 0, threadinfo
> ffff88046e156000, task ffff88046e12a660)
> Stack:
> 0000000000000000 000000000000ea50 ffff88046e157ec8 ffffffff8126fe15
> 00000261fa64e700 000000036e156010 ffff88046e156010 ffff88008c2787a0
> ffff88008c278670 0000000000000000 ffff88046e157ee8 ffffffff8126f0c4
> Call Trace:
> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> Code: 8d b7 38 e0 ff ff a8 08 75 21 31 c9 48 89 f0 48 89 ca 0f 01 c8
> 0f ae f0 48 8b 87 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9
> <e8> 79 1b e8 ff 48 89 c7 48 29 df e8 c5 9b e6 ff 48 89 c3 49 89
> Call Trace:
> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> Pid: 0, comm: kworker/0:1 Not tainted 2.6.37.2 #2
> Call Trace:
> <NMI> [<ffffffff8100a51c>] ? 0xffffffff8100a51c
> [<ffffffff81018850>] 0xffffffff81018850
> [<ffffffff8100445c>] 0xffffffff8100445c
> [<ffffffff81308fba>] 0xffffffff81308fba
> [<ffffffff811cc13a>] ? 0xffffffff811cc13a
> <<EOE>> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> NMI backtrace for cpu 4
> CPU 4
> Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE
> iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
> nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables
> ipmi_devintf ipmi_poweroff ipmi_si ipi
>
> Pid: 0, comm: kworker/0:1 Not tainted 2.6.37.2 #2 Intel Corporation
> S5500BC/S5500BC
> RIP: 0010:[<ffffffff811cc13a>] [<ffffffff811cc13a>]
> 0xffffffff811cc13a RSP: 0018:ffff88046e167e78 EFLAGS: 00000046
> RAX: 0000000000000020 RBX: 120905f0b81122a9 RCX: 0000000000000001
> RDX: 0000000000000000 RSI: ffff88046e166010 RDI: ffff88046e167fd8
> RBP: ffff88046e167ec8 R08: 0000000000000000 R09: 0000000000000001
> R10: 0000000100092880 R11: 0000000000000000 R12: 0000000000000020
> R13: 0000000000000008 R14: 0000000000000000 R15: 0000000000000000
> FS: 0000000000000000(0000) GS:ffff88008c280000(0000)
> knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0:
> 000000008005003b CR2: 00007f551f028710 CR3: 00000000013f7000 CR4:
> 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400 Process kworker/0:1 (pid: 0, threadinfo
> ffff88046e166000, task ffff88046e15ccc0)
> Stack:
> ffff88046e167e88 000000000000ea50 ffff88046e167ec8 ffffffff8126fe15
> 000002620781d600 000000046e166010 ffff88046e166010 ffff88008c2987a0
> ffff88008c298670 0000000000000000 ffff88046e167ee8 ffffffff8126f0c4
> Call Trace:
> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> Code: 8d b7 38 e0 ff ff a8 08 75 21 31 c9 48 89 f0 48 89 ca 0f 01 c8
> 0f ae f0 48 8b 87 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9
> <e8> 79 1b e8 ff 48 89 c7 48 29 df e8 c5 9b e6 ff 48 89 c3 49 89
> Call Trace:
> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> Pid: 0, comm: kworker/0:1 Not tainted 2.6.37.2 #2
> Call Trace:
> <NMI> [<ffffffff8100a51c>] ? 0xffffffff8100a51c
> [<ffffffff81018850>] 0xffffffff81018850
> [<ffffffff8100445c>] 0xffffffff8100445c
> [<ffffffff81308fba>] 0xffffffff81308fba
> [<ffffffff811cc13a>] ? 0xffffffff811cc13a
> <<EOE>> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> NMI backtrace for cpu 5
> CPU 5
> Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE
> iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
> nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables
> ipmi_devintf ipmi_poweroff ipmi_si ipi
>
> Pid: 0, comm: kworker/0:1 Not tainted 2.6.37.2 #2 Intel Corporation
> S5500BC/S5500BC
> RIP: 0010:[<ffffffff811cc13a>] [<ffffffff811cc13a>]
> 0xffffffff811cc13a RSP: 0018:ffff88046e199e78 EFLAGS: 00000046
> RAX: 0000000000000020 RBX: 120905f0d08b7a62 RCX: 0000000000000001
> RDX: 0000000000000000 RSI: ffff88046e198010 RDI: ffff88046e199fd8
> RBP: ffff88046e199ec8 R08: 0000000000000000 R09: 0000000000000001
> R10: 000000010008da5d R11: 0000000000000000 R12: 0000000000000020
> R13: 0000000000000008 R14: 0000000000000000 R15: 0000000000000000
> FS: 0000000000000000(0000) GS:ffff88008c2a0000(0000)
> knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0:
> 000000008005003b CR2: 00007f1a6e01daa0 CR3: 00000000013f7000 CR4:
> 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400 Process kworker/0:1 (pid: 0, threadinfo
> ffff88046e198000, task ffff88046e193320)
> Stack:
> 0000000000000000 000000000000ea50 ffff88046e199ec8 ffffffff8126fe15
> 00000261f5dd3c00 000000056e198010 ffff88046e198010 ffff88008c2b87a0
> ffff88008c2b8670 0000000000000000 ffff88046e199ee8 ffffffff8126f0c4
> Call Trace:
> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> Code: 8d b7 38 e0 ff ff a8 08 75 21 31 c9 48 89 f0 48 89 ca 0f 01 c8
> 0f ae f0 48 8b 87 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9
> <e8> 79 1b e8 ff 48 89 c7 48 29 df e8 c5 9b e6 ff 48 89 c3 49 89
> Call Trace:
> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> Pid: 0, comm: kworker/0:1 Not tainted 2.6.37.2 #2
> Call Trace:
> <NMI> [<ffffffff8100a51c>] ? 0xffffffff8100a51c
> [<ffffffff81018850>] 0xffffffff81018850
> [<ffffffff8100445c>] 0xffffffff8100445c
> [<ffffffff81308fba>] 0xffffffff81308fba
> [<ffffffff811cc13a>] ? 0xffffffff811cc13a
> <<EOE>> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> NMI backtrace for cpu 6
> CPU 6
> Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE
> iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
> nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables
> ipmi_devintf ipmi_poweroff ipmi_si ipi
>
> Pid: 0, comm: kworker/0:1 Not tainted 2.6.37.2 #2 Intel Corporation
> S5500BC/S5500BC
> RIP: 0010:[<ffffffff811cc13a>] [<ffffffff811cc13a>]
> 0xffffffff811cc13a RSP: 0018:ffff88046e1a9e78 EFLAGS: 00000046
> RAX: 0000000000000020 RBX: 120905f0d08d05fb RCX: 0000000000000001
> RDX: 0000000000000000 RSI: ffff88046e1a8010 RDI: ffff88046e1a9fd8
> RBP: ffff88046e1a9ec8 R08: 0000000000000000 R09: 0000000000000001
> R10: 000000010009287a R11: 0000000000000000 R12: 0000000000000020
> R13: 0000000000000008 R14: 0000000000000000 R15: 0000000000000000
> FS: 0000000000000000(0000) GS:ffff88008c2c0000(0000)
> knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0:
> 000000008005003b CR2: 00007fffbc74e368 CR3: 00000000013f7000 CR4:
> 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400 Process kworker/0:1 (pid: 0, threadinfo
> ffff88046e1a8000, task ffff88046e1a7980)
> Stack:
> 0000000000000000 000000000000ea50 ffff88046e1a9ec8 ffffffff8126fe15
> 00000261e6d80500 000000066e1a8010 ffff88046e1a8010 ffff88008c2d87a0
> ffff88008c2d8670 0000000000000000 ffff88046e1a9ee8 ffffffff8126f0c4
> Call Trace:
> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> Code: 8d b7 38 e0 ff ff a8 08 75 21 31 c9 48 89 f0 48 89 ca 0f 01 c8
> 0f ae f0 48 8b 87 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9
> <e8> 79 1b e8 ff 48 89 c7 48 29 df e8 c5 9b e6 ff 48 89 c3 49 89
> Call Trace:
> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> Pid: 0, comm: kworker/0:1 Not tainted 2.6.37.2 #2
> Call Trace:
> <NMI> [<ffffffff8100a51c>] ? 0xffffffff8100a51c
> [<ffffffff81018850>] 0xffffffff81018850
> [<ffffffff8100445c>] 0xffffffff8100445c
> [<ffffffff81308fba>] 0xffffffff81308fba
> [<ffffffff811cc13a>] ? 0xffffffff811cc13a
> <<EOE>> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> NMI backtrace for cpu 7
> CPU 7
> Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE
> iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
> nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables
> ipmi_devintf ipmi_poweroff ipmi_si ipi
>
> Pid: 0, comm: kworker/0:1 Not tainted 2.6.37.2 #2 Intel Corporation
> S5500BC/S5500BC
> RIP: 0010:[<ffffffff811cc13a>] [<ffffffff811cc13a>]
> 0xffffffff811cc13a RSP: 0018:ffff88046e1d5e78 EFLAGS: 00000046
> RAX: 0000000000000020 RBX: 120905f0d08e9c03 RCX: 0000000000000001
> RDX: 0000000000000000 RSI: ffff88046e1d4010 RDI: ffff88046e1d5fd8
> RBP: ffff88046e1d5ec8 R08: 0000000000000000 R09: 0000000000000001
> R10: 000000010008da60 R11: 0000000000000000 R12: 0000000000000020
> R13: 0000000000000008 R14: 0000000000000000 R15: 0000000000000000
> FS: 0000000000000000(0000) GS:ffff88008c2e0000(0000)
> knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0:
> 000000008005003b CR2: 00007fc00a159320 CR3: 00000000013f7000 CR4:
> 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400 Process kworker/0:1 (pid: 0, threadinfo
> ffff88046e1d4000, task ffff88046e1a6000)
> Stack:
> 0000000000000000 000000000000ea50 ffff88046e1d5ec8 ffffffff8126fe15
> 0000026204a56a00 000000076e1d4010 ffff88046e1d4010 ffff88008c2f87a0
> ffff88008c2f8670 0000000000000000 ffff88046e1d5ee8 ffffffff8126f0c4
> Call Trace:
> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> Code: 8d b7 38 e0 ff ff a8 08 75 21 31 c9 48 89 f0 48 89 ca 0f 01 c8
> 0f ae f0 48 8b 87 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9
> <e8> 79 1b e8 ff 48 89 c7 48 29 df e8 c5 9b e6 ff 48 89 c3 49 89
> Call Trace:
> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
> Pid: 0, comm: kworker/0:1 Not tainted 2.6.37.2 #2
> Call Trace:
> <NMI> [<ffffffff8100a51c>] ? 0xffffffff8100a51c
> [<ffffffff81018850>] 0xffffffff81018850
> [<ffffffff8100445c>] 0xffffffff8100445c
> [<ffffffff81308fba>] 0xffffffff81308fba
> [<ffffffff811cc13a>] ? 0xffffffff811cc13a
> <<EOE>> [<ffffffff8126fe15>] ? 0xffffffff8126fe15
> [<ffffffff8126f0c4>] 0xffffffff8126f0c4
> [<ffffffff81001d21>] 0xffffffff81001d21
> [<ffffffff8130285d>] 0xffffffff8130285d
>
> SysRq : Show Regs
>
> CPU#0: ctrl: 000000070000000f
> CPU#0: status: 0000000000000000
> CPU#0: overflow: 0000000000000000
> CPU#0: fixed: 0000000000000000
> CPU#0: pebs: 0000000000000000
> CPU#0: active: 0000000000000000
> CPU#0: gen-PMC0 ctrl: 0000000000000000
> CPU#0: gen-PMC0 count: 000000000000abcd
> CPU#0: gen-PMC0 left: 0000000000000000
> CPU#0: gen-PMC1 ctrl: 0000000000000000
> CPU#0: gen-PMC1 count: 0000000000000000
> CPU#0: gen-PMC1 left: 0000000000000000
> CPU#0: gen-PMC2 ctrl: 0000000000000000
> CPU#0: gen-PMC2 count: 0000000000000000
> CPU#0: gen-PMC2 left: 0000000000000000
> CPU#0: gen-PMC3 ctrl: 0000000000000000
> CPU#0: gen-PMC3 count: 0000000000000000
> CPU#0: gen-PMC3 left: 0000000000000000
> CPU#0: fixed-PMC0 count: 0000000000000000
> CPU#0: fixed-PMC1 count: 0000000000000000
> CPU#0: fixed-PMC2 count: 0000000000000000
>
> Sadly "d" doesn't work for whatever reason.
>
> If anyone has a quick idea how to get some more information I will be
> happy to test as I assume finishing that move will take a while
> anyway.
>
> Eike
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
