On 09/28/2014 11:08 PM, NeilBrown wrote:
On Sun, 28 Sep 2014 22:56:19 -0500 BillStuff <billstuff2001@xxxxxxxxxxxxx>
wrote:
On 09/28/2014 09:25 PM, NeilBrown wrote:
On Fri, 26 Sep 2014 17:33:58 -0500 BillStuff <billstuff2001@xxxxxxxxxxxxx>
wrote:
Hi Neil,
I found something that looks similar to the problem described in
"Re: seems like a deadlock in workqueue when md do a flush" from Sept 14th.
It's on 3.14.19 with 7 recent patches for fixing raid1 recovery hangs.
on this array:
md3 : active raid5 sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
104171200 blocks level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
bitmap: 1/5 pages [4KB], 2048KB chunk
I was running a test doing parallel kernel builds, read/write loops, and
disk add / remove / check loops,
on both this array and a raid1 array.
I was trying to stress test your recent raid1 fixes, which went well,
but then after 5 days,
the raid5 array hung up with this in dmesg:
I think this is different to the workqueue problem you mentioned, though as I
don't know exactly what caused either I cannot be certain.
From the data you provided it looks like everything is waiting on
get_active_stripe(), or on a process that is waiting on that.
That seems pretty common whenever anything goes wrong in raid5 :-(
The md3_raid5 task is listed as blocked, but not stack trace is given.
If the machine is still in the state, then
cat /proc/1698/stack
might be useful.
(echo t > /proc/sysrq-trigger is always a good idea)
Might this help? I believe the array was doing a "check" when things
hung up.
It looks like it was trying to start doing a 'check'.
The 'resync' thread hadn't been started yet.
What is 'kthreadd' doing?
My guess is that it is in try_to_free_pages() waiting for writeout
for some xfs file page onto the md array ... which won't progress until
the thread gets started.
That would suggest that we need an async way to start threads...
Thanks,
NeilBrown
I suspect your guess is correct:
kthreadd D c106ea4c 0 2 0 0x00000000
e9d6db58 00000046 e9d6db4c c106ea4c ce493c00 00000001 1e9bb7bd 0001721a
c17d6700 c17d6700 d3b6a880 e9d38510 f2cf4c00 00000000 f2e51c00 e9d6db60
f3cec0b6 e9d38510 f2e51d14 f2e51d00 f2e51c00 00043132 00000964 0000a4b0
Call Trace:
[<c106ea4c>] ? update_blocked_averages+0x1ec/0x700
[<f3cec0b6>] ? xlog_cil_force_lsn+0xd6/0x1c0 [xfs]
[<f3cc077b>] ? xfs_bmbt_get_all+0x2b/0x40 [xfs]
[<c153e7f3>] schedule+0x23/0x60
[<f3ceaa71>] _xfs_log_force_lsn+0x141/0x270 [xfs]
[<c1069ca0>] ? wake_up_process+0x40/0x40
[<f3ceabd8>] xfs_log_force_lsn+0x38/0x90 [xfs]
[<f3cd7ee0>] __xfs_iunpin_wait+0x80/0x100 [xfs]
[<f3cdb02d>] ? xfs_iunpin_wait+0x1d/0x30 [xfs]
[<c10799d0>] ? autoremove_wake_function+0x40/0x40
[<f3cdb02d>] xfs_iunpin_wait+0x1d/0x30 [xfs]
[<f3c99938>] xfs_reclaim_inode+0x58/0x2f0 [xfs]
[<f3c99e04>] xfs_reclaim_inodes_ag+0x234/0x330 [xfs]
[<f3c9a6a1>] ? xfs_inode_set_reclaim_tag+0x91/0x150 [xfs]
[<c115cc41>] ? fsnotify_clear_marks_by_inode+0x21/0xe0
[<f3ca5ac5>] ? xfs_fs_destroy_inode+0xa5/0xd0 [xfs]
[<c113b061>] ? destroy_inode+0x31/0x50
[<c113b160>] ? evict+0xe0/0x160
[<f3c9a7ad>] xfs_reclaim_inodes_nr+0x2d/0x40 [xfs]
[<f3ca5103>] xfs_fs_free_cached_objects+0x13/0x20 [xfs]
[<c11278ce>] super_cache_scan+0x12e/0x140
[<c10f2bb5>] shrink_slab_node+0x125/0x280
[<c1101a1c>] ? compact_zone+0x2c/0x450
[<c10f3489>] shrink_slab+0xd9/0xf0
[<c10f557d>] try_to_free_pages+0x25d/0x4f0
[<c10eb96d>] __alloc_pages_nodemask+0x52d/0x820
[<c103c452>] copy_process.part.47+0xd2/0x14e0
[<c153e254>] ? __schedule+0x224/0x7a0
[<c105ad80>] ? kthread_create_on_node+0x110/0x110
[<c105ad80>] ? kthread_create_on_node+0x110/0x110
[<c103da01>] do_fork+0xc1/0x320
[<c105ad80>] ? kthread_create_on_node+0x110/0x110
[<c103dc8d>] kernel_thread+0x2d/0x40
[<c105b542>] kthreadd+0x122/0x170
[<c1541837>] ret_from_kernel_thread+0x1b/0x28
[<c105b420>] ? kthread_create_on_cpu+0x60/0x60
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
