Hello Nikos,
Applied these patches and tested.
We still see hung_task_timeout back traces and the drbd Resync is blocked.
Attached the back trace, please let me know if you need any other information.
In patch "0002-dm-snapshot-rework-COW-throttling-to-fix-deadlock.patch"
I change "struct wait_queue_head" to "wait_queue_head_t" as i was
getting compilation error with former one.
On Thu, 10 Oct 2019 at 17:33, Nikos Tsironis <ntsironis@xxxxxxxxxxx> wrote:
>
> On 10/10/19 9:34 AM, Guruswamy Basavaiah wrote:
> > Hello,
> > We use 4.4.184 in our builds and the patch fails to apply.
> > Is it possible to give a patch for 4.4.x branch ?
> Hi Guru,
>
> I attach the two patches fixing the deadlock rebased on the 4.4.x branch.
>
> Nikos
>
> >
> > patching Logs.
> > patching file drivers/md/dm-snap.c
> > Hunk #1 succeeded at 19 (offset 1 line).
> > Hunk #2 succeeded at 105 (offset -1 lines).
> > Hunk #3 succeeded at 157 (offset -4 lines).
> > Hunk #4 succeeded at 1206 (offset -120 lines).
> > Hunk #5 FAILED at 1508.
> > Hunk #6 succeeded at 1412 (offset -124 lines).
> > Hunk #7 succeeded at 1425 (offset -124 lines).
> > Hunk #8 FAILED at 1925.
> > Hunk #9 succeeded at 1866 with fuzz 2 (offset -255 lines).
> > Hunk #10 succeeded at 2202 (offset -294 lines).
> > Hunk #11 succeeded at 2332 (offset -294 lines).
> > 2 out of 11 hunks FAILED -- saving rejects to file drivers/md/dm-snap.c.rej
> >
> > Guru
> >
> > On Thu, 10 Oct 2019 at 01:33, Guruswamy Basavaiah <guru2018@xxxxxxxxx> wrote:
> >>
> >> Hello Mike,
> >> I will get the testing result before end of Thursday.
> >> Guru
> >>
> >> On Wed, 9 Oct 2019 at 21:34, Mike Snitzer <snitzer@xxxxxxxxxx> wrote:
> >>>
> >>> On Wed, Oct 09 2019 at 11:44am -0400,
> >>> Nikos Tsironis <ntsironis@xxxxxxxxxxx> wrote:
> >>>
> >>>> On 10/9/19 5:13 PM, Mike Snitzer wrote:> On Tue, Oct 01 2019 at 8:43am -0400,
> >>>>> Nikos Tsironis <ntsironis@xxxxxxxxxxx> wrote:
> >>>>>
> >>>>>> On 10/1/19 3:27 PM, Guruswamy Basavaiah wrote:
> >>>>>>> Hello Nikos,
> >>>>>>> Yes, issue is consistently reproducible with us, in a particular
> >>>>>>> set-up and test case.
> >>>>>>> I will get the access to set-up next week, will try to test and let
> >>>>>>> you know the results before end of next week.
> >>>>>>>
> >>>>>>
> >>>>>> That sounds great!
> >>>>>>
> >>>>>> Thanks a lot,
> >>>>>> Nikos
> >>>>>
> >>>>> Hi Guru,
> >>>>>
> >>>>> Any chance you could try this fix that I've staged to send to Linus?
> >>>>> https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-5.4&id=633b1613b2a49304743c18314bb6e6465c21fd8a
> >>>>>
> >>>>> Shiort of that, Nikos: do you happen to have a test scenario that teases
> >>>>> out this deadlock?
> >>>>>
> >>>>
> >>>> Hi Mike,
> >>>>
> >>>> Yes,
> >>>>
> >>>> I created a 50G LV and took a snapshot of the same size:
> >>>>
> >>>> lvcreate -n data-lv -L50G testvg
> >>>> lvcreate -n snap-lv -L50G -s testvg/data-lv
> >>>>
> >>>> Then I ran the following fio job:
> >>>>
> >>>> [global]
> >>>> randrepeat=1
> >>>> ioengine=libaio
> >>>> bs=1M
> >>>> size=6G
> >>>> offset_increment=6G
> >>>> numjobs=8
> >>>> direct=1
> >>>> iodepth=32
> >>>> group_reporting
> >>>> filename=/dev/testvg/data-lv
> >>>>
> >>>> [test]
> >>>> rw=write
> >>>> timeout=180
> >>>>
> >>>> , concurrently with the following script:
> >>>>
> >>>> lvcreate -n dummy-lv -L1G testvg
> >>>>
> >>>> while true
> >>>> do
> >>>> lvcreate -n dummy-snap -L1M -s testvg/dummy-lv
> >>>> lvremove -f testvg/dummy-snap
> >>>> done
> >>>>
> >>>> This reproduced the deadlock for me. I also ran 'echo 30 >
> >>>> /proc/sys/kernel/hung_task_timeout_secs', to reduce the hung task
> >>>> timeout.
> >>>>
> >>>> Nikos.
> >>>
> >>> Very nice, well done. Curious if you've tested with the fix I've staged
> >>> (see above)? If so, does it resolve the deadlock? If you've had
> >>> success I'd be happy to update the tags in the commit header to include
> >>> your Tested-by before sending it to Linus. Also, any review of the
> >>> patch that you can do would be appreciated and with your formal
> >>> Reviewed-by reply would be welcomed and folded in too.
> >>>
> >>> Mike
> >>
> >>
> >>
> >> --
> >> Guruswamy Basavaiah
> >
> >
> >
--
Guruswamy Basavaiah
[ 279.965655] INFO: task drbd_r_r4:7898 blocked for more than 120 seconds.
[ 279.972382] Tainted: P O 4.4.184-octeon-distro.git-v2.96-4-rc-wnd #1
[ 279.980404] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 279.988248] drbd_r_r4 D ffffffff80e1db78 0 7898 2 0x00100000
[ 279.988258] Stack : ffffffff81d00000 ffffffff8114aa38 0000000000000000 ffffffff808dfdb8
7fffffffffffffff 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 80000007ea1a0498 0000000000000001 ffffffffc0280000
80000007efcf0200 ffffffff80e1db78 8000000788ae7830 ffffffff80e208d8
0000000000000001 ffffffff80b35530 8000000788ae7820 8000000788ae7820
8000000788ae7830 8000000788ae7830 ffffffff80fb772d 80000007efcf0200
80000000c9df7380 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 ffffffff80e1d0b4 0000000000000002 80000007efcf0200
ffffffffc0280000 0000000000000001 80000007efcf0330 ffffffffc027eb14
0000000000000000 8000000788ac6c00 ffffffff808c4b60 80000007efcf0338
...
[ 279.988330] Call Trace:
[ 279.988342] [<ffffffff80e1d4a8>] __schedule+0x3c0/0xa58
[ 279.988351] [<ffffffff80e1db78>] schedule+0x38/0x98
[ 279.988359] [<ffffffff80e208d8>] schedule_timeout+0x240/0x2a0
[ 279.988368] [<ffffffff80e1d0b4>] io_schedule_timeout+0x8c/0xc0
[ 279.988389] [<ffffffffc027eb14>] wait_for_in_progress+0x12c/0x168 [dm_snapshot]
[ 279.988406] [<ffffffffc027ec34>] do_origin+0xe4/0x170 [dm_snapshot]
[ 279.988446] [<ffffffffc01dc2c0>] __map_bio+0xb0/0x258 [dm_mod]
[ 279.988479] [<ffffffffc01deb94>] __split_and_process_bio+0x274/0x488 [dm_mod]
[ 279.988511] [<ffffffffc01dee3c>] dm_make_request+0x94/0x128 [dm_mod]
[ 279.988535] [<ffffffff80b3347c>] generic_make_request+0x114/0x290
[ 279.988543] [<ffffffff80b336c0>] submit_bio+0xc8/0x1e0
[ 279.988604] [<ffffffffc051cf60>] drbd_md_sync_page_io+0x360/0x670 [drbd]
[ 279.988671] [<ffffffffc052ae20>] drbd_md_write+0x1c8/0x320 [drbd]
[ 279.988738] [<ffffffffc052b0d4>] drbd_md_sync+0x15c/0x350 [drbd]
[ 279.988803] [<ffffffffc0500bcc>] drbd_start_resync+0x6ec/0x968 [drbd]
[ 279.988866] [<ffffffffc05037dc>] receive_sync_uuid+0x2d4/0x5a0 [drbd]
[ 279.988929] [<ffffffffc0515b30>] drbd_receiver+0x210/0x420 [drbd]
[ 279.988994] [<ffffffffc0523a3c>] drbd_thread_setup+0x74/0x1a8 [drbd]
[ 279.989035] [<ffffffff808b813c>] kthread+0xdc/0xf8
[ 279.989045] [<ffffffff8086bf28>] ret_from_kernel_thread+0x14/0x1c
[ 399.988466] INFO: task drbd_r_r4:7898 blocked for more than 120 seconds.
[ 399.995189] Tainted: P O 4.4.184-octeon-distro.git-v2.96-4-rc-wnd #1
[ 400.003206] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 400.011066] drbd_r_r4 D ffffffff80e1db78 0 7898 2 0x00100000
[ 400.011076] Stack : ffffffff81d00000 ffffffff8114aa38 0000000000000000 ffffffff808dfdb8
7fffffffffffffff 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 80000007ea1a0498 0000000000000001 ffffffffc0280000
80000007efcf0200 ffffffff80e1db78 8000000788ae7830 ffffffff80e208d8
0000000000000001 ffffffff80b35530 8000000788ae7820 8000000788ae7820
8000000788ae7830 8000000788ae7830 ffffffff80fb772d 80000007efcf0200
80000000c9df7380 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 ffffffff80e1d0b4 0000000000000002 80000007efcf0200
ffffffffc0280000 0000000000000001 80000007efcf0330 ffffffffc027eb14
0000000000000000 8000000788ac6c00 ffffffff808c4b60 80000007efcf0338
...
[ 400.011148] Call Trace:
[ 400.011159] [<ffffffff80e1d4a8>] __schedule+0x3c0/0xa58
[ 400.011169] [<ffffffff80e1db78>] schedule+0x38/0x98
[ 400.011177] [<ffffffff80e208d8>] schedule_timeout+0x240/0x2a0
[ 400.011186] [<ffffffff80e1d0b4>] io_schedule_timeout+0x8c/0xc0
[ 400.011205] [<ffffffffc027eb14>] wait_for_in_progress+0x12c/0x168 [dm_snapshot]
[ 400.011222] [<ffffffffc027ec34>] do_origin+0xe4/0x170 [dm_snapshot]
[ 400.011263] [<ffffffffc01dc2c0>] __map_bio+0xb0/0x258 [dm_mod]
[ 400.011295] [<ffffffffc01deb94>] __split_and_process_bio+0x274/0x488 [dm_mod]
[ 400.011328] [<ffffffffc01dee3c>] dm_make_request+0x94/0x128 [dm_mod]
[ 400.011351] [<ffffffff80b3347c>] generic_make_request+0x114/0x290
[ 400.011359] [<ffffffff80b336c0>] submit_bio+0xc8/0x1e0
[ 400.011419] [<ffffffffc051cf60>] drbd_md_sync_page_io+0x360/0x670 [drbd]
[ 400.011486] [<ffffffffc052ae20>] drbd_md_write+0x1c8/0x320 [drbd]
[ 400.011554] [<ffffffffc052b0d4>] drbd_md_sync+0x15c/0x350 [drbd]
[ 400.011618] [<ffffffffc0500bcc>] drbd_start_resync+0x6ec/0x968 [drbd]
[ 400.011681] [<ffffffffc05037dc>] receive_sync_uuid+0x2d4/0x5a0 [drbd]
[ 400.011744] [<ffffffffc0515b30>] drbd_receiver+0x210/0x420 [drbd]
[ 400.011809] [<ffffffffc0523a3c>] drbd_thread_setup+0x74/0x1a8 [drbd]
[ 400.011850] [<ffffffff808b813c>] kthread+0xdc/0xf8
[ 400.011860] [<ffffffff8086bf28>] ret_from_kernel_thread+0x14/0x1c
[ 520.011262] INFO: task drbd_r_r4:7898 blocked for more than 120 seconds.
[ 520.017985] Tainted: P O 4.4.184-octeon-distro.git-v2.96-4-rc-wnd #1
[ 520.026006] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 520.033855] drbd_r_r4 D ffffffff80e1db78 0 7898 2 0x00100000
[ 520.033896] Stack : ffffffff81d00000 ffffffff8114aa38 0000000000000000 ffffffff808dfdb8
7fffffffffffffff 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 80000007ea1a0498 0000000000000001 ffffffffc0280000
80000007efcf0200 ffffffff80e1db78 8000000788ae7830 ffffffff80e208d8
0000000000000001 ffffffff80b35530 8000000788ae7820 8000000788ae7820
8000000788ae7830 8000000788ae7830 ffffffff80fb772d 80000007efcf0200
80000000c9df7380 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 ffffffff80e1d0b4 0000000000000002 80000007efcf0200
ffffffffc0280000 0000000000000001 80000007efcf0330 ffffffffc027eb14
0000000000000000 8000000788ac6c00 ffffffff808c4b60 80000007efcf0338
...
[ 520.033976] Call Trace:
[ 520.033988] [<ffffffff80e1d4a8>] __schedule+0x3c0/0xa58
[ 520.034014] [<ffffffff80e1db78>] schedule+0x38/0x98
[ 520.034022] [<ffffffff80e208d8>] schedule_timeout+0x240/0x2a0
[ 520.034031] [<ffffffff80e1d0b4>] io_schedule_timeout+0x8c/0xc0
[ 520.034050] [<ffffffffc027eb14>] wait_for_in_progress+0x12c/0x168 [dm_snapshot]
[ 520.034067] [<ffffffffc027ec34>] do_origin+0xe4/0x170 [dm_snapshot]
[ 520.034107] [<ffffffffc01dc2c0>] __map_bio+0xb0/0x258 [dm_mod]
[ 520.034140] [<ffffffffc01deb94>] __split_and_process_bio+0x274/0x488 [dm_mod]
[ 520.034192] [<ffffffffc01dee3c>] dm_make_request+0x94/0x128 [dm_mod]
[ 520.034230] [<ffffffff80b3347c>] generic_make_request+0x114/0x290
[ 520.034241] [<ffffffff80b336c0>] submit_bio+0xc8/0x1e0
[ 520.034301] [<ffffffffc051cf60>] drbd_md_sync_page_io+0x360/0x670 [drbd]
[ 520.034368] [<ffffffffc052ae20>] drbd_md_write+0x1c8/0x320 [drbd]
[ 520.034444] [<ffffffffc052b0d4>] drbd_md_sync+0x15c/0x350 [drbd]
[ 520.034524] [<ffffffffc0500bcc>] drbd_start_resync+0x6ec/0x968 [drbd]
[ 520.034589] [<ffffffffc05037dc>] receive_sync_uuid+0x2d4/0x5a0 [drbd]
[ 520.034654] [<ffffffffc0515b30>] drbd_receiver+0x210/0x420 [drbd]
[ 520.034736] [<ffffffffc0523a3c>] drbd_thread_setup+0x74/0x1a8 [drbd]
[ 520.034778] [<ffffffff808b813c>] kthread+0xdc/0xf8
[ 520.034788] [<ffffffff8086bf28>] ret_from_kernel_thread+0x14/0x1c
[ 540.587484] dmxmsg: CAC hand 1002 sequence is 2.
[ 600.590573] dmxmsg: CAC hand 1002 sequence is 3.
[ 640.034067] INFO: task drbd_r_r4:7898 blocked for more than 120 seconds.
[ 640.040796] Tainted: P O 4.4.184-octeon-distro.git-v2.96-4-rc-wnd #1
[ 640.048820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 640.056675] drbd_r_r4 D ffffffff80e1db78 0 7898 2 0x00100000
[ 640.056685] Stack : ffffffff81d00000 ffffffff8114aa38 0000000000000000 ffffffff808dfdb8
7fffffffffffffff 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 80000007ea1a0498 0000000000000001 ffffffffc0280000
80000007efcf0200 ffffffff80e1db78 8000000788ae7830 ffffffff80e208d8
0000000000000001 ffffffff80b35530 8000000788ae7820 8000000788ae7820
8000000788ae7830 8000000788ae7830 ffffffff80fb772d 80000007efcf0200
80000000c9df7380 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 ffffffff80e1d0b4 0000000000000002 80000007efcf0200
ffffffffc0280000 0000000000000001 80000007efcf0330 ffffffffc027eb14
0000000000000000 8000000788ac6c00 ffffffff808c4b60 80000007efcf0338
...
[ 640.056757] Call Trace:
[ 640.056769] [<ffffffff80e1d4a8>] __schedule+0x3c0/0xa58
[ 640.056778] [<ffffffff80e1db78>] schedule+0x38/0x98
[ 640.056786] [<ffffffff80e208d8>] schedule_timeout+0x240/0x2a0
[ 640.056795] [<ffffffff80e1d0b4>] io_schedule_timeout+0x8c/0xc0
[ 640.056814] [<ffffffffc027eb14>] wait_for_in_progress+0x12c/0x168 [dm_snapshot]
[ 640.056831] [<ffffffffc027ec34>] do_origin+0xe4/0x170 [dm_snapshot]
[ 640.056871] [<ffffffffc01dc2c0>] __map_bio+0xb0/0x258 [dm_mod]
[ 640.056903] [<ffffffffc01deb94>] __split_and_process_bio+0x274/0x488 [dm_mod]
[ 640.056935] [<ffffffffc01dee3c>] dm_make_request+0x94/0x128 [dm_mod]
[ 640.056959] [<ffffffff80b3347c>] generic_make_request+0x114/0x290
[ 640.056967] [<ffffffff80b336c0>] submit_bio+0xc8/0x1e0
[ 640.057028] [<ffffffffc051cf60>] drbd_md_sync_page_io+0x360/0x670 [drbd]
[ 640.057094] [<ffffffffc052ae20>] drbd_md_write+0x1c8/0x320 [drbd]
[ 640.057161] [<ffffffffc052b0d4>] drbd_md_sync+0x15c/0x350 [drbd]
[ 640.057226] [<ffffffffc0500bcc>] drbd_start_resync+0x6ec/0x968 [drbd]
[ 640.057289] [<ffffffffc05037dc>] receive_sync_uuid+0x2d4/0x5a0 [drbd]
[ 640.057352] [<ffffffffc0515b30>] drbd_receiver+0x210/0x420 [drbd]
[ 640.057417] [<ffffffffc0523a3c>] drbd_thread_setup+0x74/0x1a8 [drbd]
[ 640.057458] [<ffffffff808b813c>] kthread+0xdc/0xf8
[ 640.057468] [<ffffffff8086bf28>] ret_from_kernel_thread+0x14/0x1c
[ 733.653271] Process {pid:9165, uid:0, comm:HealthDetector} is killing process {pid:12786, comm:getFCSStats.sh}
[ 760.056793] INFO: task drbd_r_r4:7898 blocked for more than 120 seconds.
[ 760.063517] Tainted: P O 4.4.184-octeon-distro.git-v2.96-4-rc-wnd #1
[ 760.071540] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 760.079402] drbd_r_r4 D ffffffff80e1db78 0 7898 2 0x00100000
[ 760.079412] Stack : ffffffff81d00000 ffffffff8114aa38 0000000000000000 ffffffff808dfdb8
7fffffffffffffff 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 80000007ea1a0498 0000000000000001 ffffffffc0280000
80000007efcf0200 ffffffff80e1db78 8000000788ae7830 ffffffff80e208d8
0000000000000001 ffffffff80b35530 8000000788ae7820 8000000788ae7820
8000000788ae7830 8000000788ae7830 ffffffff80fb772d 80000007efcf0200
80000000c9df7380 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 ffffffff80e1d0b4 0000000000000002 80000007efcf0200
ffffffffc0280000 0000000000000001 80000007efcf0330 ffffffffc027eb14
0000000000000000 8000000788ac6c00 ffffffff808c4b60 80000007efcf0338
...
[ 760.079483] Call Trace:
[ 760.079495] [<ffffffff80e1d4a8>] __schedule+0x3c0/0xa58
[ 760.079504] [<ffffffff80e1db78>] schedule+0x38/0x98
[ 760.079512] [<ffffffff80e208d8>] schedule_timeout+0x240/0x2a0
[ 760.079521] [<ffffffff80e1d0b4>] io_schedule_timeout+0x8c/0xc0
[ 760.079541] [<ffffffffc027eb14>] wait_for_in_progress+0x12c/0x168 [dm_snapshot]
[ 760.079557] [<ffffffffc027ec34>] do_origin+0xe4/0x170 [dm_snapshot]
[ 760.079598] [<ffffffffc01dc2c0>] __map_bio+0xb0/0x258 [dm_mod]
[ 760.079631] [<ffffffffc01deb94>] __split_and_process_bio+0x274/0x488 [dm_mod]
[ 760.079663] [<ffffffffc01dee3c>] dm_make_request+0x94/0x128 [dm_mod]
[ 760.079686] [<ffffffff80b3347c>] generic_make_request+0x114/0x290
[ 760.079695] [<ffffffff80b336c0>] submit_bio+0xc8/0x1e0
[ 760.079755] [<ffffffffc051cf60>] drbd_md_sync_page_io+0x360/0x670 [drbd]
[ 760.079822] [<ffffffffc052ae20>] drbd_md_write+0x1c8/0x320 [drbd]
[ 760.079890] [<ffffffffc052b0d4>] drbd_md_sync+0x15c/0x350 [drbd]
[ 760.079954] [<ffffffffc0500bcc>] drbd_start_resync+0x6ec/0x968 [drbd]
[ 760.080017] [<ffffffffc05037dc>] receive_sync_uuid+0x2d4/0x5a0 [drbd]
[ 760.080081] [<ffffffffc0515b30>] drbd_receiver+0x210/0x420 [drbd]
[ 760.080146] [<ffffffffc0523a3c>] drbd_thread_setup+0x74/0x1a8 [drbd]
[ 760.080187] [<ffffffff808b813c>] kthread+0xdc/0xf8
[ 760.080197] [<ffffffff8086bf28>] ret_from_kernel_thread+0x14/0x1c
[ 880.079674] INFO: task drbd_r_r4:7898 blocked for more than 120 seconds.
[ 880.086398] Tainted: P O 4.4.184-octeon-distro.git-v2.96-4-rc-wnd #1
[ 880.094421] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 880.102263] drbd_r_r4 D ffffffff80e1db78 0 7898 2 0x00100000
[ 880.102273] Stack : ffffffff81d00000 ffffffff8114aa38 0000000000000000 ffffffff808dfdb8
7fffffffffffffff 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 80000007ea1a0498 0000000000000001 ffffffffc0280000
80000007efcf0200 ffffffff80e1db78 8000000788ae7830 ffffffff80e208d8
0000000000000001 ffffffff80b35530 8000000788ae7820 8000000788ae7820
8000000788ae7830 8000000788ae7830 ffffffff80fb772d 80000007efcf0200
80000000c9df7380 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 ffffffff80e1d0b4 0000000000000002 80000007efcf0200
ffffffffc0280000 0000000000000001 80000007efcf0330 ffffffffc027eb14
0000000000000000 8000000788ac6c00 ffffffff808c4b60 80000007efcf0338
...
[ 880.102344] Call Trace:
[ 880.102356] [<ffffffff80e1d4a8>] __schedule+0x3c0/0xa58
[ 880.102365] [<ffffffff80e1db78>] schedule+0x38/0x98
[ 880.102373] [<ffffffff80e208d8>] schedule_timeout+0x240/0x2a0
[ 880.102382] [<ffffffff80e1d0b4>] io_schedule_timeout+0x8c/0xc0
[ 880.102402] [<ffffffffc027eb14>] wait_for_in_progress+0x12c/0x168 [dm_snapshot]
[ 880.102419] [<ffffffffc027ec34>] do_origin+0xe4/0x170 [dm_snapshot]
[ 880.102459] [<ffffffffc01dc2c0>] __map_bio+0xb0/0x258 [dm_mod]
[ 880.102491] [<ffffffffc01deb94>] __split_and_process_bio+0x274/0x488 [dm_mod]
[ 880.102523] [<ffffffffc01dee3c>] dm_make_request+0x94/0x128 [dm_mod]
[ 880.102546] [<ffffffff80b3347c>] generic_make_request+0x114/0x290
[ 880.102555] [<ffffffff80b336c0>] submit_bio+0xc8/0x1e0
[ 880.102615] [<ffffffffc051cf60>] drbd_md_sync_page_io+0x360/0x670 [drbd]
[ 880.102682] [<ffffffffc052ae20>] drbd_md_write+0x1c8/0x320 [drbd]
[ 880.102749] [<ffffffffc052b0d4>] drbd_md_sync+0x15c/0x350 [drbd]
[ 880.102814] [<ffffffffc0500bcc>] drbd_start_resync+0x6ec/0x968 [drbd]
[ 880.102877] [<ffffffffc05037dc>] receive_sync_uuid+0x2d4/0x5a0 [drbd]
[ 880.102940] [<ffffffffc0515b30>] drbd_receiver+0x210/0x420 [drbd]
[ 880.103005] [<ffffffffc0523a3c>] drbd_thread_setup+0x74/0x1a8 [drbd]
[ 880.103046] [<ffffffff808b813c>] kthread+0xdc/0xf8
[ 880.103056] [<ffffffff8086bf28>] ret_from_kernel_thread+0x14/0x1c
[ 900.595553] dmxmsg: CAC hand 1002 sequence is 8.
[ 960.598987] dmxmsg: CAC hand 1002 sequence is 9.
[ 1000.102471] INFO: task drbd_r_r4:7898 blocked for more than 120 seconds.
[ 1000.109200] Tainted: P O 4.4.184-octeon-distro.git-v2.96-4-rc-wnd #1
[ 1000.117218] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1000.125058] drbd_r_r4 D ffffffff80e1db78 0 7898 2 0x00100000
[ 1000.125068] Stack : ffffffff81d00000 ffffffff8114aa38 0000000000000000 ffffffff808dfdb8
7fffffffffffffff 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 80000007ea1a0498 0000000000000001 ffffffffc0280000
80000007efcf0200 ffffffff80e1db78 8000000788ae7830 ffffffff80e208d8
0000000000000001 ffffffff80b35530 8000000788ae7820 8000000788ae7820
8000000788ae7830 8000000788ae7830 ffffffff80fb772d 80000007efcf0200
80000000c9df7380 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 ffffffff80e1d0b4 0000000000000002 80000007efcf0200
ffffffffc0280000 0000000000000001 80000007efcf0330 ffffffffc027eb14
0000000000000000 8000000788ac6c00 ffffffff808c4b60 80000007efcf0338
...
[ 1000.125144] Call Trace:
[ 1000.125157] [<ffffffff80e1d4a8>] __schedule+0x3c0/0xa58
[ 1000.125166] [<ffffffff80e1db78>] schedule+0x38/0x98
[ 1000.125174] [<ffffffff80e208d8>] schedule_timeout+0x240/0x2a0
[ 1000.125183] [<ffffffff80e1d0b4>] io_schedule_timeout+0x8c/0xc0
[ 1000.125203] [<ffffffffc027eb14>] wait_for_in_progress+0x12c/0x168 [dm_snapshot]
[ 1000.125219] [<ffffffffc027ec34>] do_origin+0xe4/0x170 [dm_snapshot]
[ 1000.125260] [<ffffffffc01dc2c0>] __map_bio+0xb0/0x258 [dm_mod]
[ 1000.125292] [<ffffffffc01deb94>] __split_and_process_bio+0x274/0x488 [dm_mod]
[ 1000.125324] [<ffffffffc01dee3c>] dm_make_request+0x94/0x128 [dm_mod]
[ 1000.125376] [<ffffffff80b3347c>] generic_make_request+0x114/0x290
[ 1000.125394] [<ffffffff80b336c0>] submit_bio+0xc8/0x1e0
[ 1000.125458] [<ffffffffc051cf60>] drbd_md_sync_page_io+0x360/0x670 [drbd]
[ 1000.125525] [<ffffffffc052ae20>] drbd_md_write+0x1c8/0x320 [drbd]
[ 1000.125592] [<ffffffffc052b0d4>] drbd_md_sync+0x15c/0x350 [drbd]
[ 1000.125657] [<ffffffffc0500bcc>] drbd_start_resync+0x6ec/0x968 [drbd]
[ 1000.125719] [<ffffffffc05037dc>] receive_sync_uuid+0x2d4/0x5a0 [drbd]
[ 1000.125783] [<ffffffffc0515b30>] drbd_receiver+0x210/0x420 [drbd]
[ 1000.125848] [<ffffffffc0523a3c>] drbd_thread_setup+0x74/0x1a8 [drbd]
[ 1000.125888] [<ffffffff808b813c>] kthread+0xdc/0xf8
[ 1000.125898] [<ffffffff8086bf28>] ret_from_kernel_thread+0x14/0x1c
[ 1033.650169] Process {pid:9165, uid:0, comm:HealthDetector} is killing process {pid:16180, comm:getFCSStats.sh}
[ 1120.125262] INFO: task drbd_r_r4:7898 blocked for more than 120 seconds.
[ 1120.131989] Tainted: P O 4.4.184-octeon-distro.git-v2.96-4-rc-wnd #1
[ 1120.140011] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1120.147862] drbd_r_r4 D ffffffff80e1db78 0 7898 2 0x00100000
[ 1120.147872] Stack : ffffffff81d00000 ffffffff8114aa38 0000000000000000 ffffffff808dfdb8
7fffffffffffffff 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 80000007ea1a0498 0000000000000001 ffffffffc0280000
80000007efcf0200 ffffffff80e1db78 8000000788ae7830 ffffffff80e208d8
0000000000000001 ffffffff80b35530 8000000788ae7820 8000000788ae7820
8000000788ae7830 8000000788ae7830 ffffffff80fb772d 80000007efcf0200
80000000c9df7380 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 ffffffff80e1d0b4 0000000000000002 80000007efcf0200
ffffffffc0280000 0000000000000001 80000007efcf0330 ffffffffc027eb14
0000000000000000 8000000788ac6c00 ffffffff808c4b60 80000007efcf0338
...
[ 1120.147944] Call Trace:
[ 1120.147956] [<ffffffff80e1d4a8>] __schedule+0x3c0/0xa58
[ 1120.147965] [<ffffffff80e1db78>] schedule+0x38/0x98
[ 1120.147973] [<ffffffff80e208d8>] schedule_timeout+0x240/0x2a0
[ 1120.147982] [<ffffffff80e1d0b4>] io_schedule_timeout+0x8c/0xc0
[ 1120.148002] [<ffffffffc027eb14>] wait_for_in_progress+0x12c/0x168 [dm_snapshot]
[ 1120.148019] [<ffffffffc027ec34>] do_origin+0xe4/0x170 [dm_snapshot]
[ 1120.148058] [<ffffffffc01dc2c0>] __map_bio+0xb0/0x258 [dm_mod]
[ 1120.148091] [<ffffffffc01deb94>] __split_and_process_bio+0x274/0x488 [dm_mod]
[ 1120.148123] [<ffffffffc01dee3c>] dm_make_request+0x94/0x128 [dm_mod]
[ 1120.148146] [<ffffffff80b3347c>] generic_make_request+0x114/0x290
[ 1120.148154] [<ffffffff80b336c0>] submit_bio+0xc8/0x1e0
[ 1120.148216] [<ffffffffc051cf60>] drbd_md_sync_page_io+0x360/0x670 [drbd]
[ 1120.148282] [<ffffffffc052ae20>] drbd_md_write+0x1c8/0x320 [drbd]
[ 1120.148350] [<ffffffffc052b0d4>] drbd_md_sync+0x15c/0x350 [drbd]
[ 1120.148415] [<ffffffffc0500bcc>] drbd_start_resync+0x6ec/0x968 [drbd]
[ 1120.148477] [<ffffffffc05037dc>] receive_sync_uuid+0x2d4/0x5a0 [drbd]
[ 1120.148541] [<ffffffffc0515b30>] drbd_receiver+0x210/0x420 [drbd]
[ 1120.148606] [<ffffffffc0523a3c>] drbd_thread_setup+0x74/0x1a8 [drbd]
[ 1120.148647] [<ffffffff808b813c>] kthread+0xdc/0xf8
[ 1120.148657] [<ffffffff8086bf28>] ret_from_kernel_thread+0x14/0x1c
[ 1240.148088] INFO: task drbd_r_r4:7898 blocked for more than 120 seconds.
[ 1240.154815] Tainted: P O 4.4.184-octeon-distro.git-v2.96-4-rc-wnd #1
[ 1240.162841] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1240.170689] drbd_r_r4 D ffffffff80e1db78 0 7898 2 0x00100000
[ 1240.170712] Stack : ffffffff81d00000 ffffffff8114aa38 0000000000000000 ffffffff808dfdb8
7fffffffffffffff 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 80000007ea1a0498 0000000000000001 ffffffffc0280000
80000007efcf0200 ffffffff80e1db78 8000000788ae7830 ffffffff80e208d8
0000000000000001 ffffffff80b35530 8000000788ae7820 8000000788ae7820
8000000788ae7830 8000000788ae7830 ffffffff80fb772d 80000007efcf0200
80000000c9df7380 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 ffffffff80e1d0b4 0000000000000002 80000007efcf0200
ffffffffc0280000 0000000000000001 80000007efcf0330 ffffffffc027eb14
0000000000000000 8000000788ac6c00 ffffffff808c4b60 80000007efcf0338
...
[ 1240.170788] Call Trace:
[ 1240.170800] [<ffffffff80e1d4a8>] __schedule+0x3c0/0xa58
[ 1240.170809] [<ffffffff80e1db78>] schedule+0x38/0x98
[ 1240.170817] [<ffffffff80e208d8>] schedule_timeout+0x240/0x2a0
[ 1240.170826] [<ffffffff80e1d0b4>] io_schedule_timeout+0x8c/0xc0
[ 1240.170847] [<ffffffffc027eb14>] wait_for_in_progress+0x12c/0x168 [dm_snapshot]
[ 1240.170863] [<ffffffffc027ec34>] do_origin+0xe4/0x170 [dm_snapshot]
[ 1240.170904] [<ffffffffc01dc2c0>] __map_bio+0xb0/0x258 [dm_mod]
[ 1240.170936] [<ffffffffc01deb94>] __split_and_process_bio+0x274/0x488 [dm_mod]
[ 1240.170969] [<ffffffffc01dee3c>] dm_make_request+0x94/0x128 [dm_mod]
[ 1240.171009] [<ffffffff80b3347c>] generic_make_request+0x114/0x290
[ 1240.171027] [<ffffffff80b336c0>] submit_bio+0xc8/0x1e0
[ 1240.171096] [<ffffffffc051cf60>] drbd_md_sync_page_io+0x360/0x670 [drbd]
[ 1240.171170] [<ffffffffc052ae20>] drbd_md_write+0x1c8/0x320 [drbd]
[ 1240.171247] [<ffffffffc052b0d4>] drbd_md_sync+0x15c/0x350 [drbd]
[ 1240.171332] [<ffffffffc0500bcc>] drbd_start_resync+0x6ec/0x968 [drbd]
[ 1240.171396] [<ffffffffc05037dc>] receive_sync_uuid+0x2d4/0x5a0 [drbd]
[ 1240.171463] [<ffffffffc0515b30>] drbd_receiver+0x210/0x420 [drbd]
[ 1240.171541] [<ffffffffc0523a3c>] drbd_thread_setup+0x74/0x1a8 [drbd]
[ 1240.171582] [<ffffffff808b813c>] kthread+0xdc/0xf8
[ 1240.171592] [<ffffffff8086bf28>] ret_from_kernel_thread+0x14/0x1c
[ 1333.647369] Process {pid:9165, uid:0, comm:HealthDetector} is killing process {pid:17698, comm:getFCSStats.sh}
[ 1360.170893] INFO: task drbd_r_r4:7898 blocked for more than 120 seconds.
[ 1360.177617] Tainted: P O 4.4.184-octeon-distro.git-v2.96-4-rc-wnd #1
[ 1360.185639] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1360.193504] drbd_r_r4 D ffffffff80e1db78 0 7898 2 0x00100000
[ 1360.193514] Stack : ffffffff81d00000 ffffffff8114aa38 0000000000000000 ffffffff808dfdb8
7fffffffffffffff 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 80000007ea1a0498 0000000000000001 ffffffffc0280000
80000007efcf0200 ffffffff80e1db78 8000000788ae7830 ffffffff80e208d8
0000000000000001 ffffffff80b35530 8000000788ae7820 8000000788ae7820
8000000788ae7830 8000000788ae7830 ffffffff80fb772d 80000007efcf0200
80000000c9df7380 0000000000000000 80000000c9df7c38 7fffffffffffffff
ffffffffc0280000 ffffffff80e1d0b4 0000000000000002 80000007efcf0200
ffffffffc0280000 0000000000000001 80000007efcf0330 ffffffffc027eb14
0000000000000000 8000000788ac6c00 ffffffff808c4b60 80000007efcf0338
...
[ 1360.193586] Call Trace:
[ 1360.193598] [<ffffffff80e1d4a8>] __schedule+0x3c0/0xa58
[ 1360.193607] [<ffffffff80e1db78>] schedule+0x38/0x98
[ 1360.193615] [<ffffffff80e208d8>] schedule_timeout+0x240/0x2a0
[ 1360.193624] [<ffffffff80e1d0b4>] io_schedule_timeout+0x8c/0xc0
[ 1360.193644] [<ffffffffc027eb14>] wait_for_in_progress+0x12c/0x168 [dm_snapshot]
[ 1360.193660] [<ffffffffc027ec34>] do_origin+0xe4/0x170 [dm_snapshot]
[ 1360.193701] [<ffffffffc01dc2c0>] __map_bio+0xb0/0x258 [dm_mod]
[ 1360.193733] [<ffffffffc01deb94>] __split_and_process_bio+0x274/0x488 [dm_mod]
[ 1360.193765] [<ffffffffc01dee3c>] dm_make_request+0x94/0x128 [dm_mod]
[ 1360.193788] [<ffffffff80b3347c>] generic_make_request+0x114/0x290
[ 1360.193797] [<ffffffff80b336c0>] submit_bio+0xc8/0x1e0
[ 1360.193857] [<ffffffffc051cf60>] drbd_md_sync_page_io+0x360/0x670 [drbd]
[ 1360.193924] [<ffffffffc052ae20>] drbd_md_write+0x1c8/0x320 [drbd]
[ 1360.193991] [<ffffffffc052b0d4>] drbd_md_sync+0x15c/0x350 [drbd]
[ 1360.194056] [<ffffffffc0500bcc>] drbd_start_resync+0x6ec/0x968 [drbd]
[ 1360.194118] [<ffffffffc05037dc>] receive_sync_uuid+0x2d4/0x5a0 [drbd]
[ 1360.194182] [<ffffffffc0515b30>] drbd_receiver+0x210/0x420 [drbd]
[ 1360.194247] [<ffffffffc0523a3c>] drbd_thread_setup+0x74/0x1a8 [drbd]
[ 1360.194287] [<ffffffff808b813c>] kthread+0xdc/0xf8
[ 1360.194297] [<ffffffff8086bf28>] ret_from_kernel_thread+0x14/0x1c
--
dm-devel mailing list
dm-devel@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/dm-devel
