Re: [bug report] INFO: task mdX_resync:42168 blocked for more than 122 seconds

[Yu Kuai <yukuai1@xxxxxxxxxxxxxxx>]

Hi,

在 2024/05/21 12:28, Changhui Zhong 写道:
> On Tue, May 21, 2024 at 9:09 AM Yu Kuai <yukuai1@xxxxxxxxxxxxxxx> wrote:
>
> > Thanks for the test! Since raid10 has the same problem as well, then the
> > problem seems to be more common in raid. And related code to raid10 is
> > more simpler, attached is a patch to add debuginfo to raid10.
> >
> > BTW, Xiao can reporduce the problem as well, and will lend a hand as
> > well.
> >
> > Thanks,
> > Kuai
>
>
> Hi, Yu Kuai and Xiao
>
> thanks for  efforts on this issue，
> after applying the debug patch, I got the following results, please check it
>
> [  597.491083] Create raid10
> [  597.647987] device-mapper: raid: Superblocks created for new raid set
> [  597.680540] md/raid10:mdX: not clean -- starting background reconstruction
> [  597.680549] md/raid10:mdX: active with 4 out of 4 devices
> [  597.683779] mdX: bitmap file is out of date, doing full recovery
> [  597.692971] md: resync of RAID array mdX
> [  597.753583] try_raise_barrier: resync depth exceed limit
> [  597.759540] try_raise_barrier: resync depth exceed limit
> [  597.765504] try_raise_barrier: resync depth exceed limit
> [  597.771432] try_raise_barrier: resync depth exceed limit
> [  597.779400] barrier_waiting: nr_waiting 1
> [  597.783879] barrier_waiting: nr_waiting 1
> [  597.788356] barrier_waiting: nr_waiting 1
> [  597.792838] barrier_waiting: nr_waiting 1
> [  597.797327] barrier_waiting: nr_waiting 1
> [  597.801806] barrier_waiting: nr_waiting 1
> [  597.806288] barrier_waiting: nr_waiting 1
> [  597.810780] barrier_waiting: nr_waiting 1
> [  597.815252] barrier_waiting: nr_waiting 1
> [  597.819737] barrier_waiting: nr_waiting 1
> [  597.824232] barrier_waiting: nr_waiting 1
> [  597.828714] barrier_waiting: nr_waiting 1
> [  597.833196] barrier_waiting: nr_waiting 1
> [  597.837678] barrier_waiting: nr_waiting 1
> [  597.842160] barrier_waiting: nr_waiting 1
> [  597.846640] barrier_waiting: nr_waiting 1
> [  597.851128] barrier_waiting: nr_waiting 1
> [  597.855610] barrier_waiting: nr_waiting 1
> [  597.860094] barrier_waiting: nr_waiting 1
> [  597.864576] barrier_waiting: nr_waiting 1
> [  597.869060] barrier_waiting: nr_waiting 1
> [  597.873548] barrier_waiting: nr_waiting 1
> [  597.878032] barrier_waiting: nr_waiting 1
> [  597.882522] barrier_waiting: nr_waiting 1
> [  597.883457] barrier_waiting: nr_waiting 1
> [  597.891503] barrier_waiting: nr_waiting 1
> [  597.895987] barrier_waiting: nr_waiting 1
> [  597.900459] barrier_waiting: nr_waiting 1
> [  597.904940] barrier_waiting: nr_waiting 1
> [  597.909421] barrier_waiting: nr_waiting 1
> [  597.913900] barrier_waiting: nr_waiting 1
> [  597.918377] barrier_waiting: nr_waiting 1
> [  597.922857] barrier_waiting: nr_waiting 1
> [  597.927350] barrier_waiting: nr_waiting 1
> [  597.931841] barrier_waiting: nr_waiting 1
> [  597.936322] barrier_waiting: nr_waiting 1
> [  597.940805] barrier_waiting: nr_waiting 1
> [  597.945284] barrier_waiting: nr_waiting 1
> [  597.949766] barrier_waiting: nr_waiting 1
> [  597.954246] barrier_waiting: nr_waiting 1
> [  597.958726] barrier_waiting: nr_waiting 1
> [  597.963205] barrier_waiting: nr_waiting 1
> [  597.967683] barrier_waiting: nr_waiting 1
> [  597.972163] barrier_waiting: nr_waiting 1
> [  597.976641] barrier_waiting: nr_waiting 1
> [  597.981119] barrier_waiting: nr_waiting 1
> [  597.985601] barrier_waiting: nr_waiting 1
> [  597.990082] barrier_waiting: nr_waiting 1
> [  597.994562] barrier_waiting: nr_waiting 1
> [  597.999043] barrier_waiting: nr_waiting 1
> [  598.003522] barrier_waiting: nr_waiting 1
> [  598.008000] barrier_waiting: nr_waiting 1
> [  598.012485] barrier_waiting: nr_waiting 1
> [  598.016964] barrier_waiting: nr_waiting 1
> [  598.021451] barrier_waiting: nr_waiting 1
> [  598.025931] barrier_waiting: nr_waiting 1
> [  598.030409] barrier_waiting: nr_waiting 1
> [  598.034888] barrier_waiting: nr_waiting 1
> [  598.039368] barrier_waiting: nr_waiting 1
> [  598.043848] barrier_waiting: nr_waiting 1
> [  598.048325] barrier_waiting: nr_waiting 1
> [  598.052805] barrier_waiting: nr_waiting 1
> [  598.057283] barrier_waiting: nr_waiting 1
> [  598.061767] barrier_waiting: nr_waiting 1
> [  598.066246] barrier_waiting: nr_waiting 1
> [  598.070726] barrier_waiting: nr_waiting 1
> [  598.075206] barrier_waiting: nr_waiting 1
> [  598.079689] barrier_waiting: nr_waiting 1
> [  598.084170] barrier_waiting: nr_waiting 1
> [  598.088648] barrier_waiting: nr_waiting 1
> [  598.093130] barrier_waiting: nr_waiting 1
> [  598.097646] barrier_waiting: nr_waiting 1
> [  598.102126] barrier_waiting: nr_waiting 1
> [  598.106605] barrier_waiting: nr_waiting 1
> [  598.111085] barrier_waiting: nr_waiting 1
> [  598.115566] barrier_waiting: nr_waiting 1
> [  598.120043] barrier_waiting: nr_waiting 1
> [  598.124522] barrier_waiting: nr_waiting 1
> [  598.129001] barrier_waiting: nr_waiting 1
> [  598.133481] barrier_waiting: nr_waiting 1
> [  598.137958] barrier_waiting: nr_waiting 1
> [  598.142442] barrier_waiting: nr_waiting 1
> [  598.146921] barrier_waiting: nr_waiting 1
> [  598.151400] barrier_waiting: nr_waiting 1
> [  598.155878] barrier_waiting: nr_waiting 1
> [  598.160359] barrier_waiting: nr_waiting 1
> [  598.164838] barrier_waiting: nr_waiting 1
> [  598.169319] barrier_waiting: nr_waiting 1
> [  598.173803] barrier_waiting: nr_waiting 1
> [  598.178282] barrier_waiting: nr_waiting 1
> [  598.182762] barrier_waiting: nr_waiting 1
> [  598.187241] barrier_waiting: nr_waiting 1
> [  598.191722] barrier_waiting: nr_waiting 1
> [  598.196201] barrier_waiting: nr_waiting 1
> [  598.200678] barrier_waiting: nr_waiting 1
> [  598.205159] barrier_waiting: nr_waiting 1
> [  598.209640] barrier_waiting: nr_waiting 1
> [  598.214120] barrier_waiting: nr_waiting 1
> [  598.218599] barrier_waiting: nr_waiting 1
> [  598.223078] barrier_waiting: nr_waiting 1
> [  598.227557] barrier_waiting: nr_waiting 1
> [  598.232036] barrier_waiting: nr_waiting 1
> [  598.236519] barrier_waiting: nr_waiting 1
> [  598.240999] barrier_waiting: nr_waiting 1
> [  598.245478] barrier_waiting: nr_waiting 1
> [  598.249959] barrier_waiting: nr_waiting 1
> [  598.254439] barrier_waiting: nr_waiting 1
> [  598.258917] barrier_waiting: nr_waiting 1
> [  598.263397] barrier_waiting: nr_waiting 1
> [  598.267880] barrier_waiting: nr_waiting 1
> [  598.272362] barrier_waiting: nr_waiting 1
> [  598.276842] barrier_waiting: nr_waiting 1
> [  598.281322] barrier_waiting: nr_waiting 1
> [  598.285802] barrier_waiting: nr_waiting 1
> [  598.290281] barrier_waiting: nr_waiting 1
> [  598.294758] barrier_waiting: nr_waiting 1
> [  598.299256] barrier_waiting: nr_waiting 1
> [  598.303737] barrier_waiting: nr_waiting 1
> [  598.308219] barrier_waiting: nr_waiting 1
> [  598.312698] barrier_waiting: nr_waiting 1
> [  598.317176] barrier_waiting: nr_waiting 1
> [  598.321659] barrier_waiting: nr_waiting 1
> [  598.326138] barrier_waiting: nr_waiting 1
> [  598.330617] barrier_waiting: nr_waiting 1
> [  598.335100] barrier_waiting: nr_waiting 1
> [  598.339580] barrier_waiting: nr_waiting 1
> [  598.344057] barrier_waiting: nr_waiting 1
> [  598.348536] barrier_waiting: nr_waiting 1
> [  598.353016] barrier_waiting: nr_waiting 1
> [  598.357496] barrier_waiting: nr_waiting 1
> [  598.361973] barrier_waiting: nr_waiting 1
> [  598.366453] barrier_waiting: nr_waiting 1
> [  598.370932] barrier_waiting: nr_waiting 1
> [  598.375412] barrier_waiting: nr_waiting 1
> [  598.379890] barrier_waiting: nr_waiting 1
> [  598.384371] barrier_waiting: nr_waiting 1
> [  598.388856] barrier_waiting: nr_waiting 1
> [  598.393326] barrier_waiting: nr_waiting 1
> [  598.397805] barrier_waiting: nr_waiting 1
> [  598.402288] barrier_waiting: nr_waiting 1
> [  598.406772] barrier_waiting: nr_waiting 1
> [  598.411251] barrier_waiting: nr_waiting 1
> [  598.415729] barrier_waiting: nr_waiting 1
> [  598.420209] barrier_waiting: nr_waiting 1
> [  598.424686] barrier_waiting: nr_waiting 1
> [  598.429165] barrier_waiting: nr_waiting 1
> [  598.433644] barrier_waiting: nr_waiting 1
> [  598.438125] barrier_waiting: nr_waiting 1
> [  598.442604] barrier_waiting: nr_waiting 1
> [  598.447083] barrier_waiting: nr_waiting 1
> [  598.451571] barrier_waiting: nr_waiting 1
> [  598.456050] barrier_waiting: nr_waiting 1
> [  598.460531] barrier_waiting: nr_waiting 1
> [  598.465010] barrier_waiting: nr_waiting 1
> [  598.469498] barrier_waiting: nr_waiting 1
> [  598.473981] barrier_waiting: nr_waiting 1
> [  598.478460] barrier_waiting: nr_waiting 1
> [  598.482940] barrier_waiting: nr_waiting 1
> [  598.487421] barrier_waiting: nr_waiting 1
> [  598.491901] barrier_waiting: nr_waiting 1
> [  598.496382] barrier_waiting: nr_waiting 1
> [  598.500860] barrier_waiting: nr_waiting 1
> [  598.505338] barrier_waiting: nr_waiting 1
> [  598.509837] barrier_waiting: nr_waiting 1
> [  598.514317] barrier_waiting: nr_waiting 1
> [  598.518797] barrier_waiting: nr_waiting 1
> [  598.523276] barrier_waiting: nr_waiting 1
> [  598.527756] barrier_waiting: nr_waiting 1
> [  598.532241] barrier_waiting: nr_waiting 1
> [  598.536719] barrier_waiting: nr_waiting 1
> [  598.541199] barrier_waiting: nr_waiting 1
> [  598.545679] barrier_waiting: nr_waiting 1
> [  598.550159] barrier_waiting: nr_waiting 1
> [  598.554639] barrier_waiting: nr_waiting 1
> [  598.559118] barrier_waiting: nr_waiting 1
> [  598.563597] barrier_waiting: nr_waiting 1
> [  598.568077] barrier_waiting: nr_waiting 1
> [  598.572561] barrier_waiting: nr_waiting 1
> [  598.577038] barrier_waiting: nr_waiting 1
> [  598.581517] barrier_waiting: nr_waiting 1
> [  598.585995] barrier_waiting: nr_waiting 1
> [  598.590475] barrier_waiting: nr_waiting 1
> [  598.594953] barrier_waiting: nr_waiting 1
> [  598.599434] barrier_waiting: nr_waiting 1
> [  598.603913] barrier_waiting: nr_waiting 1
> [  598.608392] barrier_waiting: nr_waiting 1
> [  598.612871] barrier_waiting: nr_waiting 1
> [  598.617349] barrier_waiting: nr_waiting 1
> [  598.621828] barrier_waiting: nr_waiting 1
> [  598.626308] barrier_waiting: nr_waiting 1
> [  598.630787] barrier_waiting: nr_waiting 1
> [  598.635272] barrier_waiting: nr_waiting 1
> [  598.639753] barrier_waiting: nr_waiting 1
> [  598.644231] barrier_waiting: nr_waiting 1
> [  598.648712] barrier_waiting: nr_waiting 1
> [  598.653192] barrier_waiting: nr_waiting 1
> [  598.657671] barrier_waiting: nr_waiting 1
> [  598.662150] barrier_waiting: nr_waiting 1
> [  598.666630] barrier_waiting: nr_waiting 1
> [  598.671111] barrier_waiting: nr_waiting 1
> [  598.675589] barrier_waiting: nr_waiting 1
> [  598.680068] barrier_waiting: nr_waiting 1
> [  598.684547] barrier_waiting: nr_waiting 1
> [  598.689024] barrier_waiting: nr_waiting 1
> [  598.693501] barrier_waiting: nr_waiting 1
> [  598.697982] barrier_waiting: nr_waiting 1
> [  598.702462] barrier_waiting: nr_waiting 1
> [  598.706945] barrier_waiting: nr_waiting 1
> [  598.711425] barrier_waiting: nr_waiting 1
> [  598.715989] barrier_waiting: nr_waiting 1
> [  598.720523] barrier_waiting: nr_waiting 1
> [  598.725014] barrier_waiting: nr_waiting 1
> [  598.729500] barrier_waiting: nr_waiting 1
> [  598.733983] barrier_waiting: nr_waiting 1
> [  598.738465] barrier_waiting: nr_waiting 1
> [  598.742947] barrier_waiting: nr_waiting 1
> [  598.747426] barrier_waiting: nr_waiting 1
> [  598.751909] barrier_waiting: nr_waiting 1
> [  598.756390] barrier_waiting: nr_waiting 1
> [  598.760869] barrier_waiting: nr_waiting 1
> [  598.765360] barrier_waiting: nr_waiting 1
> [  598.769839] barrier_waiting: nr_waiting 1
> [  598.774318] barrier_waiting: nr_waiting 1
> [  598.778801] barrier_waiting: nr_waiting 1
> [  598.783280] barrier_waiting: nr_waiting 1
> [  598.787758] barrier_waiting: nr_waiting 1
> [  598.792242] barrier_waiting: nr_waiting 1
> [  598.796723] barrier_waiting: nr_waiting 1
> [  598.801200] barrier_waiting: nr_waiting 1
> [  598.805688] barrier_waiting: nr_waiting 1
> [  598.810166] barrier_waiting: nr_waiting 1
> [  598.814646] barrier_waiting: nr_waiting 1
> [  598.819126] barrier_waiting: nr_waiting 1
> [  598.823606] barrier_waiting: nr_waiting 1
> [  598.828086] barrier_waiting: nr_waiting 1
> [  598.832574] barrier_waiting: nr_waiting 1
> [  598.837052] barrier_waiting: nr_waiting 1
> [  598.841531] barrier_waiting: nr_waiting 1
> [  598.846012] barrier_waiting: nr_waiting 1
> [  598.850491] barrier_waiting: nr_waiting 1
> [  598.854969] barrier_waiting: nr_waiting 1
> [  598.859456] barrier_waiting: nr_waiting 1
> [  598.863934] barrier_waiting: nr_waiting 1
> [  598.868412] barrier_waiting: nr_waiting 1
> [  598.872895] barrier_waiting: nr_waiting 1
> [  598.877374] barrier_waiting: nr_waiting 1
> [  598.881853] barrier_waiting: nr_waiting 1
> [  598.886334] barrier_waiting: nr_waiting 1
> [  598.890813] barrier_waiting: nr_waiting 1
> [  598.895293] barrier_waiting: nr_waiting 1
> [  598.899774] barrier_waiting: nr_waiting 1
> [  598.904253] barrier_waiting: nr_waiting 1
> [  598.908735] barrier_waiting: nr_waiting 1
> [  598.913215] barrier_waiting: nr_waiting 1
> [  598.917694] barrier_waiting: nr_waiting 1
> [  598.922173] barrier_waiting: nr_waiting 1
> [  598.926663] barrier_waiting: nr_waiting 1
> [  598.931153] barrier_waiting: nr_waiting 1
> [  598.935638] barrier_waiting: nr_waiting 1
> [  598.940115] barrier_waiting: nr_waiting 1
> [  598.944599] barrier_waiting: nr_waiting 1
> [  598.949080] barrier_waiting: nr_waiting 1
> [  598.953561] barrier_waiting: nr_waiting 1
> [  598.958042] barrier_waiting: nr_waiting 1
> [  598.962523] barrier_waiting: nr_waiting 1
> [  598.967006] barrier_waiting: nr_waiting 1
> [  598.971486] barrier_waiting: nr_waiting 1
> [  598.975965] barrier_waiting: nr_waiting 1
> [  598.980446] barrier_waiting: nr_waiting 1
> [  598.984923] barrier_waiting: nr_waiting 1
> [  598.989408] barrier_waiting: nr_waiting 1
> [  598.993890] barrier_waiting: nr_waiting 1
> [  598.998367] barrier_waiting: nr_waiting 1
> [  599.002844] barrier_waiting: nr_waiting 1
> [  599.007328] barrier_waiting: nr_waiting 1
> [  599.011800] barrier_waiting: nr_waiting 1
> [  599.016278] barrier_waiting: nr_waiting 1
> [  599.020756] barrier_waiting: nr_waiting 1
> [  599.025241] barrier_waiting: nr_waiting 1
> [  599.029720] barrier_waiting: nr_waiting 1
> [  599.034199] barrier_waiting: nr_waiting 1
> [  599.038682] barrier_waiting: nr_waiting 1
> [  599.043161] barrier_waiting: nr_waiting 1
> [  599.047641] barrier_waiting: nr_waiting 1
> [  599.052120] barrier_waiting: nr_waiting 1
> [  599.056601] barrier_waiting: nr_waiting 1
> [  599.061082] barrier_waiting: nr_waiting 1
> [  599.065567] barrier_waiting: nr_waiting 1
> [  599.070045] barrier_waiting: nr_waiting 1
> [  599.074522] barrier_waiting: nr_waiting 1
> [  599.079006] barrier_waiting: nr_waiting 1
> [  599.083486] barrier_waiting: nr_waiting 1
> [  599.087968] barrier_waiting: nr_waiting 1
> [  599.092448] barrier_waiting: nr_waiting 1
> [  599.096929] barrier_waiting: nr_waiting 1
> [  599.101407] barrier_waiting: nr_waiting 1
> [  599.105886] barrier_waiting: nr_waiting 1
> [  599.110366] barrier_waiting: nr_waiting 1
> [  599.114850] barrier_waiting: nr_waiting 1
> [  599.119332] barrier_waiting: nr_waiting 1
> [  599.123809] barrier_waiting: nr_waiting 1
> [  599.128293] barrier_waiting: nr_waiting 1
> [  599.132765] barrier_waiting: nr_waiting 1
> [  599.137242] barrier_waiting: nr_waiting 1
> [  599.141723] barrier_waiting: nr_waiting 1
> [  599.146203] barrier_waiting: nr_waiting 1
> [  599.150682] barrier_waiting: nr_waiting 1
> [  599.155165] barrier_waiting: nr_waiting 1
> [  599.159644] barrier_waiting: nr_waiting 1
> [  599.164124] barrier_waiting: nr_waiting 1
> [  599.168603] barrier_waiting: nr_waiting 1
> [  599.173080] barrier_waiting: nr_waiting 1
> [  599.177564] barrier_waiting: nr_waiting 1
> [  599.182043] barrier_waiting: nr_waiting 1
> [  599.186525] barrier_waiting: nr_waiting 1
> [  599.191008] barrier_waiting: nr_waiting 1
> [  599.195488] barrier_waiting: nr_waiting 1
> [  599.199967] barrier_waiting: nr_waiting 1
> [  599.204448] barrier_waiting: nr_waiting 1
> [  599.208928] barrier_waiting: nr_waiting 1
> [  599.213408] barrier_waiting: nr_waiting 1
> [  599.217890] barrier_waiting: nr_waiting 1
> [  599.222370] barrier_waiting: nr_waiting 1
> [  599.226850] barrier_waiting: nr_waiting 1
> [  599.231329] barrier_waiting: nr_waiting 1
> [  599.235811] barrier_waiting: nr_waiting 1
> [  599.240289] barrier_waiting: nr_waiting 1
> [  599.244767] barrier_waiting: nr_waiting 1
> [  599.249247] barrier_waiting: nr_waiting 1
> [  599.253731] barrier_waiting: nr_waiting 1
> [  599.258212] barrier_waiting: nr_waiting 1
> [  599.262691] barrier_waiting: nr_waiting 1
> [  599.267170] barrier_waiting: nr_waiting 1
> [  599.271650] barrier_waiting: nr_waiting 1
> [  599.276128] barrier_waiting: nr_waiting 1
> [  599.280606] barrier_waiting: nr_waiting 1
> [  599.285084] barrier_waiting: nr_waiting 1
> [  599.289569] barrier_waiting: nr_waiting 1
> [  599.294040] barrier_waiting: nr_waiting 1
> [  599.298522] barrier_waiting: nr_waiting 1
> [  599.303006] barrier_waiting: nr_waiting 1
> [  599.307483] barrier_waiting: nr_waiting 1
> [  599.311961] barrier_waiting: nr_waiting 1
> [  599.316442] barrier_waiting: nr_waiting 1
> [  599.320923] barrier_waiting: nr_waiting 1
> [  599.325403] barrier_waiting: nr_waiting 1
> [  599.329884] barrier_waiting: nr_waiting 1
> [  599.334363] barrier_waiting: nr_waiting 1
> [  599.338842] barrier_waiting: nr_waiting 1
> [  599.343322] barrier_waiting: nr_waiting 1
> [  599.347801] barrier_waiting: nr_waiting 1
> [  599.352282] barrier_waiting: nr_waiting 1
> [  599.356762] barrier_waiting: nr_waiting 1
> [  599.361246] barrier_waiting: nr_waiting 1
> [  599.365727] barrier_waiting: nr_waiting 1
> [  599.370206] barrier_waiting: nr_waiting 1
> [  599.374685] barrier_waiting: nr_waiting 1
> [  599.379166] barrier_waiting: nr_waiting 1
> [  599.383647] barrier_waiting: nr_waiting 1
> [  599.388125] barrier_waiting: nr_waiting 1
> [  599.392605] barrier_waiting: nr_waiting 1
> [  599.397083] barrier_waiting: nr_waiting 1
> [  599.401562] barrier_waiting: nr_waiting 1
> [  599.406044] barrier_waiting: nr_waiting 1
> [  599.410525] barrier_waiting: nr_waiting 1
> [  599.415008] barrier_waiting: nr_waiting 1
> [  599.419486] barrier_waiting: nr_waiting 1
> [  599.423968] barrier_waiting: nr_waiting 1
> [  599.428446] barrier_waiting: nr_waiting 1
> [  599.432924] barrier_waiting: nr_waiting 1
> [  599.437409] barrier_waiting: nr_waiting 1
> [  599.441890] barrier_waiting: nr_waiting 1
> [  599.446373] barrier_waiting: nr_waiting 1
> [  599.450859] barrier_waiting: nr_waiting 1
> [  599.455337] barrier_waiting: nr_waiting 1
> [  599.459819] barrier_waiting: nr_waiting 1
> [  599.464306] barrier_waiting: nr_waiting 1
> [  599.468783] barrier_waiting: nr_waiting 1
> [  599.473262] barrier_waiting: nr_waiting 1
> [  599.477744] barrier_waiting: nr_waiting 1
> [  599.482224] barrier_waiting: nr_waiting 1
> [  599.486704] barrier_waiting: nr_waiting 1
> [  599.491190] barrier_waiting: nr_waiting 1
> [  599.495670] barrier_waiting: nr_waiting 1
> [  599.500150] barrier_waiting: nr_waiting 1
> [  599.504629] barrier_waiting: nr_waiting 1
> [  599.509106] barrier_waiting: nr_waiting 1
> [  599.513588] barrier_waiting: nr_waiting 1
> [  599.518072] barrier_waiting: nr_waiting 1
> [  599.522552] barrier_waiting: nr_waiting 1
> [  599.527030] barrier_waiting: nr_waiting 1
> [  599.531509] barrier_waiting: nr_waiting 1
> [  599.535988] barrier_waiting: nr_waiting 1
> [  599.540470] barrier_waiting: nr_waiting 1
> [  599.544955] barrier_waiting: nr_waiting 1
> [  599.549426] barrier_waiting: nr_waiting 1
> [  599.553904] barrier_waiting: nr_waiting 1
> [  599.558384] barrier_waiting: nr_waiting 1
> [  599.562863] barrier_waiting: nr_waiting 1
> [  599.567342] barrier_waiting: nr_waiting 1
> [  599.571820] barrier_waiting: nr_waiting 1
> [  599.576298] barrier_waiting: nr_waiting 1
> [  599.580779] barrier_waiting: nr_waiting 1
> [  599.585254] wait_barrier: nr_pending: 1
> [  599.589540] try_raise_barrier: nr_pending 1
> [  599.589547] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  599.594209] try_raise_barrier: nr_pending 1
> [  599.605582] try_raise_barrier: nr_pending 1
> [  599.605674] allow_barrier: nr_pending: 0
> [  599.610620] wait_barrier: nr_pending: 1
> [  599.614653] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  599.618935] try_raise_barrier: nr_pending 1
> [  599.618938] try_raise_barrier: nr_pending 1
> [  599.618965] raid10_read_request: r10_bio ff3b4134184a6100 start
> [  599.641092] try_raise_barrier: nr_pending 1
> [  599.641145] allow_barrier: nr_pending: 0
> [  599.646251] wait_barrier: nr_pending: 1
> [  599.650157] raid_end_bio_io: r10_bio ff3b4134184a6100 done
> [  599.654437] try_raise_barrier: nr_pending 1
> [  599.654446] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  599.654575] allow_barrier: nr_pending: 0
> [  599.654583] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  599.682715] wait_barrier: nr_pending: 1
> [  599.687000] try_raise_barrier: nr_pending 1
> [  599.687012] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  599.691672] try_raise_barrier: nr_pending 1
> [  599.703020] try_raise_barrier: nr_pending 1
> [  599.703069] allow_barrier: nr_pending: 0
> [  599.708037] wait_barrier: nr_pending: 1
> [  599.712092] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  599.716368] try_raise_barrier: nr_pending 1
> [  599.716372] try_raise_barrier: nr_pending 1
> [  599.716377] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  599.738510] try_raise_barrier: nr_pending 1
> [  599.738554] allow_barrier: nr_pending: 0
> [  599.743663] wait_barrier: nr_pending: 1
> [  599.747583] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  599.751853] try_raise_barrier: nr_pending 1
> [  599.751857] try_raise_barrier: nr_pending 1
> [  599.751864] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  599.773945] try_raise_barrier: nr_pending 1
> [  599.774058] allow_barrier: nr_pending: 0
> [  599.779116] wait_barrier: nr_pending: 1
> [  599.783020] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  599.787298] try_raise_barrier: nr_pending 1
> [  599.787307] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  599.787461] allow_barrier: nr_pending: 0
> [  599.787469] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  599.815585] wait_barrier: nr_pending: 1
> [  599.819872] try_raise_barrier: nr_pending 1
> [  599.819883] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  599.824546] try_raise_barrier: nr_pending 1
> [  599.835892] try_raise_barrier: nr_pending 1
> [  599.835904] allow_barrier: nr_pending: 0
> [  599.840922] wait_barrier: nr_pending: 1
> [  599.844957] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  599.849235] try_raise_barrier: nr_pending 1
> [  599.849239] try_raise_barrier: nr_pending 1
> [  599.849244] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  599.871381] try_raise_barrier: nr_pending 1
> [  599.871463] allow_barrier: nr_pending: 0
> [  599.876542] wait_barrier: nr_pending: 1
> [  599.880451] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  599.884730] try_raise_barrier: nr_pending 1
> [  599.884735] try_raise_barrier: nr_pending 1
> [  599.884740] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  599.884889] allow_barrier: nr_pending: 0
> [  599.884897] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  599.917789] wait_barrier: nr_pending: 1
> [  599.922088] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  599.922159] try_raise_barrier: nr_pending 1
> [  599.928829] allow_barrier: nr_pending: 0
> [  599.933736] wait_barrier: nr_pending: 1
> [  599.937771] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  599.942045] try_raise_barrier: nr_pending 1
> [  599.942049] try_raise_barrier: nr_pending 1
> [  599.942058] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  599.964202] try_raise_barrier: nr_pending 1
> [  599.964262] allow_barrier: nr_pending: 0
> [  599.969461] wait_barrier: nr_pending: 1
> [  599.973275] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  599.977553] try_raise_barrier: nr_pending 1
> [  599.977559] try_raise_barrier: nr_pending 1
> [  599.977562] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  599.977715] allow_barrier: nr_pending: 0
> [  599.977723] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  600.010600] wait_barrier: nr_pending: 1
> [  600.014887] try_raise_barrier: nr_pending 1
> [  600.014898] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  600.019559] try_raise_barrier: nr_pending 1
> [  600.030909] try_raise_barrier: nr_pending 1
> [  600.030917] allow_barrier: nr_pending: 0
> [  600.036049] wait_barrier: nr_pending: 1
> [  600.039974] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  600.044253] try_raise_barrier: nr_pending 1
> [  600.044257] try_raise_barrier: nr_pending 1
> [  600.044263] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  600.066363] try_raise_barrier: nr_pending 1
> [  600.066461] allow_barrier: nr_pending: 0
> [  600.071440] wait_barrier: nr_pending: 1
> [  600.075443] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  600.079722] try_raise_barrier: nr_pending 1
> [  600.079727] try_raise_barrier: nr_pending 1
> [  600.079731] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  600.079868] allow_barrier: nr_pending: 0
> [  600.079875] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  600.112682] wait_barrier: nr_pending: 1
> [  600.116967] try_raise_barrier: nr_pending 1
> [  600.116978] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  600.121640] try_raise_barrier: nr_pending 1
> [  600.132990] try_raise_barrier: nr_pending 1
> [  600.133057] allow_barrier: nr_pending: 0
> [  600.138070] wait_barrier: nr_pending: 1
> [  600.142062] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  600.146337] try_raise_barrier: nr_pending 1
> [  600.146342] try_raise_barrier: nr_pending 1
> [  600.146348] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  600.168487] try_raise_barrier: nr_pending 1
> [  600.168532] allow_barrier: nr_pending: 0
> [  600.173662] wait_barrier: nr_pending: 1
> [  600.177563] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  600.181840] try_raise_barrier: nr_pending 1
> [  600.181845] try_raise_barrier: nr_pending 1
> [  600.181849] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  600.181983] allow_barrier: nr_pending: 0
> [  600.181991] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  600.214790] wait_barrier: nr_pending: 1
> [  600.219075] try_raise_barrier: nr_pending 1
> [  600.219087] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  600.223748] try_raise_barrier: nr_pending 1
> [  600.235099] try_raise_barrier: nr_pending 1
> [  600.235150] allow_barrier: nr_pending: 0
> [  600.240130] wait_barrier: nr_pending: 1
> [  600.244170] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  600.248446] try_raise_barrier: nr_pending 1
> [  600.248450] try_raise_barrier: nr_pending 1
> [  600.248456] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  600.270600] try_raise_barrier: nr_pending 1
> [  600.270648] allow_barrier: nr_pending: 0
> [  600.275752] wait_barrier: nr_pending: 1
> [  600.279676] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  600.283952] try_raise_barrier: nr_pending 1
> [  600.283956] try_raise_barrier: nr_pending 1
> [  600.283961] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  600.284093] allow_barrier: nr_pending: 0
> [  600.284100] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  600.316975] wait_barrier: nr_pending: 1
> [  600.321262] try_raise_barrier: nr_pending 1
> [  600.321273] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  600.325936] try_raise_barrier: nr_pending 1
> [  600.337279] allow_barrier: nr_pending: 0
> [  600.337649] wait_barrier: nr_pending: 1
> [  600.341674] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  600.345947] try_raise_barrier: nr_pending 1
> [  600.345951] try_raise_barrier: nr_pending 1
> [  600.345958] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  600.368088] try_raise_barrier: nr_pending 1
> [  600.368129] allow_barrier: nr_pending: 0
> [  600.373070] wait_barrier: nr_pending: 1
> [  600.377153] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  600.381430] try_raise_barrier: nr_pending 1
> [  600.381434] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  600.381555] allow_barrier: nr_pending: 0
> [  600.381562] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  600.409730] wait_barrier: nr_pending: 1
> [  600.414012] try_raise_barrier: nr_pending 1
> [  600.414015] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  600.418685] try_raise_barrier: nr_pending 1
> [  600.430034] try_raise_barrier: nr_pending 1
> [  600.430085] allow_barrier: nr_pending: 0
> [  600.435126] wait_barrier: nr_pending: 1
> [  600.439107] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  600.443388] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  600.443462] try_raise_barrier: nr_pending 1
> [  600.443466] try_raise_barrier: nr_pending 1
> [  600.465548] try_raise_barrier: nr_pending 1
> [  600.465585] allow_barrier: nr_pending: 0
> [  600.470613] wait_barrier: nr_pending: 1
> [  600.474627] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  600.478903] try_raise_barrier: nr_pending 1
> [  600.478910] try_raise_barrier: nr_pending 1
> [  600.478912] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  600.479045] allow_barrier: nr_pending: 0
> [  600.479053] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  600.511838] wait_barrier: nr_pending: 1
> [  600.516124] try_raise_barrier: nr_pending 1
> [  600.516135] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  600.520797] try_raise_barrier: nr_pending 1
> [  600.532156] try_raise_barrier: nr_pending 1
> [  600.532225] allow_barrier: nr_pending: 0
> [  600.537288] wait_barrier: nr_pending: 1
> [  600.541229] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  600.545505] try_raise_barrier: nr_pending 1
> [  600.545509] try_raise_barrier: nr_pending 1
> [  600.545515] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  600.567662] try_raise_barrier: nr_pending 1
> [  600.567696] allow_barrier: nr_pending: 0
> [  600.572778] wait_barrier: nr_pending: 1
> [  600.576742] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  600.581017] try_raise_barrier: nr_pending 1
> [  600.581023] try_raise_barrier: nr_pending 1
> [  600.581028] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  600.581145] allow_barrier: nr_pending: 0
> [  600.581152] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  600.613893] wait_barrier: nr_pending: 1
> [  600.618178] try_raise_barrier: nr_pending 1
> [  600.618188] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  600.622851] try_raise_barrier: nr_pending 1
> [  600.634242] try_raise_barrier: nr_pending 1
> [  600.634317] allow_barrier: nr_pending: 0
> [  600.639159] wait_barrier: nr_pending: 1
> [  600.643299] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  600.647575] try_raise_barrier: nr_pending 1
> [  600.647579] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  600.647579] try_raise_barrier: nr_pending 1
> [  600.647633] allow_barrier: nr_pending: 0
> [  600.647637] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  600.680526] wait_barrier: nr_pending: 1
> [  600.684819] try_raise_barrier: nr_pending 1
> [  600.684841] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  600.689492] try_raise_barrier: nr_pending 1
> [  600.700853] try_raise_barrier: nr_pending 1
> [  600.700910] allow_barrier: nr_pending: 0
> [  600.706030] wait_barrier: nr_pending: 1
> [  600.709914] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  600.714193] try_raise_barrier: nr_pending 1
> [  600.714197] try_raise_barrier: nr_pending 1
> [  600.714205] raid10_read_request: r10_bio ff3b4134184a6100 start
> [  600.714376] allow_barrier: nr_pending: 0
> [  600.714383] raid_end_bio_io: r10_bio ff3b4134184a6100 done
> [  600.747144] wait_barrier: nr_pending: 1
> [  600.751430] try_raise_barrier: nr_pending 1
> [  600.751442] raid10_read_request: r10_bio ff3b4134184a6b00 start
> [  600.756102] try_raise_barrier: nr_pending 1
> [  600.767448] try_raise_barrier: nr_pending 1
> [  600.767537] allow_barrier: nr_pending: 0
> [  600.772502] wait_barrier: nr_pending: 1
> [  600.776516] raid_end_bio_io: r10_bio ff3b4134184a6b00 done
> [  600.780795] try_raise_barrier: nr_pending 1
> [  600.780799] try_raise_barrier: nr_pending 1
> [  600.780808] raid10_read_request: r10_bio ff3b4134184a7f00 start
> [  600.802876] try_raise_barrier: nr_pending 1
> [  600.803016] allow_barrier: nr_pending: 0
> [  600.808000] wait_barrier: nr_pending: 1
> [  600.811943] raid_end_bio_io: r10_bio ff3b4134184a7f00 done
> [  600.816225] try_raise_barrier: nr_pending 1
> [  600.816229] try_raise_barrier: nr_pending 1
> [  600.816245] raid10_read_request: r10_bio ff3b4134184a7400 start
> [  600.838385] try_raise_barrier: nr_pending 1
> [  600.838448] allow_barrier: nr_pending: 0
> [  600.843556] wait_barrier: nr_pending: 1
> [  600.847451] raid_end_bio_io: r10_bio ff3b4134184a7400 done
> [  600.851722] try_raise_barrier: nr_pending 1
> [  600.851726] try_raise_barrier: nr_pending 1
> [  600.851732] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  600.851897] allow_barrier: nr_pending: 0
> [  600.851904] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  600.884632] wait_barrier: nr_pending: 1
> [  600.888917] try_raise_barrier: nr_pending 1
> [  600.888929] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  600.893588] try_raise_barrier: nr_pending 1
> [  600.904881] try_raise_barrier: nr_pending 1
> [  600.905028] allow_barrier: nr_pending: 0
> [  600.909930] wait_barrier: nr_pending: 1
> [  600.913958] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  600.918233] try_raise_barrier: nr_pending 1
> [  600.918237] try_raise_barrier: nr_pending 1
> [  600.918245] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  600.940401] try_raise_barrier: nr_pending 1
> [  600.940502] allow_barrier: nr_pending: 0
> [  600.945589] wait_barrier: nr_pending: 1
> [  600.949477] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  600.953755] try_raise_barrier: nr_pending 1
> [  600.953760] try_raise_barrier: nr_pending 1
> [  600.953765] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  600.953893] allow_barrier: nr_pending: 0
> [  600.953901] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  600.986653] wait_barrier: nr_pending: 1
> [  600.990939] try_raise_barrier: nr_pending 1
> [  600.990949] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  600.995611] try_raise_barrier: nr_pending 1
> [  601.006963] try_raise_barrier: nr_pending 1
> [  601.007109] allow_barrier: nr_pending: 0
> [  601.011808] wait_barrier: nr_pending: 1
> [  601.016031] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  601.020309] try_raise_barrier: nr_pending 1
> [  601.020313] try_raise_barrier: nr_pending 1
> [  601.020319] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  601.042453] allow_barrier: nr_pending: 0
> [  601.042803] wait_barrier: nr_pending: 1
> [  601.046853] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  601.051120] try_raise_barrier: nr_pending 1
> [  601.051125] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  601.051126] try_raise_barrier: nr_pending 1
> [  601.051145] try_raise_barrier: nr_pending 1
> [  601.051415] allow_barrier: nr_pending: 0
> [  601.051422] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  601.088815] wait_barrier: nr_pending: 1
> [  601.093099] try_raise_barrier: nr_pending 1
> [  601.093111] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  601.097772] try_raise_barrier: nr_pending 1
> [  601.109103] try_raise_barrier: nr_pending 1
> [  601.109172] allow_barrier: nr_pending: 0
> [  601.114259] wait_barrier: nr_pending: 1
> [  601.118177] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  601.122446] try_raise_barrier: nr_pending 1
> [  601.122450] try_raise_barrier: nr_pending 1
> [  601.122459] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  601.144607] try_raise_barrier: nr_pending 1
> [  601.144650] allow_barrier: nr_pending: 0
> [  601.149688] wait_barrier: nr_pending: 1
> [  601.153683] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  601.157960] try_raise_barrier: nr_pending 1
> [  601.157964] try_raise_barrier: nr_pending 1
> [  601.157970] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  601.180129] try_raise_barrier: nr_pending 1
> [  601.180240] allow_barrier: nr_pending: 0
> [  601.185207] wait_barrier: nr_pending: 1
> [  601.189209] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  601.193485] try_raise_barrier: nr_pending 1
> [  601.193497] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  601.193706] allow_barrier: nr_pending: 0
> [  601.193714] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  601.221868] wait_barrier: nr_pending: 1
> [  601.226153] try_raise_barrier: nr_pending 1
> [  601.226164] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  601.230826] try_raise_barrier: nr_pending 1
> [  601.242175] try_raise_barrier: nr_pending 1
> [  601.242239] allow_barrier: nr_pending: 0
> [  601.247321] wait_barrier: nr_pending: 1
> [  601.251249] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  601.255526] try_raise_barrier: nr_pending 1
> [  601.255530] try_raise_barrier: nr_pending 1
> [  601.255536] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  601.277688] try_raise_barrier: nr_pending 1
> [  601.277818] allow_barrier: nr_pending: 0
> [  601.282730] wait_barrier: nr_pending: 1
> [  601.286761] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  601.291037] try_raise_barrier: nr_pending 1
> [  601.291043] try_raise_barrier: nr_pending 1
> [  601.291047] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  601.313190] try_raise_barrier: nr_pending 1
> [  601.313248] allow_barrier: nr_pending: 0
> [  601.318267] wait_barrier: nr_pending: 1
> [  601.322269] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  601.326544] try_raise_barrier: nr_pending 1
> [  601.326551] try_raise_barrier: nr_pending 1
> [  601.326556] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  601.326708] allow_barrier: nr_pending: 0
> [  601.326716] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  601.359595] wait_barrier: nr_pending: 1
> [  601.363882] try_raise_barrier: nr_pending 1
> [  601.363893] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  601.368556] try_raise_barrier: nr_pending 1
> [  601.379906] try_raise_barrier: nr_pending 1
> [  601.379973] allow_barrier: nr_pending: 0
> [  601.385040] wait_barrier: nr_pending: 1
> [  601.388979] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  601.393255] try_raise_barrier: nr_pending 1
> [  601.393260] try_raise_barrier: nr_pending 1
> [  601.393268] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  601.415361] try_raise_barrier: nr_pending 1
> [  601.415483] allow_barrier: nr_pending: 0
> [  601.420443] wait_barrier: nr_pending: 1
> [  601.424436] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  601.428713] try_raise_barrier: nr_pending 1
> [  601.428716] try_raise_barrier: nr_pending 1
> [  601.428723] raid10_read_request: r10_bio ff3b41340eb53600 start
> [  601.450871] try_raise_barrier: nr_pending 1
> [  601.450916] allow_barrier: nr_pending: 0
> [  601.455878] wait_barrier: nr_pending: 1
> [  601.459951] raid_end_bio_io: r10_bio ff3b41340eb53600 done
> [  601.464227] try_raise_barrier: nr_pending 1
> [  601.464231] raid10_read_request: r10_bio ff3b41340eb52d00 start
> [  601.481634] try_raise_barrier: nr_pending 1
> [  601.486439] allow_barrier: nr_pending: 0
> [  601.486947] wait_barrier: nr_pending: 1
> [  601.490841] raid_end_bio_io: r10_bio ff3b41340eb52d00 done
> [  601.495112] try_raise_barrier: nr_pending 1
> [  601.495116] try_raise_barrier: nr_pending 1
> [  601.495122] raid10_read_request: r10_bio ff3b41340eb53900 start
> [  601.517260] try_raise_barrier: nr_pending 1
> [  601.517346] allow_barrier: nr_pending: 0
> [  601.522456] wait_barrier: nr_pending: 1
> [  601.526326] raid_end_bio_io: r10_bio ff3b41340eb53900 done
> [  601.530603] try_raise_barrier: nr_pending 1
> [  601.530608] try_raise_barrier: nr_pending 1
> [  601.530612] raid10_write_request: r10_bio ff3b41340eb52d00 start
> [  601.552782] try_raise_barrier: nr_pending 1

Thanks for the test!

Now the problem is clear. r10_bio is generated and never complete, while
IO never issued to underlying loop device, and I can reporduce this
problem as well now.

The blamed commit from git bisect is correct, and there are two
conditions for this problem:

1) plug is not used for caller;
2) for raid1/raid10, bio plug is enabled;

__submit_bio_noacct
 current->bio_list = ...;
 blk_start_plug

  dm_submit_bio
   md_handle_request
    raid10_write_request
     -> generate new bio for underlying disks
     raid1_add_bio_to_plug -> bio added to plug

 blk_finish_plug
  raid10_unplug
   raid1_submit_write
    submit_bio_noacct
     if (current->bio_list)
      -> bio_list is not empty
      bio_list_add(&current->bio_list[0], bio)

 current->bio_list = NULL
 -> the bio is in bio_list, and it's dropped.

Can you give the following patch a test? It should be a final test...

Thanks,
Kuai

diff --git a/block/blk-core.c b/block/blk-core.c
index 01186333c88e..1e9208024e47 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -613,9 +613,13 @@ static inline blk_status_t 
blk_check_zone_append(struct request_queue *q,

 static void __submit_bio(struct bio *bio)
 {
+       struct blk_plug plug;
+
        if (unlikely(!blk_crypto_bio_prep(&bio)))
                return;

+       blk_start_plug(&plug);
+
        if (!bio->bi_bdev->bd_has_submit_bio) {
                blk_mq_submit_bio(bio);
        } else if (likely(bio_queue_enter(bio) == 0)) {
@@ -624,6 +628,8 @@ static void __submit_bio(struct bio *bio)
                disk->fops->submit_bio(bio);
                blk_queue_exit(disk->queue);
        }
+
+       blk_finish_plug(&plug);
 }

 /*
@@ -648,13 +654,11 @@ static void __submit_bio(struct bio *bio)
 static void __submit_bio_noacct(struct bio *bio)
 {
        struct bio_list bio_list_on_stack[2];
-       struct blk_plug plug;

        BUG_ON(bio->bi_next);

        bio_list_init(&bio_list_on_stack[0]);
        current->bio_list = bio_list_on_stack;
-       blk_start_plug(&plug);

        do {
                struct request_queue *q = bdev_get_queue(bio->bi_bdev);
@@ -688,23 +692,19 @@ static void __submit_bio_noacct(struct bio *bio)
                bio_list_merge(&bio_list_on_stack[0], 
&bio_list_on_stack[1]);
        } while ((bio = bio_list_pop(&bio_list_on_stack[0])));

-       blk_finish_plug(&plug);
        current->bio_list = NULL;
 }

 static void __submit_bio_noacct_mq(struct bio *bio)
 {
        struct bio_list bio_list[2] = { };
-       struct blk_plug plug;

        current->bio_list = bio_list;
-       blk_start_plug(&plug);

        do {
                __submit_bio(bio);
        } while ((bio = bio_list_pop(&bio_list[0])));

-       blk_finish_plug(&plug);
        current->bio_list = NULL;
 }

> [  737.970751] INFO: task mdX_resync:3115 blocked for more than 122 seconds.
> [  737.978361]       Not tainted 6.9.0+ #1
> [  737.982654] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [  737.991398] task:mdX_resync      state:D stack:0     pid:3115
> tgid:3115  ppid:2      flags:0x00004000
> [  737.991407] Call Trace:
> [  737.991410]  <TASK>
> [  737.991414]  __schedule+0x222/0x670
> [  737.991426]  schedule+0x2c/0xb0
> [  737.991434]  raise_barrier+0xca/0x1a0 [raid10]
> [  737.991450]  ? __pfx_autoremove_wake_function+0x10/0x10
> [  737.991459]  raid10_sync_request+0x2c3/0x19d0 [raid10]
> [  737.991472]  ? __switch_to_asm+0x39/0x70
> [  737.991478]  ? finish_task_switch.isra.0+0x8e/0x2a0
> [  737.991487]  ? __schedule+0x22a/0x670
> [  737.991491]  ? prepare_to_wait_event+0x5f/0x190
> [  737.991498]  md_do_sync+0x660/0x1040
> [  737.991508]  ? __pfx_autoremove_wake_function+0x10/0x10
> [  737.991517]  md_thread+0xad/0x160
> [  737.991521]  ? __pfx_md_thread+0x10/0x10
> [  737.991524]  kthread+0xdc/0x110
> [  737.991531]  ? __pfx_kthread+0x10/0x10
> [  737.991535]  ret_from_fork+0x2d/0x50
> [  737.991544]  ? __pfx_kthread+0x10/0x10
> [  737.991547]  ret_from_fork_asm+0x1a/0x30
> [  737.991553]  </TASK>
> [  860.850998] INFO: task mdX_resync:3115 blocked for more than 245 seconds.
> [  860.858606]       Not tainted 6.9.0+ #1
> [  860.862902] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [  860.871649] task:mdX_resync      state:D stack:0     pid:3115
> tgid:3115  ppid:2      flags:0x00004000
> [  860.871658] Call Trace:
> [  860.871660]  <TASK>
> [  860.871664]  __schedule+0x222/0x670
> [  860.871674]  schedule+0x2c/0xb0
> [  860.871681]  raise_barrier+0xca/0x1a0 [raid10]
> [  860.871696]  ? __pfx_autoremove_wake_function+0x10/0x10
> [  860.871704]  raid10_sync_request+0x2c3/0x19d0 [raid10]
> [  860.871717]  ? __switch_to_asm+0x39/0x70
> [  860.871722]  ? finish_task_switch.isra.0+0x8e/0x2a0
> [  860.871729]  ? __schedule+0x22a/0x670
> [  860.871733]  ? prepare_to_wait_event+0x5f/0x190
> [  860.871740]  md_do_sync+0x660/0x1040
> [  860.871748]  ? __pfx_autoremove_wake_function+0x10/0x10
> [  860.871756]  md_thread+0xad/0x160
> [  860.871761]  ? __pfx_md_thread+0x10/0x10
> [  860.871765]  kthread+0xdc/0x110
> [  860.871770]  ? __pfx_kthread+0x10/0x10
> [  860.871774]  ret_from_fork+0x2d/0x50
> [  860.871781]  ? __pfx_kthread+0x10/0x10
> [  860.871785]  ret_from_fork_asm+0x1a/0x30
> [  860.871790]  </TASK>
>
> Thanks,
> Changhui
>
> .