Alright, rebooted with "debug ignore_loglevel", log attached. I'm running an Areca ARC-1320ix-16 (with onboard expander) with 10 SATA drives attached. (Sending again because mailing list rejects HTML) On 29 April 2015 at 19:05, James Bottomley <James.Bottomley@xxxxxxxxxxxxxxxxxxxxx> wrote: > On Wed, 2015-04-29 at 09:24 -0700, Adam Talbot wrote: >> Attached is what I see on my 4.0.0 patched kernel. I am going through >> a SAS expander: >> [ 5.176095] scsi host4: mvsas >> [ 5.180076] sas: phy-4:0 added to port-4:0, phy_mask:0x1 (50030480002f51ff) >> [ 5.180079] drivers/scsi/mvsas/mv_sas.c 1118:set wide port phy map 1 >> [ 5.190007] sas: phy1 matched wide port0 >> [ 5.190010] sas: phy-4:1 added to port-4:0, phy_mask:0x3 (50030480002f51ff) >> [ 5.190012] drivers/scsi/mvsas/mv_sas.c 1118:set wide port phy map 3 >> [ 5.209850] sas: phy2 matched wide port0 >> [ 5.209853] sas: phy-4:2 added to port-4:0, phy_mask:0x7 (50030480002f51ff) >> [ 5.209854] drivers/scsi/mvsas/mv_sas.c 1118:set wide port phy map 7 >> [ 5.239609] sas: phy3 matched wide port0 >> [ 5.239612] sas: phy-4:3 added to port-4:0, phy_mask:0xf (50030480002f51ff) >> [ 5.239613] drivers/scsi/mvsas/mv_sas.c 1118:set wide port phy map f >> [ 5.279368] sas: DOING DISCOVERY on port 0, pid:6 >> [ 5.280102] sas: ex 50030480002f51ff phy00:S:9 attached: >> 5005043011ab0000 (host) >> [ 5.280356] sas: ex 50030480002f51ff phy01:S:9 attached: >> 5005043011ab0000 (host) >> [ 5.280622] sas: ex 50030480002f51ff phy02:S:9 attached: >> 5005043011ab0000 (host) >> [ 5.280877] sas: ex 50030480002f51ff phy03:S:9 attached: >> 5005043011ab0000 (host) >> [ 5.281141] sas: ex 50030480002f51ff phy04:D:9 attached: >> 50030480002f51c4 (stp) >> [ 5.281339] sas: ex 50030480002f51ff phy05:D:9 attached: >> 50030480002f51c5 (stp) >> [ 5.281547] sas: ex 50030480002f51ff phy06:D:9 attached: >> 50030480002f51c6 (stp) >> [ 5.281801] sas: ex 50030480002f51ff phy07:D:9 attached: >> 50030480002f51c7 (stp) >> [ 5.282003] sas: ex 50030480002f51ff phy08:D:9 attached: >> 50030480002f51c8 (stp) >> [ 5.282216] sas: ex 50030480002f51ff phy09:D:8 attached: >> 50030480002f51c9 (stp) >> [ 5.282419] sas: ex 50030480002f51ff phy10:D:8 attached: >> 50030480002f51ca (stp) >> [ 5.282636] sas: ex 50030480002f51ff phy11:D:8 attached: >> 50030480002f51cb (stp) >> [ 5.282862] sas: ex 50030480002f51ff phy12:D:8 attached: >> 50030480002f51cc (stp) >> [ 5.283042] sas: ex 50030480002f51ff phy13:D:9 attached: >> 50030480002f51cd (stp) >> [ 5.283202] sas: ex 50030480002f51ff phy14:D:8 attached: >> 50030480002f51ce (stp) >> [ 5.283355] sas: ex 50030480002f51ff phy15:D:0 attached: >> 0000000000000000 (no device) >> [ 5.283504] sas: ex 50030480002f51ff phy16:D:0 attached: >> 0000000000000000 (no device) >> [ 5.283655] sas: ex 50030480002f51ff phy17:D:0 attached: >> 0000000000000000 (no device) >> [ 5.283808] sas: ex 50030480002f51ff phy18:D:0 attached: >> 0000000000000000 (no device) >> [ 5.283957] sas: ex 50030480002f51ff phy19:D:0 attached: >> 0000000000000000 (no device) >> [ 5.284084] sas: ex 50030480002f51ff phy20:T:0 attached: >> 0000000000000000 (no device) >> [ 5.284235] sas: ex 50030480002f51ff phy21:T:0 attached: >> 0000000000000000 (no device) >> [ 5.284385] sas: ex 50030480002f51ff phy22:T:0 attached: >> 0000000000000000 (no device) >> [ 5.284508] sas: ex 50030480002f51ff phy23:T:0 attached: >> 0000000000000000 (no device) >> [ 5.284659] sas: ex 50030480002f51ff phy24:T:0 attached: >> 0000000000000000 (no device) >> [ 5.284787] sas: ex 50030480002f51ff phy25:T:0 attached: >> 0000000000000000 (no device) >> [ 5.284938] sas: ex 50030480002f51ff phy26:T:0 attached: >> 0000000000000000 (no device) >> [ 5.285091] sas: ex 50030480002f51ff phy27:T:0 attached: >> 0000000000000000 (no device) >> [ 5.285220] sas: ex 50030480002f51ff phy28:D:9 attached: >> 50030480002f51fd (host+target) >> [ 5.285346] sas: ex 50030480002f51ff phy29:D:0 attached: >> 0000000000000000 (no device) >> [ 5.286367] sas: DONE DISCOVERY on port 0, pid:6, result:0 >> [ 5.286426] sas: Enter sas_scsi_recover_host busy: 0 failed: 0 >> [ 5.286438] sas: ata5: end_device-4:0:4: dev error handler >> [ 5.286501] sas: ata6: end_device-4:0:5: dev error handler >> [ 5.286622] sas: ata7: end_device-4:0:6: dev error handler >> [ 5.286717] sas: ata8: end_device-4:0:7: dev error handler >> [ 5.286802] sas: ata9: end_device-4:0:8: dev error handler >> [ 5.286866] sas: ata10: end_device-4:0:9: dev error handler >> [ 5.286922] sas: ata11: end_device-4:0:10: dev error handler >> [ 5.287008] sas: ata12: end_device-4:0:11: dev error handler >> [ 5.287096] sas: ata13: end_device-4:0:12: dev error handler >> [ 5.287183] sas: ata14: end_device-4:0:13: dev error handler >> [ 5.287265] sas: ata15: end_device-4:0:14: dev error handler > > Right, but I actually need the configuration of the system that is > currently oopsing. Your trace seems to show a normal SATA discovery > (the error handler shows up because a reset fis is part of SATA > discovery). > > James > >
[ 96.800626] mvsas 0000:02:00.0: mvsas: driver version 0.8.16 [ 96.806547] mvsas 0000:02:00.0: mvsas: PCI-E x8, Bandwidth Usage: 5.0 Gbps [ 105.059102] scsi host6: mvsas [ 105.071339] sas: phy-6:0 added to port-6:0, phy_mask:0x1 (5001b4d5183a503f) [ 105.081301] sas: DOING DISCOVERY on port 0, pid:250 [ 105.081940] sas: ex 5001b4d5183a503f phy00:U:9 attached: 5001b4d5183a5000 (stp) [ 105.082138] sas: ex 5001b4d5183a503f phy01:U:9 attached: 5001b4d5183a5001 (stp) [ 105.082340] sas: ex 5001b4d5183a503f phy02:U:9 attached: 5001b4d5183a5002 (stp) [ 105.082536] sas: ex 5001b4d5183a503f phy03:U:9 attached: 5001b4d5183a5003 (stp) [ 105.082708] sas: ex 5001b4d5183a503f phy04:D:0 attached: 0000000000000000 (no device) [ 105.082877] sas: ex 5001b4d5183a503f phy05:D:0 attached: 0000000000000000 (no device) [ 105.083042] sas: ex 5001b4d5183a503f phy06:D:0 attached: 0000000000000000 (no device) [ 105.083208] sas: ex 5001b4d5183a503f phy07:D:0 attached: 0000000000000000 (no device) [ 105.083372] sas: ex 5001b4d5183a503f phy08:D:0 attached: 0000000000000000 (no device) [ 105.083534] sas: ex 5001b4d5183a503f phy09:D:0 attached: 0000000000000000 (no device) [ 105.083696] sas: ex 5001b4d5183a503f phy10:D:0 attached: 0000000000000000 (no device) [ 105.083859] sas: ex 5001b4d5183a503f phy11:D:0 attached: 0000000000000000 (no device) [ 105.084029] sas: ex 5001b4d5183a503f phy12:D:0 attached: 0000000000000000 (no device) [ 105.084194] sas: ex 5001b4d5183a503f phy13:D:0 attached: 0000000000000000 (no device) [ 105.084364] sas: ex 5001b4d5183a503f phy14:D:0 attached: 0000000000000000 (no device) [ 105.084530] sas: ex 5001b4d5183a503f phy15:D:0 attached: 0000000000000000 (no device) [ 105.084727] sas: ex 5001b4d5183a503f phy16:U:0 attached: 0000000000000000 (no device) [ 105.084920] sas: ex 5001b4d5183a503f phy17:U:0 attached: 0000000000000000 (no device) [ 105.085114] sas: ex 5001b4d5183a503f phy18:U:0 attached: 0000000000000000 (no device) [ 105.085308] sas: ex 5001b4d5183a503f phy19:U:0 attached: 0000000000000000 (no device) [ 105.085481] sas: ex 5001b4d5183a503f phy20:D:0 attached: 0000000000000000 (no device) [ 105.085654] sas: ex 5001b4d5183a503f phy21:D:0 attached: 0000000000000000 (no device) [ 105.085816] sas: ex 5001b4d5183a503f phy22:D:0 attached: 0000000000000000 (no device) [ 105.085985] sas: ex 5001b4d5183a503f phy23:D:0 attached: 0000000000000000 (no device) [ 105.086187] sas: ex 5001b4d5183a503f phy24:S:A attached: 5005043011ab0000 (host) [ 105.086385] sas: ex 5001b4d5183a503f phy25:S:A attached: 5005043011ab0000 (host) [ 105.086584] sas: ex 5001b4d5183a503f phy26:S:A attached: 5005043011ab0000 (host) [ 105.086786] sas: ex 5001b4d5183a503f phy27:S:A attached: 5005043011ab0000 (host) [ 105.087004] sas: ex 5001b4d5183a503f phy28:U:9 attached: 5001b4d5183a501c (stp) [ 105.087205] sas: ex 5001b4d5183a503f phy29:U:9 attached: 5001b4d5183a501d (stp) [ 105.087408] sas: ex 5001b4d5183a503f phy30:U:A attached: 5001b4d5183a501e (stp) [ 105.087606] sas: ex 5001b4d5183a503f phy31:U:9 attached: 5001b4d5183a501f (stp) [ 105.087791] sas: ex 5001b4d5183a503f phy32:D:0 attached: 0000000000000000 (no device) [ 105.087888] sas: ex 5001b4d5183a503f phy33:D:0 attached: 0000000000000000 (no device) [ 105.096231] sas: ex 5001b4d5183a503f phy34:D:0 attached: 0000000000000000 (no device) [ 105.096423] sas: ex 5001b4d5183a503f phy35:D:0 attached: 0000000000000000 (no device) [ 105.096621] sas: ex 5001b4d5183a503f phy36:D:A attached: 5001b4d5183a503d (host+target) [ 105.096821] sas: ex 5001b4d5183a503f phy37:D:0 attached: 0000000000000000 (no device) [ 105.098240] sas: DONE DISCOVERY on port 0, pid:250, result:0 [ 105.098320] sas: Enter sas_scsi_recover_host busy: 0 failed: 0 [ 105.098334] sas: ata7: end_device-6:0:0: dev error handler [ 105.098337] sas: ata8: end_device-6:0:1: dev error handler [ 105.098344] sas: ata9: end_device-6:0:2: dev error handler [ 105.098351] sas: ata10: end_device-6:0:3: dev error handler [ 105.098355] sas: ata11: end_device-6:0:28: dev error handler [ 105.098362] sas: ata12: end_device-6:0:29: dev error handler [ 105.098395] sas: ata13: end_device-6:0:30: dev error handler [ 105.098419] sas: ata14: end_device-6:0:31: dev error handler [ 110.248449] ata12.00: qc timeout (cmd 0xec) [ 110.248452] ata11.00: qc timeout (cmd 0xec) [ 110.248480] ata11.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 110.248490] ata8.00: qc timeout (cmd 0xec) [ 110.248501] ata8.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 110.248586] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 110.248592] ata9.00: qc timeout (cmd 0xec) [ 110.248611] ata9.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 110.248633] ata7.00: qc timeout (cmd 0xec) [ 110.248651] ata7.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 110.248659] ata10.00: qc timeout (cmd 0xec) [ 110.248665] ata10.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 110.248673] ata13.00: qc timeout (cmd 0xec) [ 110.248681] ata13.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 110.248689] ata14.00: qc timeout (cmd 0xec) [ 110.248695] ata14.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 110.851518] ata12.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 110.880293] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 112.429217] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 112.580880] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 112.784502] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 112.816451] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 112.847317] ata13.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80) [ 117.995461] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 122.396479] ata8.00: qc timeout (cmd 0xec) [ 122.396482] ata11.00: qc timeout (cmd 0xec) [ 122.396508] ata11.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 122.448762] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 122.448781] ata8.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 122.448839] ata9.00: qc timeout (cmd 0xec) [ 122.448844] ata9.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 122.628245] ata7.00: qc timeout (cmd 0xec) [ 122.647598] ata7.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 122.780097] ata10.00: qc timeout (cmd 0xec) [ 122.799541] ata10.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 122.899934] ata14.00: qc timeout (cmd 0xec) [ 122.899945] ata14.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 123.043836] ata12.00: qc timeout (cmd 0xec) [ 123.043840] ata12.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 124.696696] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 124.748765] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 124.857893] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 124.969983] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 125.196294] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 125.280166] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 127.990965] ata13.00: qc timeout (cmd 0xec) [ 128.010416] ata13.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 128.039150] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 154.628723] ata11.00: qc timeout (cmd 0xec) [ 154.648169] ata11.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 154.676881] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 154.676889] ata8.00: qc timeout (cmd 0xec) [ 154.676894] ata8.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 154.736626] ata9.00: qc timeout (cmd 0xec) [ 154.736630] ata9.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 154.836453] ata7.00: qc timeout (cmd 0xec) [ 154.836456] ata7.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 154.940427] ata10.00: qc timeout (cmd 0xec) [ 154.940431] ata10.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 155.172199] ata14.00: qc timeout (cmd 0xec) [ 155.191632] ata14.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 155.256114] ata12.00: qc timeout (cmd 0xec) [ 155.256118] ata12.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 157.526159] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 178.253527] sas: Enter sas_scsi_recover_host busy: 1 failed: 1 [ 178.259412] sas: trying to find task 0xffff880214348800 [ 178.264660] sas: sas_scsi_find_task: aborting task 0xffff880214348800 [ 198.339499] sas: sas_scsi_find_task: querying task 0xffff880214348800 [ 218.415801] sas: sas_scsi_find_task: task 0xffff880214348800 failed to abort [ 218.422879] sas: task 0xffff880214348800 is not at LU: I_T recover [ 218.429080] sas: I_T nexus reset for dev 5001b4d5183a503d [ 218.457082] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 220.533343] sas: task done but aborted [ 220.537116] sas: I_T 5001b4d5183a503d recovered [ 220.541757] sas: ata7: end_device-6:0:0: dev error handler [ 220.541769] sas: ata8: end_device-6:0:1: dev error handler [ 220.541771] sas: ata9: end_device-6:0:2: dev error handler [ 220.541774] sas: ata10: end_device-6:0:3: dev error handler [ 220.541785] sas: ata12: end_device-6:0:29: dev error handler [ 220.541785] sas: ata13: end_device-6:0:30: dev error handler [ 220.541788] sas: ata11: end_device-6:0:28: dev error handler [ 220.541789] sas: ata14: end_device-6:0:31: dev error handler [ 220.586707] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 220.604342] scsi 6:0:8:0: Enclosure Areca ARC-8017-4.01.16 0116 PQ: 0 ANSI: 5 [ 240.608062] INFO: task kworker/u16:0:6 blocked for more than 120 seconds. [ 240.614874] Tainted: G OE 3.19.0-15-generic #15-Ubuntu [ 240.621503] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 240.629347] kworker/u16:0 D ffff880215633ae8 0 6 2 0x00000000 [ 240.636526] Workqueue: events_unbound async_run_entry_fn [ 240.641905] ffff880215633ae8 ffff88021558b110 0000000000014200 ffff880215633fd8 [ 240.649514] 0000000000014200 ffff880215650000 ffff88021558b110 ffff880215633ad8 [ 240.657099] ffff880215633c68 7fffffffffffffff ffff880215633c60 ffff88021558b110 [ 240.664709] Call Trace: [ 240.667190] [<ffffffff817c49c9>] schedule+0x29/0x70 [ 240.672177] [<ffffffff817c7fac>] schedule_timeout+0x20c/0x280 [ 240.678032] [<ffffffff8109ed3d>] ? ttwu_do_activate.constprop.94+0x5d/0x70 [ 240.685018] [<ffffffff810a1c19>] ? try_to_wake_up+0x1e9/0x330 [ 240.690870] [<ffffffff817c6384>] wait_for_completion+0xa4/0x170 [ 240.696904] [<ffffffff810a1dd0>] ? wake_up_state+0x20/0x20 [ 240.703042] [<ffffffff8108d62c>] flush_workqueue+0x10c/0x5f0 [ 240.708808] [<ffffffff8108dbce>] drain_workqueue+0x9e/0x170 [ 240.714489] [<ffffffffc04cd151>] __sas_drain_work+0x51/0x120 [libsas] [ 240.721052] [<ffffffffc04cd268>] sas_drain_work+0x48/0x50 [libsas] [ 240.727348] [<ffffffffc04eba3d>] mvs_scan_finished+0x2d/0x40 [mvsas] [ 240.733804] [<ffffffff815609a7>] do_scsi_scan_host+0x77/0xa0 [ 240.739587] [<ffffffff81560bac>] do_scan_async+0x1c/0x190 [ 240.745102] [<ffffffff8109730c>] async_run_entry_fn+0x4c/0x170 [ 240.751042] [<ffffffff8108eb78>] process_one_work+0x158/0x430 [ 240.756902] [<ffffffff8108f6bb>] worker_thread+0x5b/0x530 [ 240.762410] [<ffffffff8108f660>] ? rescuer_thread+0x3a0/0x3a0 [ 240.768287] [<ffffffff81094759>] kthread+0xc9/0xe0 [ 240.773206] [<ffffffff81094690>] ? kthread_create_on_node+0x1c0/0x1c0 [ 240.779752] [<ffffffff817c9298>] ret_from_fork+0x58/0x90 [ 240.785170] [<ffffffff81094690>] ? kthread_create_on_node+0x1c0/0x1c0 [ 251.205686] sas: Enter sas_scsi_recover_host busy: 1 failed: 1 [ 251.211564] sas: trying to find task 0xffff8800cf438500 [ 251.216823] sas: sas_scsi_find_task: aborting task 0xffff8800cf438500 [ 271.291646] sas: sas_scsi_find_task: querying task 0xffff8800cf438500 [ 291.367979] sas: sas_scsi_find_task: task 0xffff8800cf438500 failed to abort [ 291.375060] sas: task 0xffff8800cf438500 is not at LU: I_T recover [ 291.381258] sas: I_T nexus reset for dev 5001b4d5183a503d [ 291.409235] sas: smp_execute_task: task to dev 5001b4d5183a503f response: 0x0 status 0x2 [ 293.457540] sas: task done but aborted [ 293.461312] sas: I_T 5001b4d5183a503d recovered [ 293.465950] sas: ata7: end_device-6:0:0: dev error handler [ 293.465954] sas: ata8: end_device-6:0:1: dev error handler [ 293.465964] sas: ata9: end_device-6:0:2: dev error handler [ 293.465967] sas: ata10: end_device-6:0:3: dev error handler [ 293.465974] sas: ata12: end_device-6:0:29: dev error handler [ 293.465975] sas: ata11: end_device-6:0:28: dev error handler [ 293.465980] sas: ata13: end_device-6:0:30: dev error handler [ 293.465983] sas: ata14: end_device-6:0:31: dev error handler [ 293.576160] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 324.093863] sas: Enter sas_scsi_recover_host busy: 1 failed: 1 [ 324.099735] sas: trying to find task 0xffff880212be3b00 [ 324.104995] sas: sas_scsi_find_task: aborting task 0xffff880212be3b00 [ 324.111577] sas: task done but aborted [ 324.115373] sas: sas_scsi_find_task: task 0xffff880212be3b00 is done [ 324.121749] sas: sas_eh_handle_sas_errors: task 0xffff880212be3b00 is done [ 324.128739] sas: ata7: end_device-6:0:0: dev error handler [ 324.128744] sas: ata8: end_device-6:0:1: dev error handler [ 324.128754] sas: ata9: end_device-6:0:2: dev error handler [ 324.128764] sas: ata10: end_device-6:0:3: dev error handler [ 324.128765] sas: ata11: end_device-6:0:28: dev error handler [ 324.128772] sas: ata12: end_device-6:0:29: dev error handler [ 324.128775] sas: ata13: end_device-6:0:30: dev error handler [ 324.128778] sas: ata14: end_device-6:0:31: dev error handler [ 324.173706] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 355.167309] sas: Enter sas_scsi_recover_host busy: 1 failed: 1 [ 355.173169] sas: trying to find task 0xffff8800cf438800 [ 355.178417] sas: sas_scsi_find_task: aborting task 0xffff8800cf438800 [ 360.669828] INFO: task kworker/u16:0:6 blocked for more than 120 seconds. [ 360.676650] Tainted: G OE 3.19.0-15-generic #15-Ubuntu [ 360.683277] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 360.691133] kworker/u16:0 D ffff880215633ae8 0 6 2 0x00000000 [ 360.698299] Workqueue: events_unbound async_run_entry_fn [ 360.703681] ffff880215633ae8 ffff88021558b110 0000000000014200 ffff880215633fd8 [ 360.711283] 0000000000014200 ffff880215650000 ffff88021558b110 ffff880215633ad8 [ 360.718901] ffff880215633c68 7fffffffffffffff ffff880215633c60 ffff88021558b110 [ 360.726510] Call Trace: [ 360.728992] [<ffffffff817c49c9>] schedule+0x29/0x70 [ 360.733979] [<ffffffff817c7fac>] schedule_timeout+0x20c/0x280 [ 360.739834] [<ffffffff8109ed3d>] ? ttwu_do_activate.constprop.94+0x5d/0x70 [ 360.746812] [<ffffffff810a1c19>] ? try_to_wake_up+0x1e9/0x330 [ 360.752673] [<ffffffff817c6384>] wait_for_completion+0xa4/0x170 [ 360.758707] [<ffffffff810a1dd0>] ? wake_up_state+0x20/0x20 [ 360.764300] [<ffffffff8108d62c>] flush_workqueue+0x10c/0x5f0 [ 360.770064] [<ffffffff8108dbce>] drain_workqueue+0x9e/0x170 [ 360.775746] [<ffffffffc04cd151>] __sas_drain_work+0x51/0x120 [libsas] [ 360.782292] [<ffffffffc04cd268>] sas_drain_work+0x48/0x50 [libsas] [ 360.788577] [<ffffffffc04eba3d>] mvs_scan_finished+0x2d/0x40 [mvsas] [ 360.795043] [<ffffffff815609a7>] do_scsi_scan_host+0x77/0xa0 [ 360.800819] [<ffffffff81560bac>] do_scan_async+0x1c/0x190 [ 360.806323] [<ffffffff8109730c>] async_run_entry_fn+0x4c/0x170 [ 360.812263] [<ffffffff8108eb78>] process_one_work+0x158/0x430 [ 360.818134] [<ffffffff8108f6bb>] worker_thread+0x5b/0x530 [ 360.823649] [<ffffffff8108f660>] ? rescuer_thread+0x3a0/0x3a0 [ 360.829510] [<ffffffff81094759>] kthread+0xc9/0xe0 [ 360.834973] [<ffffffff81094690>] ? kthread_create_on_node+0x1c0/0x1c0 [ 360.841519] [<ffffffff817c9298>] ret_from_fork+0x58/0x90 [ 360.846939] [<ffffffff81094690>] ? kthread_create_on_node+0x1c0/0x1c0 [ 375.253286] sas: sas_scsi_find_task: querying task 0xffff8800cf438800 [ 395.329594] sas: sas_scsi_find_task: task 0xffff8800cf438800 failed to abort [ 395.336674] sas: task 0xffff8800cf438800 is not at LU: I_T recover [ 395.342870] sas: I_T nexus reset for dev 5001b4d5183a503d [ 397.409829] sas: task done but aborted [ 397.413601] sas: I_T 5001b4d5183a503d recovered [ 397.418244] sas: ata7: end_device-6:0:0: dev error handler [ 397.418252] sas: ata8: end_device-6:0:1: dev error handler [ 397.418254] sas: ata9: end_device-6:0:2: dev error handler [ 397.418259] sas: ata10: end_device-6:0:3: dev error handler [ 397.418259] sas: ata11: end_device-6:0:28: dev error handler [ 397.418261] sas: ata12: end_device-6:0:29: dev error handler [ 397.418262] sas: ata13: end_device-6:0:30: dev error handler [ 397.418264] sas: ata14: end_device-6:0:31: dev error handler [ 397.509790] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 404.023207] sas: Enter sas_scsi_recover_host busy: 1 failed: 1 [ 404.029077] sas: trying to find task 0xffff8800cf438e00 [ 404.034342] sas: sas_scsi_find_task: aborting task 0xffff8800cf438e00 [ 424.101167] sas: sas_scsi_find_task: querying task 0xffff8800cf438e00 [ 444.177474] sas: sas_scsi_find_task: task 0xffff8800cf438e00 failed to abort [ 444.184546] sas: task 0xffff8800cf438e00 is not at LU: I_T recover [ 444.190762] sas: I_T nexus reset for dev 5001b4d5183a503d [ 446.248405] sas: task done but aborted [ 446.252175] sas: I_T 5001b4d5183a503d recovered [ 446.256819] sas: ata7: end_device-6:0:0: dev error handler [ 446.256824] sas: ata8: end_device-6:0:1: dev error handler [ 446.256833] sas: ata9: end_device-6:0:2: dev error handler [ 446.256837] sas: ata10: end_device-6:0:3: dev error handler [ 446.256846] sas: ata11: end_device-6:0:28: dev error handler [ 446.256847] sas: ata12: end_device-6:0:29: dev error handler [ 446.256853] sas: ata13: end_device-6:0:30: dev error handler [ 446.256857] sas: ata14: end_device-6:0:31: dev error handler [ 446.302354] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 446.328165] sas: phy1 matched wide port0 [ 446.328170] sas: phy-6:1 added to port-6:0, phy_mask:0x3 (5001b4d5183a503f) [ 446.348000] sas: phy2 matched wide port0 [ 446.348003] sas: phy-6:2 added to port-6:0, phy_mask:0x7 (5001b4d5183a503f) [ 446.377743] sas: phy3 matched wide port0 [ 446.377747] sas: phy-6:3 added to port-6:0, phy_mask:0xf (5001b4d5183a503f) [ 446.417398] sas: phy0 matched wide port0 [ 446.417401] sas: phy-6:4 added to port-6:0, phy_mask:0xf (5001b4d5183a503f) [ 446.485246] sas: phy1 matched wide port0 [ 446.489202] sas: phy-6:5 added to port-6:0, phy_mask:0xf (5001b4d5183a503f) [ 446.544991] sas: phy2 matched wide port0 [ 446.548950] sas: phy-6:6 added to port-6:0, phy_mask:0xf (5001b4d5183a503f) [ 446.604723] sas: phy3 matched wide port0 [ 446.608671] sas: phy-6:7 added to port-6:0, phy_mask:0xf (5001b4d5183a503f) [ 446.664514] sas: broadcast received: 0 [ 446.664861] scsi 6:0:8:0: Attached scsi generic sg3 type 13 [ 446.673921] sas: broadcast received: 0 [ 446.677699] sas: broadcast received: 0 [ 446.681492] sas: broadcast received: 0 [ 446.685265] sas: broadcast received: 0 [ 446.689050] sas: broadcast received: 0 [ 446.692832] sas: broadcast received: 0 [ 446.696608] sas: broadcast received: 0 [ 446.700382] sas: REVALIDATING DOMAIN on port 0, pid:250 [ 446.706001] sas: Expander phy change count has changed [ 446.711432] sas: ex 5001b4d5183a503f phy0 originated BROADCAST(CHANGE) [ 446.718461] sas: ex 5001b4d5183a503f phy 0x0 broadcast flutter [ 446.724526] sas: ex 5001b4d5183a503f phy1 originated BROADCAST(CHANGE) [ 446.731482] sas: ex 5001b4d5183a503f phy 0x1 broadcast flutter [ 446.737556] sas: ex 5001b4d5183a503f phy2 originated BROADCAST(CHANGE) [ 446.744570] sas: ex 5001b4d5183a503f phy 0x2 broadcast flutter [ 446.750633] sas: ex 5001b4d5183a503f phy3 originated BROADCAST(CHANGE) [ 446.757646] sas: ex 5001b4d5183a503f phy 0x3 broadcast flutter [ 446.768099] sas: ex 5001b4d5183a503f phy28 originated BROADCAST(CHANGE) [ 446.775116] sas: ex 5001b4d5183a503f phy 0x1c broadcast flutter [ 446.781248] sas: ex 5001b4d5183a503f phy29 originated BROADCAST(CHANGE) [ 446.788347] sas: ex 5001b4d5183a503f phy 0x1d broadcast flutter [ 446.794567] sas: ex 5001b4d5183a503f phy30 originated BROADCAST(CHANGE) [ 446.801603] sas: ex 5001b4d5183a503f phy 0x1e broadcast flutter [ 446.807737] sas: ex 5001b4d5183a503f phy31 originated BROADCAST(CHANGE) [ 446.814760] sas: ex 5001b4d5183a503f phy 0x1f broadcast flutter [ 446.821622] sas: ex 5001b4d5183a503f phy36 originated BROADCAST(CHANGE) [ 446.828625] sas: ex 5001b4d5183a503f phy 0x24 broadcast flutter [ 446.835006] sas: done REVALIDATING DOMAIN on port 0, pid:250, res 0x0
