Hi everyone, We have encountered a problem with LIO ISCSI and VMware ESXi 5.5 that is preventing us from using LIO in production. The test setup works and performs reasonably well. Unfortunately every 1 to 2 months everything hosted on the LIO target crashes and the ESX management tools become unresponsive until the storage server is rebooted. Even though the ESXi nodes are not responding (see logs below), I can discover, connect and mount the problematic LUN on any other Server. Error messages on the storage server: [5094061.802861] iSCSI/iqn.1998-01.com.vmware:esx03-0db0bf2c: Unsupported SCSI Opcode 0x4d, sending CHECK_CONDITION. [5094086.496004] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x0000003e [5094087.054555] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x0000003e [5094115.472850] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x0000003f [5094116.763556] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x000000aa [5094130.725901] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x000000ab [5094144.445284] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000040 [5094145.228081] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x000000ab The "Unsupported SCSI Opcode 0x4d" also occurs during normal operation. Error messages on ESX(shortened): 2017-04-18T17:50:17.434Z cpu14:33518)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba32:CH:0 T:6 CN:0: iSCSI connection is being marked "ONLINE" 2017-04-18T17:50:17.434Z cpu14:33518)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000002 TARGET: iqn.2016-11.local.xxx:vmstore01 TPGT: 1 TSIH: 0] 2017-04-18T17:50:17.434Z cpu14:33518)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 10.15.24.135:40650 R: 10.15.24.65:3260] 2017-04-18T17:50:18.976Z cpu13:104435914)HBX: 2959: Waiting for timed out [HB state abcdef02 offset 3928064 gen 55 stampUS 43972608588966 uuid 56573462-6e6fe026-3aa9-1cc1de771070 jrnl <FB 1722600> drv 14.60] on vol 'vmstore01-lun0' 2017-04-18T17:50:18.976Z cpu14:104107012)HBX: 2959: Waiting for timed out [HB state abcdef02 offset 3928064 gen 55 stampUS 43972608588966 uuid 56573462-6e6fe026-3aa9-1cc1de771070 jrnl <FB 1722600> drv 14.60] on vol 'vmstore01-lun0' 2017-04-18T17:50:28.098Z cpu10:33518)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba32:CH:0 T:6 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4) 2017-04-18T17:50:28.098Z cpu9:33518)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000002 TARGET: iqn.2016-11.local.xxx:vmstore01 TPGT: 1 TSIH: 0] 2017-04-18T17:50:28.098Z cpu9:33518)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.15.24.135:40650 R: 10.15.24.65:3260] 2017-04-18T17:50:28.098Z cpu11:33520)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6001405e7e34bd545064ac286fee4e07" state in doubt; requested fast path state update... 2017-04-18T17:50:28.098Z cpu9:33518)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba32:CH:0 T:6 L:0 : Task mgmt "Abort Task" with itt=0x13e81f1f (refITT=0x13e81f1e) timed out. 2017-04-18T17:50:28.098Z cpu11:33520)ScsiDeviceIO: 2325: Cmd(0x413683ad8cc0) 0x89, CmdSN 0xa12cd4 from world 32824 to dev "naa.6001405e7e34bd545064ac286fee4e07" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x5 0x25 0x0. 2017-04-18T17:50:28.099Z cpu12:115955886)VMW_SATP_ALUA: satp_alua_issueCommandOnPath:656: Path "vmhba32:C0:T6:L0" (UP) command 0xa3 failed with status Timeout. H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2017-04-18T17:50:28.183Z cpu11:117253905)NMP: nmp_ThrottleLogForDevice:2349: Cmd 0x89 (0x413683ad8cc0, 32824) to dev "naa.6001405e7e34bd545064ac286fee4e07" on path "vmhba32:C0:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x5 0x25 0x0. Act:EVAL 2017-04-18T17:50:29.054Z cpu11:112429340)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6001405e7e34bd545064ac286fee4e07" state in doubt; requested fast path state update... 2017-04-18T17:50:29.165Z cpu11:86245289)NMP: nmp_ThrottleLogForDevice:2349: Cmd 0x89 (0x413683ad8cc0, 32824) to dev "naa.6001405e7e34bd545064ac286fee4e07" on path "vmhba32:C0:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x5 0x25 0x0. Act:EVAL 2017-04-18T17:50:30.896Z cpu8:33518)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x4109d7a54110 network resource pool netsched.pools.persist.iscsi associated 2017-04-18T17:50:30.896Z cpu8:33518)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x4109d7a54110 network tracker id 15 tracker.iSCSI.10.15.24.65 associated 2017-04-18T17:50:30.987Z cpu12:86245285)NMP: nmp_ThrottleLogForDevice:2349: Cmd 0x28 (0x413684164580, 32823) to dev "naa.6001405e7e34bd545064ac286fee4e07" on path "vmhba32:C0:T6:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL 2017-04-18T17:50:31.059Z cpu12:112429340)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6001405e7e34bd545064ac286fee4e07" state in doubt; requested fast path state update... Software: Ubuntu Server 16.04 kernel 4.4.0-62-generic backstore: iblock (LVM lv on HW raid) Hardware: LSI Megaraid 9271-4i Supermicro X9DR3-F 64G ECC RAM Intel Corporation 82599ES 10G network The storage server is still in the "failed" state. I can provide logs, remote access or anything else that might help resolve this issue. Other people with this problem: https://communities.vmware.com/thread/543819 -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
