Please keep me in CC, I am not subscribed.
I've been using the iscsi target for a little while to provide extra storage
to 2 of our VMware hosts. Every so often, both of these hosts hang (vms
living on local storage continue to work) until the target is restarted.
Once I restart the target the hosts are no longer hung.
I installed debian testing (stretch) on the target host due to issues with
getting targetcli installed from the stable release.
Any suggestions?
Kernel is:
Linux it-iscsi 4.6.0-1-amd64 #1 SMP Debian 4.6.2-2 (2016-06-25) x86_64 GNU/Linux
The network connections to the target is like this:
host 1: 10G direct attach sfp+ using a dual port intel nic.
host 2: 1G on a seperate vlan using intel LOM.
Lspci for the nics:
03:00.0 0200: 8086:10fb (rev 01)
Subsystem: 8086:0003
Flags: bus master, fast devsel, latency 0, IRQ 50
Memory at fae80000 (64-bit, prefetchable) [size=512K]
I/O ports at e020 [disabled] [size=32]
Memory at faf04000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 90-e2-ba-ff-ff-85-5e-24
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Kernel driver in use: ixgbe
Kernel modules: ixgbe
03:00.1 0200: 8086:10fb (rev 01)
Subsystem: 8086:0003
Flags: bus master, fast devsel, latency 0, IRQ 59
Memory at fae00000 (64-bit, prefetchable) [size=512K]
I/O ports at e000 [disabled] [size=32]
Memory at faf00000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 90-e2-ba-ff-ff-85-5e-24
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Kernel driver in use: ixgbe
Kernel modules: ixgbe
05:00.0 0200: 8086:1521 (rev 01)
Subsystem: 15d9:1521
Flags: bus master, fast devsel, latency 0, IRQ 29
Memory at fbe20000 (32-bit, non-prefetchable) [size=128K]
Memory at fbec4000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-25-90-ff-ff-7c-c7-c6
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1a0] Transaction Processing Hints
Capabilities: [1c0] Latency Tolerance Reporting
Capabilities: [1d0] Access Control Services
Kernel driver in use: igb
Kernel modules: igb
05:00.1 0200: 8086:1521 (rev 01)
Subsystem: 15d9:1521
Flags: bus master, fast devsel, latency 0, IRQ 39
Memory at fbe00000 (32-bit, non-prefetchable) [size=128K]
Memory at fbec0000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=10 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-25-90-ff-ff-7c-c7-c6
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Capabilities: [1a0] Transaction Processing Hints
Capabilities: [1d0] Access Control Services
Kernel driver in use: igb
Kernel modules: igb
I see lots of logs that look like this:
Feb 14 02:07:11 it-iscsi kernel: [10758736.997587] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000054
I see those after I restart the target as well.
I also see entries like this:
Feb 13 08:21:50 it-iscsi kernel: [10694810.011861] ABORT_TASK: Found referenced iSCSI task_tag: 43134935
Feb 13 08:21:50 it-iscsi kernel: [10694810.025100] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 43134935
Feb 13 08:21:50 it-iscsi kernel: [10694810.143353] Detected MISCOMPARE for addr: ffff880859000000 buf: ffff880856354200
Feb 13 08:21:50 it-iscsi kernel: [10694810.156591] Target/iblock: Send MISCOMPARE check condition and sense
Feb 13 08:21:54 it-iscsi kernel: [10694813.810799] Detected MISCOMPARE for addr: ffff88080415e000 buf: ffff8808051bea00
Feb 13 08:21:54 it-iscsi kernel: [10694813.824048] Target/iblock: Send MISCOMPARE check condition and sense
Feb 13 08:21:54 it-iscsi kernel: [10694813.875094] Detected MISCOMPARE for addr: ffff88080441d000 buf: ffff88080478e000
Feb 13 08:21:54 it-iscsi kernel: [10694813.888400] Target/iblock: Send MISCOMPARE check condition and sense
Feb 13 10:22:20 it-iscsi kernel: [10702041.122420] ABORT_TASK: Found referenced iSCSI task_tag: 43960759
Feb 13 10:22:20 it-iscsi kernel: [10702041.135568] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 43960759
Feb 13 10:22:21 it-iscsi kernel: [10702041.254315] Detected MISCOMPARE for addr: ffff88085821e000 buf: ffff880803c55c00
Feb 13 10:22:21 it-iscsi kernel: [10702041.267558] Target/iblock: Send MISCOMPARE check condition and sense
Feb 13 12:32:12 it-iscsi kernel: [10709833.321628] ABORT_TASK: Found referenced iSCSI task_tag: 44472179
Feb 13 12:32:12 it-iscsi kernel: [10709833.334913] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 44472179
Feb 13 12:32:12 it-iscsi kernel: [10709833.469175] Detected MISCOMPARE for addr: ffff880804dff000 buf: ffff880804dbc000
Feb 13 12:32:12 it-iscsi kernel: [10709833.482611] Target/iblock: Send MISCOMPARE check condition and sense
Feb 13 12:32:24 it-iscsi kernel: [10709845.323068] ABORT_TASK: Found referenced iSCSI task_tag: 44479846
Feb 13 12:32:24 it-iscsi kernel: [10709845.336423] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 44479846
Feb 13 12:32:24 it-iscsi kernel: [10709845.446280] Detected MISCOMPARE for addr: ffff880803e8f000 buf: ffff88080532ba00
Feb 13 12:32:24 it-iscsi kernel: [10709845.459774] Target/iblock: Send MISCOMPARE check condition and sense
And some of these
Feb 13 20:32:28 it-iscsi kernel: [10738652.384443] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 194748343
Feb 13 20:32:56 it-iscsi kernel: [10738679.855002] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 194748354
Feb 13 20:33:50 it-iscsi kernel: [10738734.247422] ABORT_TASK: Found referenced iSCSI task_tag: 194748375
Feb 13 20:33:50 it-iscsi kernel: [10738734.260882] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 194748375
Feb 13 20:44:32 it-iscsi kernel: [10739376.273769] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 194748609
Feb 13 20:44:59 it-iscsi kernel: [10739403.514156] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 194748618
Feb 13 21:03:43 it-iscsi kernel: [10740526.956467] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 48147818
Feb 13 21:07:33 it-iscsi kernel: [10740757.309878] Exiting Time2Retain handler because session_reinstatement=1
Feb 13 21:12:06 it-iscsi kernel: [10741030.387219] Exiting Time2Retain handler because session_reinstatement=1
target config (not modified)
storage iblock {
disk esx_cds {
path /dev/itiscsidatalvm/esx_cds
wwn b23cb921-0e10-4ad3-8569-61aac4fa9412
attribute {
block_size 512
emulate_3pc no
emulate_caw no
emulate_dpo yes
emulate_fua_read yes
emulate_fua_write no
emulate_model_alias no
emulate_rest_reord no
emulate_tas no
emulate_tpu no
emulate_tpws no
emulate_ua_intlck_ctrl no
emulate_write_cache no
enforce_pr_isids yes
fabric_max_sectors 8192
force_pr_aptpl 0
is_nonrot no
max_unmap_block_desc_count 0
max_unmap_lba_count 0
max_write_same_len 65535
optimal_sectors 640
pi_prot_format 0
pi_prot_type 0
queue_depth 128
unmap_granularity 0
unmap_granularity_alignment 0
unmap_zeroes_data 0
}
}
disk esx_ds1 {
path /dev/itiscsidatalvm/esx_ds1
wwn 71c5020b-f610-4b24-81de-7cf0dec214b9
attribute {
block_size 512
emulate_3pc yes
emulate_caw yes
emulate_dpo yes
emulate_fua_read yes
emulate_fua_write yes
emulate_model_alias no
emulate_rest_reord no
emulate_tas yes
emulate_tpu no
emulate_tpws no
emulate_ua_intlck_ctrl no
emulate_write_cache no
enforce_pr_isids yes
fabric_max_sectors 8192
force_pr_aptpl 0
is_nonrot no
max_unmap_block_desc_count 0
max_unmap_lba_count 0
max_write_same_len 65535
optimal_sectors 640
pi_prot_format 0
pi_prot_type 0
queue_depth 128
unmap_granularity 0
unmap_granularity_alignment 0
unmap_zeroes_data 0
}
}
disk itbackup-archive1 {
path /dev/itiscsidatalvm/itbackup-archive1
wwn 777b4bab-1cf5-46ab-aef4-27bc4131ee74
attribute {
block_size 512
emulate_3pc yes
emulate_caw yes
emulate_dpo yes
emulate_fua_read yes
emulate_fua_write yes
emulate_model_alias no
emulate_rest_reord no
emulate_tas yes
emulate_tpu no
emulate_tpws no
emulate_ua_intlck_ctrl no
emulate_write_cache no
enforce_pr_isids yes
fabric_max_sectors 8192
force_pr_aptpl 0
is_nonrot no
max_unmap_block_desc_count 0
max_unmap_lba_count 0
max_write_same_len 65535
optimal_sectors 640
pi_prot_format 0
pi_prot_type 0
queue_depth 128
unmap_granularity 0
unmap_granularity_alignment 0
unmap_zeroes_data 0
}
}
disk itbackup-repo2 {
path /dev/itiscsidatalvm/itbackup-repo2
wwn 4ea1db71-5e24-40d6-9662-69d479fc482e
attribute {
block_size 512
emulate_3pc yes
emulate_caw yes
emulate_dpo yes
emulate_fua_read yes
emulate_fua_write yes
emulate_model_alias no
emulate_rest_reord no
emulate_tas yes
emulate_tpu no
emulate_tpws no
emulate_ua_intlck_ctrl no
emulate_write_cache no
enforce_pr_isids yes
fabric_max_sectors 8192
force_pr_aptpl 0
is_nonrot no
max_unmap_block_desc_count 0
max_unmap_lba_count 0
max_write_same_len 65535
optimal_sectors 640
pi_prot_format 0
pi_prot_type 0
queue_depth 128
unmap_granularity 0
unmap_granularity_alignment 0
unmap_zeroes_data 0
}
}
}
fabric iscsi {
discovery_auth {
enable no
mutual_password ""
mutual_userid ""
password ""
userid ""
}
target iqn.1998-03.com.electro-mechanical:esx-ds tpgt 1 {
enable yes
attribute {
authentication no
cache_dynamic_acls no
default_cmdsn_depth 64
default_erl 0
demo_mode_discovery yes
demo_mode_write_protect yes
fabric_prot_type 0
generate_node_acls no
login_timeout 15
netif_timeout 2
prod_mode_write_protect no
t10_pi 0
tpg_enabled_sendtargets 1
}
auth {
password ""
password_mutual ""
userid ""
userid_mutual ""
}
parameter {
AuthMethod "CHAP,None"
DataDigest "CRC32C,None"
DataPDUInOrder yes
DataSequenceInOrder yes
DefaultTime2Retain 20
DefaultTime2Wait 2
ErrorRecoveryLevel no
FirstBurstLength 65536
HeaderDigest "CRC32C,None"
IFMarkInt Reject
IFMarker no
ImmediateData yes
InitialR2T yes
MaxBurstLength 262144
MaxConnections 1
MaxOutstandingR2T 1
MaxRecvDataSegmentLength 8192
MaxXmitDataSegmentLength 262144
OFMarkInt Reject
OFMarker no
TargetAlias "LIO Target"
}
lun 0 backend iblock:esx_ds1
acl iqn.1998-01.com.vmware:emc-esx1-37f354d6 {
attribute {
dataout_timeout 3
dataout_timeout_retries 5
default_erl 0
nopin_response_timeout 30
nopin_timeout 15
random_datain_pdu_offsets no
random_datain_seq_offsets no
random_r2t_offsets no
}
auth {
password ""
password_mutual ""
userid ""
userid_mutual ""
}
mapped_lun 0 {
target_lun 0
write_protect no
}
}
acl iqn.1998-01.com.vmware:it-vsphere-3c3433a7 {
attribute {
dataout_timeout 3
dataout_timeout_retries 5
default_erl 0
nopin_response_timeout 30
nopin_timeout 15
random_datain_pdu_offsets no
random_datain_seq_offsets no
random_r2t_offsets no
}
auth {
password ""
password_mutual ""
userid ""
userid_mutual ""
}
mapped_lun 0 {
target_lun 0
write_protect no
}
}
portal 10.255.0.1:3260
portal 10.255.1.1:3260
}
target iqn.1998-03.com.electro-mechanical:itbackup-repos tpgt 1 {
enable yes
attribute {
authentication no
cache_dynamic_acls no
default_cmdsn_depth 64
default_erl 0
demo_mode_discovery yes
demo_mode_write_protect yes
fabric_prot_type 0
generate_node_acls no
login_timeout 15
netif_timeout 2
prod_mode_write_protect no
t10_pi 0
tpg_enabled_sendtargets 1
}
auth {
password ""
password_mutual ""
userid ""
userid_mutual ""
}
parameter {
AuthMethod "CHAP,None"
DataDigest "CRC32C,None"
DataPDUInOrder yes
DataSequenceInOrder yes
DefaultTime2Retain 20
DefaultTime2Wait 2
ErrorRecoveryLevel no
FirstBurstLength 65536
HeaderDigest "CRC32C,None"
IFMarkInt Reject
IFMarker no
ImmediateData yes
InitialR2T yes
MaxBurstLength 262144
MaxConnections 1
MaxOutstandingR2T 1
MaxRecvDataSegmentLength 8192
MaxXmitDataSegmentLength 262144
OFMarkInt Reject
OFMarker no
TargetAlias "LIO Target"
}
lun 0 backend iblock:itbackup-repo2
acl iqn.1991-05.com.microsoft:it-backup.electro-mechanical.com {
attribute {
dataout_timeout 3
dataout_timeout_retries 5
default_erl 0
nopin_response_timeout 30
nopin_timeout 15
random_datain_pdu_offsets no
random_datain_seq_offsets no
random_r2t_offsets no
}
auth {
password ""
password_mutual ""
userid ""
userid_mutual ""
}
mapped_lun 0 {
target_lun 0
write_protect no
}
}
portal 10.255.1.1:3260
}
}
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
