from the syslog of the initiator side, it seams the sd_revalide_disk was trigged more
than once, and it succeed first time, when logged "sd 7:0:0:4: [sdf] Attached SCSI disk",
i don't when and where the sd_revalidate was called again, and it failed when send SCSI
command TEST_UNIT_READ and RRAD_CAPACITY.
At 2014-09-11 11:35:57, "席智勇" <xizhiyong18@xxxxxxx> wrote:
>first of all, thanks for your reply.
>in my program, it will use 128 LUNS at the most each target, so LUNS id will
>not outnumber 255.
>
>here is some syslog when the problem occur. you can search [sdf] to find some
>info, like "[sdf] READ CAPACITY(16) failed" or "[sdf] Unit Not Ready", and "sdf: detected capacity change from 9664266240 to 0",
>"Sense Key : Illegal Request [current]", "Add. Sense: Logical unit not supported"
>------------------------------------------------------------------------------------------
>
>#:~/tgt_debug$ cat syslog |grep -C 1 -e "sdf\|sd 7:0:0:4"
>
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.426520] sd 7:0:0:4: [sdf] 18875520 512-byte logical blocks: (9.66 GB/9.00 GiB)
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.426629] sd 7:0:0:3: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>--
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.427236] sd 7:0:0:5: Attached scsi generic sg12 type 0
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.427409] sd 7:0:0:4: [sdf] Write Protect is off
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.427414] sd 7:0:0:4: [sdf] Mode Sense: 69 00 00 08
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.427545] sd 7:0:0:5: [sdg] 2097280 512-byte logical blocks: (1.07 GB/1.00 GiB)
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.427875] sd 7:0:0:4: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.427926] scsi 7:0:0:7: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
>--
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.443681] sd 7:0:0:5: [sdg] Attached SCSI disk
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.458714] sdf: unknown partition table
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.460786] sd 7:0:0:4: [sdf] Attached SCSI disk
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.476316] sdh: unknown partition table
>--
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.701790] Add. Sense: Reported luns data has changed
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.703810] sd 7:0:0:4: [sdf] Unit Not Ready
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.703814] sd 7:0:0:4: [sdf]
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.703815] Sense Key : Unit Attention [current]
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.703818] sd 7:0:0:4: [sdf]
>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.703821] Add. Sense: Reported luns data has changed
>--
>Aug 26 16:02:26 10-120-120-35 kernel: [1144400.336478] Add. Sense: Reported luns data has changed
>Aug 26 16:02:26 10-120-120-35 kernel: [1144400.338547] sd 7:0:0:4: [sdf] Unit Not Ready
>Aug 26 16:02:26 10-120-120-35 kernel: [1144400.338552] sd 7:0:0:4: [sdf]
>Aug 26 16:02:26 10-120-120-35 kernel: [1144400.338554] Sense Key : Unit Attention [current]
>Aug 26 16:02:26 10-120-120-35 kernel: [1144400.338558] sd 7:0:0:4: [sdf]
>Aug 26 16:02:26 10-120-120-35 kernel: [1144400.338561] Add. Sense: Reported luns data has changed
>--
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.752462] sdd: detected capacity change from 1073807360 to 0
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.752987] sd 7:0:0:4: [sdf] Unit Not Ready
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.752990] sd 7:0:0:4: [sdf]
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.752991] Sense Key : Unit Attention [current]
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.752994] sd 7:0:0:4: [sdf]
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.752997] Add. Sense: Reported luns data has changed
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753254] sd 7:0:0:4: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753472] sd 7:0:0:4: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753655] sd 7:0:0:4: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753666] sd 7:0:0:4: [sdf] READ CAPACITY(16) failed
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753672] sd 7:0:0:4: [sdf]
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753674] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753676] sd 7:0:0:4: [sdf]
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753677] Sense Key : Unit Attention [current]
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753680] sd 7:0:0:4: [sdf]
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753682] Add. Sense: Reported luns data has changed
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753883] sd 7:0:0:4: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754075] sd 7:0:0:4: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754275] sd 7:0:0:4: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754286] sd 7:0:0:4: [sdf] READ CAPACITY failed
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754291] sd 7:0:0:4: [sdf]
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754292] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754295] sd 7:0:0:4: [sdf]
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754296] Sense Key : Unit Attention [current]
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754299] sd 7:0:0:4: [sdf]
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754301] Add. Sense: Reported luns data has changed
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.756150] sdf: detected capacity change from 9664266240 to 0
>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.756639] sd 7:0:0:5: [sdg] Unit Not Ready
>--
>Aug 26 16:02:29 10-120-120-35 kernel: [1144402.925588] scsi 3:0:0:21: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
>Aug 26 16:02:29 10-120-120-35 kernel: [1144402.925959] sd 7:0:0:4: [sdf] 18875520 512-byte logical blocks: (9.66 GB/9.00 GiB)
>Aug 26 16:02:29 10-120-120-35 kernel: [1144402.926000] sd 3:0:0:21: Attached scsi generic sg59 type 0
>--
>Aug 26 16:02:29 10-120-120-35 kernel: [1144402.927749] sd 3:0:0:22: [sdbc] 18875520 512-byte logical blocks: (9.66 GB/9.00 GiB)
>Aug 26 16:02:29 10-120-120-35 kernel: [1144402.927762] sdf: detected capacity change from 0 to 9664266240
>Aug 26 16:02:29 10-120-120-35 kernel: [1144402.927843] scsi 3:0:0:23: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
>--
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.191743] Add. Sense: Reported luns data has changed
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.195776] sd 7:0:0:4: [sdf] Unit Not Ready
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.195779] sd 7:0:0:4: [sdf]
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.195780] Sense Key : Unit Attention [current]
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.195784] sd 7:0:0:4: [sdf]
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.195786] Add. Sense: Reported luns data has changed
>--
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.592427] sd 7:0:0:39: [sdbx] Mode Sense: 69 00 00 08
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.592809] sd 7:0:0:40: Attached scsi generic sg82 type 0
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.594486] sd 7:0:0:39: [sdbx] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.594696] sd 7:0:0:40: [sdby] 18875520 512-byte logical blocks: (9.66 GB/9.00 GiB)
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.595825] sd 7:0:0:40: [sdby] Write Protect is off
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.595831] sd 7:0:0:40: [sdby] Mode Sense: 69 00 00 08
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.596261] sd 7:0:0:40: [sdby] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.605687] sdbx: unknown partition table
>--
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.610455] sd 7:0:0:39: [sdbx] Attached SCSI disk
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.613057] sd 7:0:0:40: [sdby] Attached SCSI disk
>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.617631] sd 3:0:0:13: [sdaf] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
>--
>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.432518] md113: unknown partition table
>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.662356] sd 7:0:0:4: [sdf] Synchronizing SCSI cache
>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.663163] sd 7:0:0:4: [sdf]
>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.663167] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.663170] sd 7:0:0:4: [sdf]
>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.663172] Sense Key : Illegal Request [current]
>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.663175] sd 7:0:0:4: [sdf]
>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.663178] Add. Sense: Logical unit not supported
>
>
>At 2014-09-11 02:21:59, "ronnie sahlberg" <ronniesahlberg@xxxxxxxxx> wrote:
>>When you say "many devices" exactly how many are you referring to?
>>Can you check if this problem you have only occur when you go from 255
>>LUNs to something higher?
>>
>>
>>The default LUN addressing scheme only allows LUNs numbered 0 to 255.
>>To go to higher LUN numbers, which TGTD supports, doe require that you
>>switch to a different addressing format
>>and it may be that your initiator can not handle those other modes.
>>
>>Often it might look like you have LUNs 0-255 and then there is a jump
>>to the next LUN which has the number 16640 if the initiator can not
>>handle addressing modes properly.
>>
>>
>>I do not know how IET does the LUN numbering.
>>
>>If you can collect a wireshark trace of iscsiadm --login from both
>>STGT and IET then I can take a quick look and see if that is the issue
>>you are facing.
>>
>>
>>
>>regards
>>ronnie sahlberg
>>
>>
>>On Mon, Sep 1, 2014 at 11:00 PM, 席智勇 <xizhiyong18@xxxxxxx> wrote:
>>>
>>> when i export many block device by tgt , in one single target , this means many exported lun in one target, to another
>>> machine by iscsi。
>>>
>>> in another machine , attach the devices by open-iscsi。when i wait the
>>> size of the device attached change from 0 to correct size， i create raid1 with two devices by mdadm。at this time, mdadm report like
>>> "/dev/sdem is not suitable for this array.", i read the code of mdadm ,that because it cannot open the device。sometime it even report
>>> "failed to open /dev/** after earlier success - aborting "。
>>>
>>> i do not know whether it's because i used the device when it's not ready yet, or the problem of tgt。
>>>
>>> all the above said happened when many target lun(tgt) created at the same
>>> time concurrently。it do not happen in nomal time. when i change tgt to iet, it
>>> do not happen either。
>>>
>>>
>>>
>>> tgt version: 1.0.48
>>>
>>> open-iscsi version:iscsiadm version 2.0-873
>>>
>>> linux system: Linux 3.10.40-amd64 GNU/Linux
>>>
>>>
>>>
>>>
>>>
>>> -------------
>>>
>>> Zhiyong Xi
?韬{.n?壏煯壄?%娝?檩?w?{.n?壏 ?{ay?蕠跈?jf＂穐殢飦?戧鐉_璁(殠娸"濟?m??G珴?⒏?櫒璀?ｘ忈
