Thanks for your advise first.
i think the reason why the attached device' capacity drop to 0 is:
the RESD_CAPCITY command failed because the ASC_REPORTED_LUNS_DATA_HAS_CHANGED sense code.
just like the reason why TEST_UNIT_READY failed. (they both called in sd_reavalidate_disk).
what i want know is, the difference between IET and TGT about the ASC_REPORTED_LUNS_DATA_HAS_CHANGED, as what i
mentioned :
>>>> in tgt, i found that, when a lun is created or destroyed, ASC_REPORTED_LUNS_DATA_HAS_CHANGED was add to I_T by
>>>> ua_sense_add(), except for command REPORT LUN,
>>>> in IET, i do not find the same program logic,
can u or someone give me some hint.
At 2014-09-24 03:48:00, "ronnie sahlberg" <ronniesahlberg@xxxxxxxxx> wrote:
>On Tue, Sep 23, 2014 at 1:31 AM, 席智勇 <xizhiyong18@xxxxxxx> wrote:
>>
>> in tgt, i found that, when a lun is created or destroyed, ASC_REPORTED_LUNS_DATA_HAS_CHANGED was add to I_T by
>> ua_sense_add(), except for command REPORT LUN,
>> in IET, i do not find the same program logic,
>> is this the reason why i met the problem when i using tgt, but not when IET？
>
>
>That is very likely and probably why you see this.
>It is odd that your initiator appears to temporarily drop the capacity
>to 0 on this
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.752462] sdd: detected capacity change from 1073807360 to 0
>until it has re-issued a new READCAPACITY to re-learn the correct
>capacity again.
>
>I would have imagined that an initiator would have kept the old data
>until the point it could verify whether the device had actually
>changed or not.
>But maybe they have reason for this? But it sounds somewhat odd
>initiator behaviour.
>
>
>For device startup, when you start tgtd and there are ways to stop the
>initiator from being able to access any LUNs at all until you have
>finished setting up everything in TGTD.
>
>If you use tgtadm to set up the target and all the LUNs, then if you
>change your scripts so that you create all the LUNs and all the
>configuration before allowing any initiators to connect,
>then I think your initiators will not see these unit attentions.
>
>I.e. change the scripts to wait doing this line at the very end when
>LUN creation has finished :
> # Allow ALL initiators to connect to this target
> tgtadm --lld iscsi --mode target --op bind --tid 1 --initiator-address ALL
>
>
>I do not know the corresponding way to do this with tgt-admin since I
>never use that tool :-(
>
>
>If this only happens when you start the tgt service and you use
>/etc/init.d/ scripts from your system, you can check in which order
>the script does these things.
>I never use the scripts that come from the distros so I don;t know the
>exact details in which order they do things.
>
>
>regards
>ronnie sahlberg
>
>
>>
>>
>> At 2014-09-22 20:23:27, "席智勇" <xizhiyong18@xxxxxxx> wrote:
>>>may you give some advise to me about the problem report by me.
>>>
>>>-------- Forwarding messages --------
>>>From: "席智勇" <xizhiyong18@xxxxxxx>
>>>Date: 2014-09-22 03:29:51
>>>To: "席智勇" <xizhiyong18@xxxxxxx>
>>>Cc: "ronnie sahlberg" <ronniesahlberg@xxxxxxxxx>,stgt@xxxxxxxxxxxxxxx
>>>Subject: Reply:Reply:Re: attached devices exported by tgt not oprational
>>>from the syslog of the initiator side, it seams the sd_revalide_disk was trigged more
>>>than once, and it succeed first time, when logged "sd 7:0:0:4: [sdf] Attached SCSI disk",
>>>i don't when and where the sd_revalidate was called again, and it failed when send SCSI
>>>command TEST_UNIT_READ and RRAD_CAPACITY.
>>>
>>>
>>>
>>>At 2014-09-11 11:35:57, "席智勇" <xizhiyong18@xxxxxxx> wrote:
>>>>first of all, thanks for your reply.
>>>>in my program, it will use 128 LUNS at the most each target, so LUNS id will
>>>>not outnumber 255.
>>>>
>>>>here is some syslog when the problem occur. you can search [sdf] to find some
>>>>info, like "[sdf] READ CAPACITY(16) failed" or "[sdf] Unit Not Ready", and "sdf: detected capacity change from 9664266240 to 0",
>>>>"Sense Key : Illegal Request [current]", "Add. Sense: Logical unit not supported"
>>>>------------------------------------------------------------------------------------------
>>>>
>>>>#:~/tgt_debug$ cat syslog |grep -C 1 -e "sdf\|sd 7:0:0:4"
>>>>
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.426520] sd 7:0:0:4: [sdf] 18875520 512-byte logical blocks: (9.66 GB/9.00 GiB)
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.426629] sd 7:0:0:3: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>>>>--
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.427236] sd 7:0:0:5: Attached scsi generic sg12 type 0
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.427409] sd 7:0:0:4: [sdf] Write Protect is off
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.427414] sd 7:0:0:4: [sdf] Mode Sense: 69 00 00 08
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.427545] sd 7:0:0:5: [sdg] 2097280 512-byte logical blocks: (1.07 GB/1.00 GiB)
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.427875] sd 7:0:0:4: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.427926] scsi 7:0:0:7: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
>>>>--
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.443681] sd 7:0:0:5: [sdg] Attached SCSI disk
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.458714] sdf: unknown partition table
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.460786] sd 7:0:0:4: [sdf] Attached SCSI disk
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.476316] sdh: unknown partition table
>>>>--
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.701790] Add. Sense: Reported luns data has changed
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.703810] sd 7:0:0:4: [sdf] Unit Not Ready
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.703814] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.703815] Sense Key : Unit Attention [current]
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.703818] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:25 10-120-120-35 kernel: [1144399.703821] Add. Sense: Reported luns data has changed
>>>>--
>>>>Aug 26 16:02:26 10-120-120-35 kernel: [1144400.336478] Add. Sense: Reported luns data has changed
>>>>Aug 26 16:02:26 10-120-120-35 kernel: [1144400.338547] sd 7:0:0:4: [sdf] Unit Not Ready
>>>>Aug 26 16:02:26 10-120-120-35 kernel: [1144400.338552] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:26 10-120-120-35 kernel: [1144400.338554] Sense Key : Unit Attention [current]
>>>>Aug 26 16:02:26 10-120-120-35 kernel: [1144400.338558] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:26 10-120-120-35 kernel: [1144400.338561] Add. Sense: Reported luns data has changed
>>>>--
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.752462] sdd: detected capacity change from 1073807360 to 0
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.752987] sd 7:0:0:4: [sdf] Unit Not Ready
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.752990] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.752991] Sense Key : Unit Attention [current]
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.752994] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.752997] Add. Sense: Reported luns data has changed
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753254] sd 7:0:0:4: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753472] sd 7:0:0:4: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753655] sd 7:0:0:4: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753666] sd 7:0:0:4: [sdf] READ CAPACITY(16) failed
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753672] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753674] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753676] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753677] Sense Key : Unit Attention [current]
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753680] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753682] Add. Sense: Reported luns data has changed
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.753883] sd 7:0:0:4: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754075] sd 7:0:0:4: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754275] sd 7:0:0:4: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754286] sd 7:0:0:4: [sdf] READ CAPACITY failed
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754291] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754292] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754295] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754296] Sense Key : Unit Attention [current]
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754299] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.754301] Add. Sense: Reported luns data has changed
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.756150] sdf: detected capacity change from 9664266240 to 0
>>>>Aug 26 16:02:28 10-120-120-35 kernel: [1144402.756639] sd 7:0:0:5: [sdg] Unit Not Ready
>>>>--
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144402.925588] scsi 3:0:0:21: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144402.925959] sd 7:0:0:4: [sdf] 18875520 512-byte logical blocks: (9.66 GB/9.00 GiB)
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144402.926000] sd 3:0:0:21: Attached scsi generic sg59 type 0
>>>>--
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144402.927749] sd 3:0:0:22: [sdbc] 18875520 512-byte logical blocks: (9.66 GB/9.00 GiB)
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144402.927762] sdf: detected capacity change from 0 to 9664266240
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144402.927843] scsi 3:0:0:23: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
>>>>--
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.191743] Add. Sense: Reported luns data has changed
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.195776] sd 7:0:0:4: [sdf] Unit Not Ready
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.195779] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.195780] Sense Key : Unit Attention [current]
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.195784] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.195786] Add. Sense: Reported luns data has changed
>>>>--
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.592427] sd 7:0:0:39: [sdbx] Mode Sense: 69 00 00 08
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.592809] sd 7:0:0:40: Attached scsi generic sg82 type 0
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.594486] sd 7:0:0:39: [sdbx] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.594696] sd 7:0:0:40: [sdby] 18875520 512-byte logical blocks: (9.66 GB/9.00 GiB)
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.595825] sd 7:0:0:40: [sdby] Write Protect is off
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.595831] sd 7:0:0:40: [sdby] Mode Sense: 69 00 00 08
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.596261] sd 7:0:0:40: [sdby] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.605687] sdbx: unknown partition table
>>>>--
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.610455] sd 7:0:0:39: [sdbx] Attached SCSI disk
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.613057] sd 7:0:0:40: [sdby] Attached SCSI disk
>>>>Aug 26 16:02:29 10-120-120-35 kernel: [1144403.617631] sd 3:0:0:13: [sdaf] Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.
>>>>--
>>>>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.432518] md113: unknown partition table
>>>>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.662356] sd 7:0:0:4: [sdf] Synchronizing SCSI cache
>>>>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.663163] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.663167] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>>>>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.663170] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.663172] Sense Key : Illegal Request [current]
>>>>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.663175] sd 7:0:0:4: [sdf]
>>>>Aug 26 16:02:33 10-120-120-35 kernel: [1144407.663178] Add. Sense: Logical unit not supported
>>>>
>>>>
>>>>At 2014-09-11 02:21:59, "ronnie sahlberg" <ronniesahlberg@xxxxxxxxx> wrote:
>>>>>When you say "many devices" exactly how many are you referring to?
>>>>>Can you check if this problem you have only occur when you go from 255
>>>>>LUNs to something higher?
>>>>>
>>>>>
>>>>>The default LUN addressing scheme only allows LUNs numbered 0 to 255.
>>>>>To go to higher LUN numbers, which TGTD supports, doe require that you
>>>>>switch to a different addressing format
>>>>>and it may be that your initiator can not handle those other modes.
>>>>>
>>>>>Often it might look like you have LUNs 0-255 and then there is a jump
>>>>>to the next LUN which has the number 16640 if the initiator can not
>>>>>handle addressing modes properly.
>>>>>
>>>>>
>>>>>I do not know how IET does the LUN numbering.
>>>>>
>>>>>If you can collect a wireshark trace of iscsiadm --login from both
>>>>>STGT and IET then I can take a quick look and see if that is the issue
>>>>>you are facing.
>>>>>
>>>>>
>>>>>
>>>>>regards
>>>>>ronnie sahlberg
>>>>>
>>>>>
>>>>>On Mon, Sep 1, 2014 at 11:00 PM, 席智勇 <xizhiyong18@xxxxxxx> wrote:
>>>>>>
>>>>>> when i export many block device by tgt , in one single target , this means many exported lun in one target, to another
>>>>>> machine by iscsi。
>>>>>>
>>>>>> in another machine , attach the devices by open-iscsi。when i wait the
>>>>>> size of the device attached change from 0 to correct size， i create raid1 with two devices by mdadm。at this time, mdadm report like
>>>>>> "/dev/sdem is not suitable for this array.", i read the code of mdadm ,that because it cannot open the device。sometime it even report
>>>>>> "failed to open /dev/** after earlier success - aborting "。
>>>>>>
>>>>>> i do not know whether it's because i used the device when it's not ready yet, or the problem of tgt。
>>>>>>
>>>>>> all the above said happened when many target lun(tgt) created at the same
>>>>>> time concurrently。it do not happen in nomal time. when i change tgt to iet, it
>>>>>> do not happen either。
>>>>>>
>>>>>>
>>>>>>
>>>>>> tgt version: 1.0.48
>>>>>>
>>>>>> open-iscsi version:iscsiadm version 2.0-873
>>>>>>
>>>>>> linux system: Linux 3.10.40-amd64 GNU/Linux
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -------------
>>>>>>
>>>>>> Zhiyong Xi
?韬{.n?壏煯壄?%娝?檩?w?{.n?壏 ?{ay?蕠跈?jf＂穐殢飦?戧鐉_璁(殠娸"濟?m??G珴?⒏?櫒璀?ｘ忈
