On Wed, 2012-05-30 at 15:37 +0200, Florian Heigl wrote:
> Hi,
>
Hello Florian,
>
> I noticed that when I add two LIO based iSCSI storages to a Linux box
> running Veritas volume manager that it will think all LIO targets are
> "the same" enclosure.
>
> An enclosure is detected by the multipath drivers in VxVM. I think they
> use some sort of scsi serial number or something to identify that.
> (I know there's API drivers that are "certified" but thats for extra
> features)
>
Thanks for reporting the issue with Vertias VxVM.
> So, anyway.
> I would have thought the "array" presented by system "Sechs" should end
> up with a different name than the "array" presented by my laptop.
>
>
Can you please share the top-level rtsadmin/targetcli shell output of
your setup to give an idea of how things are currently configured..?
>
> Rationale & Steps:
> ------------------
> I needed to provide storage for a lab and used LIO on Fedora 16 for that.
OK, you'll want to make sure to use at least 3.2.18 or 3.3.7 stable
kernels on FC16.
> I basically made one "storage box" serving via iSCSI, with just 6 1GB luns.
> On the client (Centos6.2) I've installed Veritas SF basic and everything
> worked OK.
>
> I can see an "enclosure" for my LIO target and have made it name the
> disks by the presented LUN numbers:
>
> [root@Zwei plugins]# vxdmpadm listenclosure
> ENCLR_NAME ENCLR_TYPE ENCLR_SNO STATUS ARRAY_TYPE
> LUN_COUNT
> =======================================================================================
> other_disks OTHER_DISKS OTHER_DISKS CONNECTED
> OTHER_DISKS 1
> LIO-Sechs aluadisk ALUAdisk CONNECTED ALUA
> 9
>
>
>
> [root@Zwei plugins]# vxdisk list
> DEVICE TYPE DISK GROUP STATUS
> LIO-Sechs_0 auto:cdsdisk LIO-Sechs_0 datadg online
> LIO-Sechs_1 auto:cdsdisk LIO-Sechs_1 datadg online
> LIO-Sechs_2 auto:cdsdisk LIO-Sechs_2 datadg online
> LIO-Sechs_3 auto:none - - online invalid
> LIO-Sechs_4 auto:none - - online invalid
> LIO-Sechs_5 auto:none - - online invalid
> sda auto:none - - online invalid
>
>
>
>
> Now after scanning I get 3 new disks (6-8), but they're named after the
> enclosure "LIO-Sechs"
>
> [root@Zwei plugins]# vxdisk scandisks
> [root@Zwei plugins]# vxdisk list
> DEVICE TYPE DISK GROUP STATUS
> LIO-Sechs_0 auto:cdsdisk LIO-Sechs_0 datadg online
> LIO-Sechs_1 auto:cdsdisk LIO-Sechs_1 datadg online
> LIO-Sechs_2 auto:cdsdisk LIO-Sechs_2 datadg online
> LIO-Sechs_3 auto:none - - online invalid
> LIO-Sechs_4 auto:none - - online invalid
> LIO-Sechs_5 auto:none - - online invalid
> LIO-Sechs_6 auto - - error
> LIO-Sechs_7 auto - - error
> LIO-Sechs_8 auto - - error
> sda auto:none - - online invalid
>
>
> This is because the volume manager (or more exactly, it's alua driver)
> came to think all these LUNs come from the same array.
>
Mmmmm.. By default all Target Fabric Ports using TCM emulation go into
the default ALUA Target Port Group (ID=0). One default ALUA Target Port
Group per configured backend in /sys/kernel/config/target/$HBA/$DEV/.
That is, a backend device with multiple iscsi-target port endpoint
exports will have those fabric ports within a single group at
../$HBA/$DEV/alua/default_tg_pt_gp/ unless otherwise explictly defined.
>From configfs this looks like with multi-fabric export:
# head /sys/kernel/config/target/core/iblock_0/mpt_fusion/alua/default_tg_pt_gp/members
==> /sys/kernel/config/target/core/iblock_0/mpt_fusion/alua/default_tg_pt_gp/members <==
iSCSI/iqn.2003-01.org.linux-iscsi.debian-amd64.x8664:sn.6747a471775f/tpgt_1/lun_1
iSCSI/iqn.2003-01.org.linux-iscsi.debian-amd64.x8664:sn.1bc6fcb58f24/tpgt_1/lun_0
loopback/naa.6001405df1bafb29/tpgt_1/lun_0
vhost/naa.60014050088ae39a/tpgt_1/lun_0
the SCSI client sees the following in EVPD=0x83 for each connected target port:
# sg_inq -i /dev/sdX
<SNIP>
Designation descriptor number 4, descriptor length: 8
transport: Serial Attached SCSI (SPL)
designator_type: Target port group, code_set: Binary
associated with the target port
Target port group: 0x0
Designation descriptor number 5, descriptor length: 8
designator_type: Logical unit group, code_set: Binary
associated with the addressed logical unit
Logical unit group: 0x0
<SNIP>
You can also query the REPORT_TARGET_PORT_GROUPS for group information:
# sg_rtpg -d /dev/sdX
Report target port groups:
target port group id : 0x0 , Pref=0
target port group asymmetric access state : 0x00 (active/optimized)
T_SUP : 1, O_SUP : 1, LBD_SUP : 0, U_SUP : 1, S_SUP : 1, AN_SUP : 1, AO_SUP : 1
status code : 0x00 (no status available)
vendor unique status : 0x00
target port count : 01
Relative target port ids:
0x01
You might want to try adding a new ../$HBA/$DEV/alua/$MY_ALUA_GROUP
using 'tcm_node --addaluatpg', and making then making the explict
iscsi-target port association in:
echo $MY_ALUA_GROUP > /sys/kernel/config/target/iscsi/$TARGETNAME/$TPGT/lun/$LUN_ID/alua_tg_pt_gp
so that different target ports of the same backend device are split up
into separate ALUA target port groups..
> [root@Zwei plugins]# vxdmpadm listenclosure
> ENCLR_NAME ENCLR_TYPE ENCLR_SNO STATUS ARRAY_TYPE
> LUN_COUNT
> =======================================================================================
> other_disks OTHER_DISKS OTHER_DISKS CONNECTED
> OTHER_DISKS 1
> LIO-Sechs aluadisk ALUAdisk CONNECTED ALUA
> 9
>
> This means that a lot things can go wrong, i.e. mirrors being mis-located.
>
> My question is now if anyone else can verify this or if they know where
> LIO sets calculates it's scsi serial numbers and device description.
>
> I have already checked the serial numbers. he is for 2 consecutive luns
> from the first LIO box and 2 from the second.
> udid: LIO-ORG%5FFILEIO%5FALUAdisk%5F600140524EBE4E5D6F448398558D778E
> udid: LIO-ORG%5FFILEIO%5FALUAdisk%5F6001405A4729837F3D54BD6B467D5215
> udid: LIO-ORG%5FFILEIO%5FALUAdisk%5F600140570C836BD57C94B99BB201790F
> udid: LIO-ORG%5FFILEIO%5FALUAdisk%5F60014058123A0B0A8824DEC8469BEBAC
>
> It doesn't seem like a node id or such is encoded there.
> Any pointers on how to mess with these serials?
>
>
These are automatically set by rtslib and rtsadmin/targetcli code. It
generates a unique UUID used for Unit Serial Number (EVPD=0x80) for each
target backend device that is created. The underlying configfs attr is
at /sys/kernel/config/target/core/$HBA/$DEV/wwn/vpd_unit_serial.
TCM also uses this Unit Serial Number value to encode the maine
EVPD=0x83 designator (IEEE NAA Registered Extended) for proper fabric
independent LUN world-wide naming. You can check this value using
sg_inq -i:
# sg_inq -i /dev/sdX
VPD INQUIRY: Device Identification page
Designation descriptor number 1, descriptor length: 20
designator_type: NAA, code_set: Binary
associated with the addressed logical unit
NAA 6, IEEE Company_id: 0x1405
Vendor Specific Identifier: 0xa61454a8b
Vendor Specific Identifier Extension: 0x3db4ba8a48be1fcc
[0x6001405a61454a8b3db4ba8a48be1fcc]
Designation descriptor number 2, descriptor length: 56
designator_type: T10 vendor identification, code_set: ASCII
associated with the addressed logical unit
vendor id: LIO-ORG
vendor specific: FILEIO:a61454a8-b3db-4ba8-a48b-e1fcce88f25d
<SNIP>
Also what does lsscsi --transport look like on the iSCSI client..?
> Are there any plans for getting Veritas certification like the VMWare
> one? It might still down the road until companies replace their Vmax
> with LIO, but it can't hurt to pave the way :)
>
> (Especially since thin provisioning / reclamation is done with those
> libraries)
>
>
This is something that I think we'd be interested in at some point in
the future. (mwf CC'ed)
Thanks again for reporting!
--nab
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
