Hi. I have been looking for a solution for a while now, and found none, so this post here is my last attempt to solve the LIO issue i am encountering, before I give up and move to some other solution... Description: OS: Centos 7.1, latest updates (correct for Aug, 2015). Targetcli version: rpm -qa | grep targetcli targetcli-2.1.fb37-3.el7.noarch Kernel: uname -a Linux controller1 3.10.0-229.11.1.el7.x86_64 #1 SMP Thu Aug 6 01:06:18 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux If there's anything missing, let me know. Problem summary: In a PCS-based HA cluster, when failing over the LUN, the lun serial changes, and this causes multipath clients to misbehave (especially after an iSCSI client reboot). Some more of the setup: the setup makes use of two nodes with PCS-based cluster. The cluster setup was a modified follow up of this site: https://bm-stor.com/index.php/blog/Linux-cluster-with-ZFS-on-Cluster-in-a-Box/ , except that I use multipathing and not network teaming. iSCSI layout: targetcli ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 2] | | o- lun2-tier2 ............................................................ [/dev/mapper/T2-lun2 (6.6TiB) write-thru activated] | | o- lun3-tier3 ......................................................... [/dev/mapper/T3-lun1 (1024.0GiB) write-thru activated] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 2] | o- iqn.2005-05.com.poliva:cib.tier2 .................................................................................. [TPGs: 1] | | o- tpg1 .................................................................................................. [gen-acls, no-auth] | | o- acls .......................................................................................................... [ACLs: 0] | | o- luns .......................................................................................................... [LUNs: 1] | | | o- lun2 ......................................................................... [block/lun2-tier2 (/dev/mapper/T2-lun2)] | | o- portals .................................................................................................... [Portals: 2] | | o- 10.254.254.4:3260 ................................................................................................ [OK] | | o- 10.254.255.4:3260 ................................................................................................ [OK] | o- iqn.2005-05.com.poliva:cib.tier3 .................................................................................. [TPGs: 1] | o- tpg1 .................................................................................................. [gen-acls, no-auth] | o- acls .......................................................................................................... [ACLs: 0] | o- luns .......................................................................................................... [LUNs: 1] | | o- lun3 ......................................................................... [block/lun3-tier3 (/dev/mapper/T3-lun1)] | o- portals .................................................................................................... [Portals: 2] | o- 10.254.254.5:3260 ................................................................................................ [OK] | o- 10.254.255.5:3260 ................................................................................................ [OK] o- loopback ......................................................................................................... [Targets: 0] I do not use (unless required to) ACLs for the time being. After a LUN takeover/takeback (aka - relocation to another host), the IP address is backup (within about 10-20 seconds), the iSCSI target is up and available, and all cluster resources show as healthy., however, the client host, especially if rebooted, will not see the same identifier for this LUN (serial, word 83, name it as you like). It is 100% reproducible if two conditions happen: 1. The LUN is migrated to the other node 2. The client machine is rebooted (order is optional). # multipath -ll mpathc (36001405e3de7a9800000000000000000) dm-2 LIO-ORG,lun3-tier3 size=1024G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 0:0:0:3 sdb 8:16 active ready running `- 1:0:0:3 sda 8:0 active ready running [root@temp-iSCSI ~]# multipath -F [root@temp-iSCSI ~]# iscsiadm -m node -U all Logging out of session [sid: 1, target: iqn.2005-05.com.poliva:cib.tier3, portal: 10.254.255.5,3260] Logging out of session [sid: 2, target: iqn.2005-05.com.poliva:cib.tier3, portal: 10.254.254.5,3260] Logout of [sid: 1, target: iqn.2005-05.com.poliva:cib.tier3, portal: 10.254.255.5,3260] successful. Logout of [sid: 2, target: iqn.2005-05.com.poliva:cib.tier3, portal: 10.254.254.5,3260] successful. [root@temp-iSCSI ~]# iscsiadm -m node -L all Logging in to [iface: default, target: iqn.2005-05.com.poliva:cib.tier3, portal: 10.254.255.5,3260] (multiple) Logging in to [iface: default, target: iqn.2005-05.com.poliva:cib.tier3, portal: 10.254.254.5,3260] (multiple) Login to [iface: default, target: iqn.2005-05.com.poliva:cib.tier3, portal: 10.254.255.5,3260] successful. Login to [iface: default, target: iqn.2005-05.com.poliva:cib.tier3, portal: 10.254.254.5,3260] successful. [root@temp-iSCSI ~]# multipath [root@temp-iSCSI ~]# multipath -ll mpathd (3600140599e044f8681345d3aa4824abc) dm-2 LIO-ORG,lun3-tier3 size=1024G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 2:0:0:3 sdb 8:16 active ready running `- 3:0:0:3 sda 8:0 active ready running I am not using ACLs for the time being. I will integrate ACLs later on. Thanks for any insight, or even a simple tip on how I can maintain dedicated HA solution. Thanks! Etzion -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
