On 12/12/2014 02:12 PM, Nicholas A. Bellinger wrote: > Hi Ilias, > > Thanks for your detailed email. Comments are inline below. > > On Wed, 2014-12-10 at 18:36 +0200, Ilias Tsitsimpis wrote: >> Hi all, >> >> I am trying out Linux SCSI Target and it seems that the way persistent >> reservations are handled does not conform with SPC-4. >> SPC4r17 (5.7.10) states that: >> >> The persistent reservation holder is determined by the type of the >> persistent reservation as follows: >> a) For a persistent reservation of the type Write Exclusive – All >> Registrants or Exclusive Access – All Registrants, the persistent >> reservation holder is any registered I_T nexus; >> >> While Table 51 states that: >> >> 'Write Exclusive – All Registrants' or 'Exclusive Access – All >> Registrants' reservations shall be released when the registration >> for the last registered I_T nexus is removed or when the type or >> scope is changed. >> >> This doesn't seem to be the case with Linux SCSI Target. More >> particularly, I have set up an iSCSI target, using the Linux SCSI Target >> implementation, and I am connecting to it through two different VMs >> using the open-iscsi project. >> >> root@node1:~# sg_inq /dev/sda >> standard INQUIRY: >> PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] >> [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2 >> SCCS=1 ACC=0 TPGS=3 3PC=1 Protect=0 BQue=0 >> EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 >> [RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1 >> length=36 (0x24) Peripheral device type: disk >> Vendor identification: LIO-ORG >> Product identification: FILEIO >> Product revision level: 4.0 >> Unit serial number: 9efc01f5-429d-42be-b507-9dc4439f511d >> >> root@node2:~# sg_inq /dev/sda >> standard INQUIRY: >> PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] >> [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2 >> SCCS=1 ACC=0 TPGS=3 3PC=1 Protect=0 BQue=0 >> EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 >> [RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1 >> length=36 (0x24) Peripheral device type: disk >> Vendor identification: LIO-ORG >> Product identification: FILEIO >> Product revision level: 4.0 >> Unit serial number: 9efc01f5-429d-42be-b507-9dc4439f511d >> >> >> I register both of the IT_nexuses with the target. >> >> root@node1:~# sg_persist --out --register --param-sark 0x1 /dev/sda >> root@node2:~# sg_persist --out --register --param-sark 0x2 /dev/sda I see a couple of problems here. First, understand that group reservations are on an ITL-nexus basis. So by using /dev/sda twice you are really registering the key "1" and then attempting to reregister the key "2". You actually _can_ reregister a new key for an existing ITL nexus, but (1) I don't think that's what you wanted to do, and (2) you did it incorrectly. To actually re-register the ITL Nexus, you need to provide the current key when setting a new key, e.g.: sg_persist --out --register --param-rk=1 --param-sark=2 /dev/sda >> >> root@node1:~# sg_persist --in --read-full-status /dev/sda >> LIO-ORG FILEIO 4.0 >> Peripheral device type: disk >> PR generation=0x2 >> Key=0x1 >> All target ports bit clear >> Relative port address: 0x1 >> not reservation holder >> Transport Id of initiator: >> iSCSI name and session id: iqn.1993-08.org.debian:01:node1 >> Key=0x2 >> All target ports bit clear >> Relative port address: 0x1 >> not reservation holder >> Transport Id of initiator: >> iSCSI name and session id: iqn.1993-08.org.debian:01:node2 >> This is an bug with the SCSI target, in my opinion. Since you did not supply the reservation key in your second attempt, you should have gotten a check condition. [See SPC4r33, Section 5.9 Persistent Reservation Overview, Table 9 Register behaviors for a REGISTER service action.] > > Looks fine. > >> Then I get a persistent reservation with type 'Exclusive Access - All >> Registrants' for node1. >> >> root@node1:~# sg_persist --out --reserve --prout-type=8 --param-rk=0x1 /dev/sda >> root@node1:~# sg_persist --in --read-full-status /dev/sda >> LIO-ORG FILEIO 4.0 >> Peripheral device type: disk >> PR generation=0x2 >> Key=0x1 >> All target ports bit clear >> Relative port address: 0x1 >> << Reservation holder >> >> scope: LU_SCOPE, type: Exclusive Access, all registrants >> Transport Id of initiator: >> iSCSI name and session id: iqn.1993-08.org.debian:01:node1 >> Key=0x2 >> All target ports bit clear >> Relative port address: 0x1 >> not reservation holder >> Transport Id of initiator: >> iSCSI name and session id: iqn.1993-08.org.debian:01:node2 >> >> The node2 is listed as "not reservation holder". >> Shouldn't node2 be displayed as a Reservation holder since the >> persistent reservation is of type 'Exclusive Access, all registrants'? >> Since you are seeing two keys here for one ITL nexus (whic is a bug), I don't think it matters that the second key does not show up as a reservation holder. What you really want to do is test from two different initiators. You can do this in serveral ways. One simple way is to use two different hosts. But you can also do this from one host using two different interfaces. This can be two separate physical interfaces, or you can set up virtual interfaces with separate iSCSI Initiator names using open-iscsi, as I do for testing. (Let me know if you want more detail on this.) I created a document that explains this stuff in english -- well, as much as I have mastered english, anyway, at: http://gonzoleeman.net/documents/scsi-3-pgr-tutorial-v1.0 > > Good question. So from the perspective of CDB filtering logic in > drivers/target/target_core_pr.c:core_scsi3_pr_seq_non_holder(), > the /dev/sdb device above (registration active, not explicit reservation > holder) will still be treated like a implicit reservation holder with an > all registrants type reservation: > > Specifically the code here: > > switch (pr_reg_type) { > ....... > > case PR_TYPE_WRITE_EXCLUSIVE_ALLREG: > we = 1; > case PR_TYPE_EXCLUSIVE_ACCESS_ALLREG: > /* > * Each registered I_T Nexus is a reservation holder. > */ > all_reg = 1; > if ((se_deve->def_pr_registered) && !(ignore_reg)) > registered_nexus = 1; > break; > > and further down here: > > if (we && !registered_nexus) { > .............. > > } else if ((reg_only) || (all_reg)) { > if (registered_nexus) { > /* > * For PR_*_REG_ONLY and PR_*_ALL_REG reservations, > * allow commands from registered nexuses. > */ > > pr_debug("Allowing implicit CDB: 0x%02x for %s" > " reservation\n", cdb[0], > core_scsi3_pr_dump_type(pr_reg_type)); > > return 0; > } > } > > ensures that when another I_T nexus takes an explicit reservation with > an all registrants type reservation (eg: /dev/sda), that all of the > implicit reservation holders (eg: /dev/sdb) are treated the same. > > So from the perspective of the READ_FULL_STATUS output above, my > original interpretation was that the R_HOLDER is intended to display the > explicit reservation holder obtained through a RESERVE service action, > and not the implicit reservation holders. Except for the second key on a single ITL nexus, I believe the code is doing the right thing. I actually have a persistent reservation test suite, written in Python, that uses sg_persist to validate SCSI targets. I'll check to see if I'm testing that a re-register fails, and I'll post the test suite results here one way or the other. > > After looking at SPC-4 again, I don't see any language about R_HOLDER + > being specific to explicit RESERVEs or implicit all registrants > reservations. > >> If I then proceed to unregister node1, the reservation gets released >> altogether. Shouldn't the reservation remain, until the last registered >> I_T nexus goes away? I think this violates SPC-4, please see the excerpt >> above. >> >> root@node1:~# sg_persist --out --register --param-rk=0x1 --param-sark=0x0 /dev/sda >> root@node1:~# sg_persist --in --read-full-status /dev/sda >> LIO-ORG FILEIO 4.0 >> Peripheral device type: disk >> PR generation=0x2 >> Key=0x2 >> All target ports bit clear >> Relative port address: 0x1 >> not reservation holder >> Transport Id of initiator: >> iSCSI name and session id: iqn.1993-08.org.debian:01:node2 >> >> root@node2:~# sg_persist --in --read-reservation /dev/sda >> LIO-ORG FILEIO 4.0 >> Peripheral device type: disk >> PR generation=0x2, there is NO reservation held >> >> Is this a bug or am I missing something? >> > > One requirement about the current implementation of all registrants > logic is that it requires one explicit reservation to be in place, for > all of the other registrations to be treated as implicit reservations as > per the spec. > > This is because as you might have noticed above, the reservation type to > declare all registrants is not indicated at registration time, but at > explicit reservation time. > > So given the ambiguity in the spec in this regard, I think this would be > a great question for the T10 list. I'll try to articulate the points > above, and send out an email to them shortly for clarification. > > Thanks for reporting! Nicholas: I have been working on PGR for years. The PGR SCSI spec was added by a co-worker of mine at Sun. I'd be glad to try to answer your question, but I'm not clear what you're asking here. It is the case that in all-registrants reservations, having the original reservation requester go away should not make the reservation go away. Is that what you are asking? -- Lee Duncan -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
