Hi Lee, Thanks for testing this. Please see my comments inline. On Wed, Dec 17, 2014 at 03:36PM, Lee Duncan wrote: > On 12/12/2014 02:12 PM, Nicholas A. Bellinger wrote: > > Hi Ilias, > > > > Thanks for your detailed email. Comments are inline below. > > > > On Wed, 2014-12-10 at 18:36 +0200, Ilias Tsitsimpis wrote: > >> Hi all, > >> > >> I am trying out Linux SCSI Target and it seems that the way persistent > >> reservations are handled does not conform with SPC-4. > >> SPC4r17 (5.7.10) states that: > >> > >> The persistent reservation holder is determined by the type of the > >> persistent reservation as follows: > >> a) For a persistent reservation of the type Write Exclusive – All > >> Registrants or Exclusive Access – All Registrants, the persistent > >> reservation holder is any registered I_T nexus; > >> > >> While Table 51 states that: > >> > >> 'Write Exclusive – All Registrants' or 'Exclusive Access – All > >> Registrants' reservations shall be released when the registration > >> for the last registered I_T nexus is removed or when the type or > >> scope is changed. > >> > >> This doesn't seem to be the case with Linux SCSI Target. More > >> particularly, I have set up an iSCSI target, using the Linux SCSI Target > >> implementation, and I am connecting to it through two different VMs > >> using the open-iscsi project. > >> > >> root@node1:~# sg_inq /dev/sda > >> standard INQUIRY: > >> PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] > >> [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2 > >> SCCS=1 ACC=0 TPGS=3 3PC=1 Protect=0 BQue=0 > >> EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 > >> [RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1 > >> length=36 (0x24) Peripheral device type: disk > >> Vendor identification: LIO-ORG > >> Product identification: FILEIO > >> Product revision level: 4.0 > >> Unit serial number: 9efc01f5-429d-42be-b507-9dc4439f511d > >> > >> root@node2:~# sg_inq /dev/sda > >> standard INQUIRY: > >> PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] > >> [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2 > >> SCCS=1 ACC=0 TPGS=3 3PC=1 Protect=0 BQue=0 > >> EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 > >> [RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1 > >> length=36 (0x24) Peripheral device type: disk > >> Vendor identification: LIO-ORG > >> Product identification: FILEIO > >> Product revision level: 4.0 > >> Unit serial number: 9efc01f5-429d-42be-b507-9dc4439f511d > >> > >> > >> I register both of the IT_nexuses with the target. > >> > >> root@node1:~# sg_persist --out --register --param-sark 0x1 /dev/sda > >> root@node2:~# sg_persist --out --register --param-sark 0x2 /dev/sda > > I see a couple of problems here. > > First, understand that group reservations are on an ITL-nexus basis. So > by using /dev/sda twice you are really registering the key "1" and then > attempting to reregister the key "2". I run the above commands in two different machines (node1 and node2). So I am registering two different ITL nexuses with the same target. > You actually _can_ reregister a new key for an existing ITL nexus, but > (1) I don't think that's what you wanted to do, and (2) you did it > incorrectly. > > To actually re-register the ITL Nexus, you need to provide the current > key when setting a new key, e.g.: > > sg_persist --out --register --param-rk=1 --param-sark=2 /dev/sda > > >> > >> root@node1:~# sg_persist --in --read-full-status /dev/sda > >> LIO-ORG FILEIO 4.0 > >> Peripheral device type: disk > >> PR generation=0x2 > >> Key=0x1 > >> All target ports bit clear > >> Relative port address: 0x1 > >> not reservation holder > >> Transport Id of initiator: > >> iSCSI name and session id: iqn.1993-08.org.debian:01:node1 > >> Key=0x2 > >> All target ports bit clear > >> Relative port address: 0x1 > >> not reservation holder > >> Transport Id of initiator: > >> iSCSI name and session id: iqn.1993-08.org.debian:01:node2 > >> > > This is an bug with the SCSI target, in my opinion. Since you did not > supply the reservation key in your second attempt, you should have > gotten a check condition. > > [See SPC4r33, Section 5.9 Persistent Reservation Overview, Table 9 > Register behaviors for a REGISTER service action.] > > > > > Looks fine. > > > >> Then I get a persistent reservation with type 'Exclusive Access - All > >> Registrants' for node1. > >> > >> root@node1:~# sg_persist --out --reserve --prout-type=8 --param-rk=0x1 /dev/sda > >> root@node1:~# sg_persist --in --read-full-status /dev/sda > >> LIO-ORG FILEIO 4.0 > >> Peripheral device type: disk > >> PR generation=0x2 > >> Key=0x1 > >> All target ports bit clear > >> Relative port address: 0x1 > >> << Reservation holder >> > >> scope: LU_SCOPE, type: Exclusive Access, all registrants > >> Transport Id of initiator: > >> iSCSI name and session id: iqn.1993-08.org.debian:01:node1 > >> Key=0x2 > >> All target ports bit clear > >> Relative port address: 0x1 > >> not reservation holder > >> Transport Id of initiator: > >> iSCSI name and session id: iqn.1993-08.org.debian:01:node2 > >> > >> The node2 is listed as "not reservation holder". > >> Shouldn't node2 be displayed as a Reservation holder since the > >> persistent reservation is of type 'Exclusive Access, all registrants'? > >> > > Since you are seeing two keys here for one ITL nexus (whic is a bug), I > don't think it matters that the second key does not show up as a > reservation holder. > > What you really want to do is test from two different initiators. You > can do this in serveral ways. One simple way is to use two different > hosts. But you can also do this from one host using two different > interfaces. This can be two separate physical interfaces, or you can set > up virtual interfaces with separate iSCSI Initiator names using > open-iscsi, as I do for testing. (Let me know if you want more detail on > this.) This is exactly what I did. Please re-read my original mail and keep in mind that I run the register commands in two different machines. Thanks, Ilias > I created a document that explains this stuff in english -- well, as > much as I have mastered english, anyway, at: > > http://gonzoleeman.net/documents/scsi-3-pgr-tutorial-v1.0 > > > > > Good question. So from the perspective of CDB filtering logic in > > drivers/target/target_core_pr.c:core_scsi3_pr_seq_non_holder(), > > the /dev/sdb device above (registration active, not explicit reservation > > holder) will still be treated like a implicit reservation holder with an > > all registrants type reservation: > > > > Specifically the code here: > > > > switch (pr_reg_type) { > > ....... > > > > case PR_TYPE_WRITE_EXCLUSIVE_ALLREG: > > we = 1; > > case PR_TYPE_EXCLUSIVE_ACCESS_ALLREG: > > /* > > * Each registered I_T Nexus is a reservation holder. > > */ > > all_reg = 1; > > if ((se_deve->def_pr_registered) && !(ignore_reg)) > > registered_nexus = 1; > > break; > > > > and further down here: > > > > if (we && !registered_nexus) { > > .............. > > > > } else if ((reg_only) || (all_reg)) { > > if (registered_nexus) { > > /* > > * For PR_*_REG_ONLY and PR_*_ALL_REG reservations, > > * allow commands from registered nexuses. > > */ > > > > pr_debug("Allowing implicit CDB: 0x%02x for %s" > > " reservation\n", cdb[0], > > core_scsi3_pr_dump_type(pr_reg_type)); > > > > return 0; > > } > > } > > > > ensures that when another I_T nexus takes an explicit reservation with > > an all registrants type reservation (eg: /dev/sda), that all of the > > implicit reservation holders (eg: /dev/sdb) are treated the same. > > > > So from the perspective of the READ_FULL_STATUS output above, my > > original interpretation was that the R_HOLDER is intended to display the > > explicit reservation holder obtained through a RESERVE service action, > > and not the implicit reservation holders. > > Except for the second key on a single ITL nexus, I believe the code is > doing the right thing. > > I actually have a persistent reservation test suite, written in Python, > that uses sg_persist to validate SCSI targets. I'll check to see if I'm > testing that a re-register fails, and I'll post the test suite results > here one way or the other. > > > > > After looking at SPC-4 again, I don't see any language about R_HOLDER + > > being specific to explicit RESERVEs or implicit all registrants > > reservations. > > > >> If I then proceed to unregister node1, the reservation gets released > >> altogether. Shouldn't the reservation remain, until the last registered > >> I_T nexus goes away? I think this violates SPC-4, please see the excerpt > >> above. > >> > >> root@node1:~# sg_persist --out --register --param-rk=0x1 --param-sark=0x0 /dev/sda > >> root@node1:~# sg_persist --in --read-full-status /dev/sda > >> LIO-ORG FILEIO 4.0 > >> Peripheral device type: disk > >> PR generation=0x2 > >> Key=0x2 > >> All target ports bit clear > >> Relative port address: 0x1 > >> not reservation holder > >> Transport Id of initiator: > >> iSCSI name and session id: iqn.1993-08.org.debian:01:node2 > >> > >> root@node2:~# sg_persist --in --read-reservation /dev/sda > >> LIO-ORG FILEIO 4.0 > >> Peripheral device type: disk > >> PR generation=0x2, there is NO reservation held > >> > >> Is this a bug or am I missing something? > >> > > > > One requirement about the current implementation of all registrants > > logic is that it requires one explicit reservation to be in place, for > > all of the other registrations to be treated as implicit reservations as > > per the spec. > > > > This is because as you might have noticed above, the reservation type to > > declare all registrants is not indicated at registration time, but at > > explicit reservation time. > > > > So given the ambiguity in the spec in this regard, I think this would be > > a great question for the T10 list. I'll try to articulate the points > > above, and send out an email to them shortly for clarification. > > > > Thanks for reporting! > > Nicholas: I have been working on PGR for years. The PGR SCSI spec was > added by a co-worker of mine at Sun. I'd be glad to try to answer your > question, but I'm not clear what you're asking here. > > It is the case that in all-registrants reservations, having the original > reservation requester go away should not make the reservation go away. > Is that what you are asking? > > -- > Lee Duncan
Attachment:
signature.asc
Description: Digital signature
