I have a testing lab where I'm attempting to get some more experience with LIO and targetcli. Is there an IRC channel where cluster and/or target folks hang out?
I have:
- A 3 node RHEL6.2 cluster with clvmd and GFS2
- A Fedora 16 box (kernel 3.1.6) with LIO/targetcli-2.0rc1.fb3-2 for my shared storage
Is there special configuration needs to be done on the target to enable PR, because it doesn't seem to be working. I'm not able to get fence_scsi working.
I'm seeing registrations, but no reservation.
From one of the RHEL6 cluster nodes, and attempt to do a read-key fails:
# sg_persist -n -i -k -d /dev/sdf
PR in: aborted command
Which generates this message over on my Fedora 16 scsi target:
filp_open(/var/target/pr/
aptpl_086b4b49-8736-45e1-a80c-2ddeb8a5a01e) for APTPL metadata failed
# cat /sys/kernel/config/target/core/iblock_0/store01/pr/res_*
APTPL Bit Status: Disabled
Ready to process PR APTPL metadata..
No SPC-3 Reservation holder
No SPC-3 Reservation holder
0x00000008
No SPC-3 Reservation holder
SPC-3 PR Registrations:
iSCSI Node: iqn.1994-05.com.redhat:226e63cf8cf5,i,0x00023d010000 Key: 0x00000000aa230001 PRgen: 0x00000003
iSCSI Node: iqn.1994-05.com.redhat:93471b6582,i,0x00023d010000 Key: 0x00000000aa230002 PRgen: 0x00000007
iSCSI Node: iqn.1994-05.com.redhat:21d24bc1b670,i,0x00023d010000 Key: 0x00000000aa230003 PRgen: 0x00000005
No SPC-3 Reservation holder
SPC3_PERSISTENT_RESERVATIONS
On the cluster node, this is what my fence_scsi log file looks like:
Jan 5 17:38:45 fence_scsi: [debug] main::do_register_ignore (node_key=aa230003, dev=/dev/sdf)
Jan 5 17:38:45 fence_scsi: [debug] main::do_reset (dev=/dev/sdf, status=0) (cmd=sg_turs /dev/sdf)
Jan 5 17:38:45 fence_scsi: [debug] main::do_register_ignore (err=0) (cmd=sg_persist -n -o -I -S aa230003 -d /dev/sdf)
Jan 5 17:38:45 fence_scsi: [error] main::get_reservation_key (err=11) (cmd=sg_persist -n -i -r -d /dev/sdf)
Running that failing sg_persist gives "PR in: aborted command"
A wireshark packet capture of that command shows:
=== packet generated by sg_persist ===
iSCSI (SCSI Command)
Opcode: SCSI Command (0x01)
.0.. .... = I: Queued delivery
Flags: 0xc1
1... .... = F: Final PDU in sequence
.1.. .... = R: Data will be read from target
..0. .... = W: No data will be written to target
.... .001 = Attr: Simple (0x01)
TotalAHSLength: 0x00
DataSegmentLength: 0x00000000
LUN: 0000000000000000
InitiatorTaskTag: 0x20000000
ExpectedDataTransferLength: 0x00002000
CmdSN: 0x000000e2
ExpStatSN: 0x4c6ecb1b
SCSI CDB Persistent Reserve In
[LUN: 0x0000]
[Command Set:Direct Access Device (0x00) (Using default commandset)]
Opcode: Persistent Reserve In (0x5e)
.... 0001 = Service Action: Read Reservation (0x01)
Allocation Length: 8192
Control: 0x00
00.. .... = Vendor specific: 0x00
..00 0... = Reserved: 0x00
.... .0.. = NACA: Normal ACA is not set
.... ..0. = Obsolete: 0x00
.... ...0 = Obsolete: 0x00
=== response from target ===
iSCSI (SCSI Response)
Opcode: SCSI Response (0x21)
Flags: 0x80
...0 .... = o: No overflow of read part of bi-directional command
.... 0... = u: No underflow of read part of bi-directional command
.... .0.. = O: No residual overflow occurred
.... ..0. = U: No residual underflow occurred
Response: Command completed at target (0x00)
Status: Check Condition (0x02)
TotalAHSLength: 0x00
DataSegmentLength: 0x00000062
InitiatorTaskTag: 0x20000000
StatSN: 0x4c6ecb1b
ExpCmdSN: 0x000000e3
MaxCmdSN: 0x000000f2
ExpDataSN: 0x00000000
BidiReadResidualCount: 0x00000000
ResidualCount: 0x00000000
Request in: 1
Time from request: 0.000096000 seconds
SenseLength: 0x0060
SCSI: SNS Info
[LUN: 0x0000]
Valid: 0
.111 0000 = SNS Error Type: Current Error (0x70)
Filemark: 0, EOM: 0, ILI: 0
.... 1011 = Sense Key: Command Aborted (0x0b)
Sense Info: 0x00000000
Additional Sense Length: 0
Command-Specific Information: 00000000
Additional Sense Code+Qualifier: Invalid Field In Cdb (0x2400)
Field Replaceable Unit Code: 0x00
0... .... = SKSV: False
Sense Key Specific: 000000
(there are more packets in the capture if needed)
My target's saveconfig.json looks like this:# cat /sys/kernel/config/target/core/iblock_0/store01/pr/res_*
APTPL Bit Status: Disabled
Ready to process PR APTPL metadata..
No SPC-3 Reservation holder
No SPC-3 Reservation holder
0x00000008
No SPC-3 Reservation holder
SPC-3 PR Registrations:
iSCSI Node: iqn.1994-05.com.redhat:226e63cf8cf5,i,0x00023d010000 Key: 0x00000000aa230001 PRgen: 0x00000003
iSCSI Node: iqn.1994-05.com.redhat:93471b6582,i,0x00023d010000 Key: 0x00000000aa230002 PRgen: 0x00000007
iSCSI Node: iqn.1994-05.com.redhat:21d24bc1b670,i,0x00023d010000 Key: 0x00000000aa230003 PRgen: 0x00000005
No SPC-3 Reservation holder
SPC3_PERSISTENT_RESERVATIONS
On the cluster node, this is what my fence_scsi log file looks like:
Jan 5 17:38:45 fence_scsi: [debug] main::do_register_ignore (node_key=aa230003, dev=/dev/sdf)
Jan 5 17:38:45 fence_scsi: [debug] main::do_reset (dev=/dev/sdf, status=0) (cmd=sg_turs /dev/sdf)
Jan 5 17:38:45 fence_scsi: [debug] main::do_register_ignore (err=0) (cmd=sg_persist -n -o -I -S aa230003 -d /dev/sdf)
Jan 5 17:38:45 fence_scsi: [error] main::get_reservation_key (err=11) (cmd=sg_persist -n -i -r -d /dev/sdf)
Running that failing sg_persist gives "PR in: aborted command"
A wireshark packet capture of that command shows:
=== packet generated by sg_persist ===
iSCSI (SCSI Command)
Opcode: SCSI Command (0x01)
.0.. .... = I: Queued delivery
Flags: 0xc1
1... .... = F: Final PDU in sequence
.1.. .... = R: Data will be read from target
..0. .... = W: No data will be written to target
.... .001 = Attr: Simple (0x01)
TotalAHSLength: 0x00
DataSegmentLength: 0x00000000
LUN: 0000000000000000
InitiatorTaskTag: 0x20000000
ExpectedDataTransferLength: 0x00002000
CmdSN: 0x000000e2
ExpStatSN: 0x4c6ecb1b
SCSI CDB Persistent Reserve In
[LUN: 0x0000]
[Command Set:Direct Access Device (0x00) (Using default commandset)]
Opcode: Persistent Reserve In (0x5e)
.... 0001 = Service Action: Read Reservation (0x01)
Allocation Length: 8192
Control: 0x00
00.. .... = Vendor specific: 0x00
..00 0... = Reserved: 0x00
.... .0.. = NACA: Normal ACA is not set
.... ..0. = Obsolete: 0x00
.... ...0 = Obsolete: 0x00
=== response from target ===
iSCSI (SCSI Response)
Opcode: SCSI Response (0x21)
Flags: 0x80
...0 .... = o: No overflow of read part of bi-directional command
.... 0... = u: No underflow of read part of bi-directional command
.... .0.. = O: No residual overflow occurred
.... ..0. = U: No residual underflow occurred
Response: Command completed at target (0x00)
Status: Check Condition (0x02)
TotalAHSLength: 0x00
DataSegmentLength: 0x00000062
InitiatorTaskTag: 0x20000000
StatSN: 0x4c6ecb1b
ExpCmdSN: 0x000000e3
MaxCmdSN: 0x000000f2
ExpDataSN: 0x00000000
BidiReadResidualCount: 0x00000000
ResidualCount: 0x00000000
Request in: 1
Time from request: 0.000096000 seconds
SenseLength: 0x0060
SCSI: SNS Info
[LUN: 0x0000]
Valid: 0
.111 0000 = SNS Error Type: Current Error (0x70)
Filemark: 0, EOM: 0, ILI: 0
.... 1011 = Sense Key: Command Aborted (0x0b)
Sense Info: 0x00000000
Additional Sense Length: 0
Command-Specific Information: 00000000
Additional Sense Code+Qualifier: Invalid Field In Cdb (0x2400)
Field Replaceable Unit Code: 0x00
0... .... = SKSV: False
Sense Key Specific: 000000
(there are more packets in the capture if needed)
{
"storage_objects": [
{
"attributes": {
"block_size": 512,
"emulate_dpo": 0,
"emulate_fua_read": 0,
"emulate_fua_write": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"is_nonrot": 0,
"max_sectors": 1024,
"max_unmap_block_desc_count": 0,
"max_unmap_lba_count": 0,
"optimal_sectors": 1024,
"queue_depth": 128,
"task_timeout": 0,
"unmap_granularity": 0,
"unmap_granularity_alignment": 0
},
"dev": "/dev/vg_station11/iscsi-
lun01",
"name": "store01",
"plugin": "block",
"wwn": "086b4b49-8736-45e1-a80c-2ddeb8a5a01e"
}
],
"targets": [
{
"fabric": "iscsi",
"tpgs": [
{
"attributes": {
"authentication": 0,
"cache_dynamic_acls": 0,
"default_cmdsn_depth": 16,
"demo_mode_write_protect": 1,
"generate_node_acls": 0,
"login_timeout": 15,
"netif_timeout": 2,
"prod_mode_write_protect": 0
},
"luns": [
{
"index": 0,
"storage_object": "/backstores/block/store01"
}
],
"node_acls": [
{
"attributes": {
"dataout_timeout": 3,
"dataout_timeout_retries": 5,
"default_erl": 0,
"nopin_response_timeout": 5,
"nopin_timeout": 5,
"random_datain_pdu_offsets": 0,
"random_datain_seq_offsets": 0,
"random_r2t_offsets": 0
},
"chap_mutual_password": "",
"chap_mutual_userid": "",
"chap_password": "",
"chap_userid": "",
"mapped_luns": [
{
"index": 0,
"write_protect": false
}
],
"node_wwn": "iqn.1994-05.com.redhat:21d24bc1b670",
"tcq_depth": 16
},
{
"attributes": {
"dataout_timeout": 3,
"dataout_timeout_retries": 5,
"default_erl": 0,
"nopin_response_timeout": 5,
"nopin_timeout": 5,
"random_datain_pdu_offsets": 0,
"random_datain_seq_offsets": 0,
"random_r2t_offsets": 0
},
"chap_mutual_password": "",
"chap_mutual_userid": "",
"chap_password": "",
"chap_userid": "",
"mapped_luns": [
{
"index": 0,
"write_protect": false
}
],
"node_wwn": "iqn.1994-05.com.redhat:93471b6582",
"tcq_depth": 16
},
{
"attributes": {
"dataout_timeout": 3,
"dataout_timeout_retries": 5,
"default_erl": 0,
"nopin_response_timeout": 5,
"nopin_timeout": 5,
"random_datain_pdu_offsets": 0,
"random_datain_seq_offsets": 0,
"random_r2t_offsets": 0
},
"chap_mutual_password": "",
"chap_mutual_userid": "",
"chap_password": "",
"chap_userid": "",
"mapped_luns": [
{
"index": 0,
"write_protect": false
}
],
"node_wwn": "iqn.1994-05.com.redhat:226e63cf8cf5",
"tcq_depth": 16
}
],
"portals": [
{
"ip_address": "10.100.0.12",
"port": 3260
}
],
"tag": 1
}
],
"wwn": "iqn.2003-01.org.linux-iscsi.station11.x8664:sn.32668e1cd52d"
}
]
}
"name": "store01",
"plugin": "block",
"wwn": "086b4b49-8736-45e1-a80c-2ddeb8a5a01e"
}
],
"targets": [
{
"fabric": "iscsi",
"tpgs": [
{
"attributes": {
"authentication": 0,
"cache_dynamic_acls": 0,
"default_cmdsn_depth": 16,
"demo_mode_write_protect": 1,
"generate_node_acls": 0,
"login_timeout": 15,
"netif_timeout": 2,
"prod_mode_write_protect": 0
},
"luns": [
{
"index": 0,
"storage_object": "/backstores/block/store01"
}
],
"node_acls": [
{
"attributes": {
"dataout_timeout": 3,
"dataout_timeout_retries": 5,
"default_erl": 0,
"nopin_response_timeout": 5,
"nopin_timeout": 5,
"random_datain_pdu_offsets": 0,
"random_datain_seq_offsets": 0,
"random_r2t_offsets": 0
},
"chap_mutual_password": "",
"chap_mutual_userid": "",
"chap_password": "",
"chap_userid": "",
"mapped_luns": [
{
"index": 0,
"write_protect": false
}
],
"node_wwn": "iqn.1994-05.com.redhat:21d24bc1b670",
"tcq_depth": 16
},
{
"attributes": {
"dataout_timeout": 3,
"dataout_timeout_retries": 5,
"default_erl": 0,
"nopin_response_timeout": 5,
"nopin_timeout": 5,
"random_datain_pdu_offsets": 0,
"random_datain_seq_offsets": 0,
"random_r2t_offsets": 0
},
"chap_mutual_password": "",
"chap_mutual_userid": "",
"chap_password": "",
"chap_userid": "",
"mapped_luns": [
{
"index": 0,
"write_protect": false
}
],
"node_wwn": "iqn.1994-05.com.redhat:93471b6582",
"tcq_depth": 16
},
{
"attributes": {
"dataout_timeout": 3,
"dataout_timeout_retries": 5,
"default_erl": 0,
"nopin_response_timeout": 5,
"nopin_timeout": 5,
"random_datain_pdu_offsets": 0,
"random_datain_seq_offsets": 0,
"random_r2t_offsets": 0
},
"chap_mutual_password": "",
"chap_mutual_userid": "",
"chap_password": "",
"chap_userid": "",
"mapped_luns": [
{
"index": 0,
"write_protect": false
}
],
"node_wwn": "iqn.1994-05.com.redhat:226e63cf8cf5",
"tcq_depth": 16
}
],
"portals": [
{
"ip_address": "10.100.0.12",
"port": 3260
}
],
"tag": 1
}
],
"wwn": "iqn.2003-01.org.linux-iscsi.station11.x8664:sn.32668e1cd52d"
}
]
}
-- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
