This patch contains iSCSI-SCST's README file. Signed-off-by: Vladislav Bolkhovitin <vst@xxxxxxxx> --- README.iscsi | 585 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 585 insertions(+) diff -uprN orig/linux-2.6.33/Documentation/scst/README.iscsi linux-2.6.33/Documentation/scst/README.iscsi --- orig/linux-2.6.33/Documentation/scst/README.iscsi +++ linux-2.6.33/Documentation/scst/README.iscsi @@ -0,0 +1,585 @@ +iSCSI SCST target driver +======================== + +ISCSI-SCST is a deeply reworked fork of iSCSI Enterprise Target (IET) +(http://iscsitarget.sourceforge.net). Reasons of the fork were: + + - To be able to use full power of SCST core. + + - To fix all the problems, corner cases issues and iSCSI standard + violations which IET has. + +See more info at http://iscsi-scst.sourceforge.net. + +Usage +----- + +See in http://iscsi-scst.sourceforge.net/iscsi-scst-howto.txt how to +configure iSCSI-SCST. + +In 2.0.0 usage of iscsi-scstd.conf as well as iscsi-scst-adm utility is +obsolete. Use the sysfs interface facilities instead. + +It is recommended to use TEST UNIT READY ("tur") command to check if +iSCSI-SCST target is alive in MPIO configurations. + +Also see SCST README file how to tune for the best performance. + +CAUTION: Working of target and initiator on the same host isn't fully +======= supported. See SCST README file for details. + +Sysfs interface +--------------- + +Root of SCST sysfs interface is /sys/kernel/scst_tgt. Root of iSCSI-SCST +is /sys/kernel/scst_tgt/targets/iscsi. It has the following entries: + + - None, one or more subdirectories for targets with name equal to names + of the corresponding targets. + + - IncomingUser[num] - optional one or more attributes containing user + name and password for incoming discovery user name. Not exist by + default and can be added through "mgmt" entry, see below. + + - OutgoingUser - optional attribute containing user name and password + for outgoing discovery user name. Not exist by default and can be + added through "mgmt" entry, see below. + + - iSNSServer - contains name or IP address of iSNS server with optional + "AccessControl" attribute, which allows to enable iSNS access + control. Empty by default. + + - enabled - using this attribute you can enable or disable iSCSI-SCST + accept new connections. It allows to finish configuring global + iSCSI-SCST attributes before it starts accepting new connections. 0 + by default. + + - open_state - read-only attribute, which allows to see if the user + space part of iSCSI-SCST connected to the kernel part. + + - trace_level - allows to enable and disable various tracing + facilities. See content of this file for help how to use it. + + - version - read-only attribute, which allows to see version of + iSCSI-SCST and enabled optional features. + + - mgmt - main management entry, which allows to configure iSCSI-SCST. + Namely, add/delete targets as well as add/delete optional global and + per-target attributes. See content of this file for help how to use + it. + +Each iSCSI-SCST sysfs file (attribute) can contain in the last line mark +"[key]". It is automatically added mark used to allow scstadmin to see +which attributes it should save in the config file. You can ignore it. + +Each target subdirectory contains the following entries: + + - ini_groups - subdirectory defining initiator groups for this target, + used to define per-initiator access control. See SCST core README for + more details. + + - luns - subdirectory defining LUNs of this target. See SCST core + README for more details. + + - sessions - subdirectory containing connected to this target sessions. + + - IncomingUser[num] - optional one or more attributes containing user + name and password for incoming user name. Not exist by default and can + be added through the "mgmt" entry, see above. + + - OutgoingUser - optional attribute containing user name and password + for outgoing user name. Not exist by default and can be added through + the "mgmt" entry, see above. + + - Entries defining default iSCSI parameters values used during iSCSI + parameters negotiation. Only entries which can be changed or make + sense are listed there. + + - QueuedCommands - defines maximum number of commands queued to any + session of this target. Default is 32 commands. + + - RspTimeout - defines the maximum time in seconds a command can wait for + response from initiator, otherwise the corresponding connection will + be closed. For performance reasons it is implemented as a timer, + which once in RspTimeout time checks the oldest command waiting for + response and, if it's older than RspTimeout, then it closes the + connection. Hence, a stalled connection will be closed in time + between RspTimeout and 2*RspTimeout. Default is 30 seconds. + + - NopInInterval - defines interval between NOP-In requests, which the + target will send on idle connections to check if the initiator is + still alive. If there is no NOP-Out reply from the initiator in + RspTimeout time, the corresponding connection will be closed. Default + is 30 seconds. If it's set to 0, then NOP-In requests are disabled. + + - enabled - using this attribute you can enable or disable iSCSI-SCST + accept new connections to this target. It allows to finish + configuring it before it starts accepting new connections. 0 by + default. + + - rel_tgt_id - allows to read or write SCSI Relative Target Port + Identifier attribute. This identifier is used to identify SCSI Target + Ports by some SCSI commands, mainly by Persistent Reservations + commands. This identifier must be unique among all SCST targets, but + for convenience SCST allows disabled targets to have not unique + rel_tgt_id. In this case SCST will not allow to enable this target + until rel_tgt_id becomes unique. This attribute initialized unique by + SCST by default. + + - tid - TID of this target. + +Subdirectory "sessions" contains one subdirectory for each connected +session with name equal to name of the connected initiator. + +Each session subdirectory contains the following entries: + + - One subdirectory for each TCP connection in this session. ISCSI-SCST + supports 1 connection per session, but the session subdirectory can + contain several connections: one active and other being closed. + + - Entries defining negotiated iSCSI parameters. Only parameters which + can be changed or make sense are listed there. + + - initiator_name - contains initiator name + + - sid - contains SID of this session + + - reinstating - contains reinstatement state of this session + + - force_close - write-only attribute, which allows to force close this + session. This is the only writable session attribute. + + - active_commands - contains number of active, i.e. not yet or being + executed, SCSI commands in this session. + + - commands - contains overall number of SCSI commands in this session. + +Each connection subdirectory contains the following entries: + + - cid - contains CID of this connection. + + - ip - contains IP address of the connected initiator. + + - state - contains processing state of this connection. + +Below is a sample script, which configures 1 virtual disk "disk1" using +/disk1 image and one target iqn.2006-10.net.vlnb:tgt with all default +parameters: + +#!/bin/bash + +modprobe scst +modprobe scst_vdisk + +echo "add_device disk1 filename=/disk1; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt + +service iscsi-scst start + +echo "add_target iqn.2006-10.net.vlnb:tgt" >/sys/kernel/scst_tgt/targets/iscsi/mgmt +echo "add disk1 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt + +echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled +echo 1 >/sys/kernel/scst_tgt/targets/iscsi/enabled + +Below is more advanced sample script, which configures more virtual +devices of various types, including virtual CDROM and 2 targets, one +with all default parameters, another one with some not default +parameters, incoming and outgoing user names for CHAP authentification, +and special permissions for initiator iqn.2005-03.org.open-iscsi:cacdcd2520, +which will see another set of devices. Also this sample configures CHAP +authentication for discovery sessions and iSNS server with access control. + +#!/bin/bash + +modprobe scst +modprobe scst_vdisk + +echo "add_device disk1 filename=/disk1; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt +echo "add_device disk2 filename=/disk2; blocksize=4096; nv_cache=1" >/sys/kernel/scst_tgt/handlers/vdisk_fileio/mgmt +echo "add_device blockio filename=/dev/sda5" >/sys/kernel/scst_tgt/handlers/vdisk_blockio/mgmt +echo "add_device nullio" >/sys/kernel/scst_tgt/handlers/vdisk_nullio/mgmt +echo "add_device cdrom" >/sys/kernel/scst_tgt/handlers/vcdrom/mgmt + +service iscsi-scst start + +echo "192.168.1.16 AccessControl" >/sys/kernel/scst_tgt/targets/iscsi/iSNSServer +echo "add_attribute IncomingUser joeD 12charsecret" >/sys/kernel/scst_tgt/targets/iscsi/mgmt +echo "add_attribute OutgoingUser jackD 12charsecret1" >/sys/kernel/scst_tgt/targets/iscsi/mgmt + +echo "add_target iqn.2006-10.net.vlnb:tgt" >/sys/kernel/scst_tgt/targets/iscsi/mgmt + +echo "add disk1 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt +echo "add cdrom 1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/mgmt + +echo "add_target iqn.2006-10.net.vlnb:tgt1" >/sys/kernel/scst_tgt/targets/iscsi/mgmt +echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 IncomingUser1 joe2 12charsecret2" >/sys/kernel/scst_tgt/targets/iscsi/mgmt +echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 IncomingUser joe 12charsecret" >/sys/kernel/scst_tgt/targets/iscsi/mgmt +echo "add_target_attribute iqn.2006-10.net.vlnb:tgt1 OutgoingUser jim1 12charpasswd" >/sys/kernel/scst_tgt/targets/iscsi/mgmt +echo "No" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/InitialR2T +echo "Yes" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ImmediateData +echo "8192" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxRecvDataSegmentLength +echo "8192" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxXmitDataSegmentLength +echo "131072" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxBurstLength +echo "32768" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/FirstBurstLength +echo "1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/MaxOutstandingR2T +echo "CRC32C,None" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/HeaderDigest +echo "CRC32C,None" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/DataDigest +echo "32" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/QueuedCommands + +echo "add disk2 0" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/mgmt +echo "add nullio 26" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/mgmt + +echo "create special_ini" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/mgmt +echo "add blockio 0 read_only=1" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/luns/mgmt +echo "add iqn.2005-03.org.open-iscsi:cacdcd2520" >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/initiators/mgmt + +echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/enabled +echo 1 >/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt1/enabled + +echo 1 >/sys/kernel/scst_tgt/targets/iscsi/enabled + +The resulting overall SCST sysfs hierarchy with an initiator connected to +both iSCSI-SCST targets will look like: + +/sys/kernel/scst_tgt +|-- devices +| |-- blockio +| | |-- blocksize +| | |-- exported +| | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/ini_groups/special_ini/luns/0 +| | |-- filename +| | |-- handler -> ../../handlers/vdisk_blockio +| | |-- read_only +| | |-- removable +| | |-- resync_size +| | |-- size_mb +| | |-- t10_dev_id +| | |-- threads_num +| | |-- threads_pool_type +| | |-- type +| | `-- usn +| |-- cdrom +| | |-- exported +| | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/1 +| | |-- filename +| | |-- handler -> ../../handlers/vcdrom +| | |-- size_mb +| | |-- t10_dev_id +| | |-- threads_num +| | |-- threads_pool_type +| | |-- type +| | `-- usn +| |-- disk1 +| | |-- blocksize +| | |-- exported +| | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/0 +| | |-- filename +| | |-- handler -> ../../handlers/vdisk_fileio +| | |-- nv_cache +| | |-- o_direct +| | |-- read_only +| | |-- removable +| | |-- resync_size +| | |-- size_mb +| | |-- t10_dev_id +| | |-- type +| | |-- usn +| | `-- write_through +| |-- disk2 +| | |-- blocksize +| | |-- exported +| | | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/0 +| | |-- filename +| | |-- handler -> ../../handlers/vdisk_fileio +| | |-- nv_cache +| | |-- o_direct +| | |-- read_only +| | |-- removable +| | |-- resync_size +| | |-- size_mb +| | |-- t10_dev_id +| | |-- threads_num +| | |-- threads_pool_type +| | |-- threads_num +| | |-- threads_pool_type +| | |-- type +| | |-- usn +| | `-- write_through +| `-- nullio +| |-- blocksize +| |-- exported +| | `-- export0 -> ../../../targets/iscsi/iqn.2006-10.net.vlnb:tgt1/luns/26 +| |-- handler -> ../../handlers/vdisk_nullio +| |-- read_only +| |-- removable +| |-- size_mb +| |-- t10_dev_id +| |-- threads_num +| |-- threads_pool_type +| |-- type +| `-- usn +|-- handlers +| |-- vcdrom +| | |-- cdrom -> ../../devices/cdrom +| | |-- mgmt +| | |-- trace_level +| | `-- type +| |-- vdisk_blockio +| | |-- blockio -> ../../devices/blockio +| | |-- mgmt +| | |-- trace_level +| | `-- type +| |-- vdisk_fileio +| | |-- disk1 -> ../../devices/disk1 +| | |-- disk2 -> ../../devices/disk2 +| | |-- mgmt +| | |-- trace_level +| | `-- type +| `-- vdisk_nullio +| |-- mgmt +| |-- nullio -> ../../devices/nullio +| |-- trace_level +| `-- type +|-- sgv +| |-- global_stats +| |-- sgv +| | `-- stats +| |-- sgv-clust +| | `-- stats +| `-- sgv-dma +| `-- stats +|-- targets +| `-- iscsi +| |-- IncomingUser +| |-- OutgoingUser +| |-- enabled +| |-- iSNSServer +| |-- iqn.2006-10.net.vlnb:tgt +| | |-- DataDigest +| | |-- FirstBurstLength +| | |-- HeaderDigest +| | |-- ImmediateData +| | |-- InitialR2T +| | |-- MaxBurstLength +| | |-- MaxOutstandingR2T +| | |-- MaxRecvDataSegmentLength +| | |-- MaxXmitDataSegmentLength +| | |-- NopInInterval +| | |-- QueuedCommands +| | |-- RspTimeout +| | |-- enabled +| | |-- ini_groups +| | | `-- mgmt +| | |-- luns +| | | |-- 0 +| | | | |-- device -> ../../../../../devices/disk1 +| | | | `-- read_only +| | | |-- 1 +| | | | |-- device -> ../../../../../devices/cdrom +| | | | `-- read_only +| | | `-- mgmt +| | |-- rel_tgt_id +| | |-- sessions +| | | `-- iqn.2005-03.org.open-iscsi:cacdcd2520 +| | | |-- 10.170.75.2 +| | | | |-- cid +| | | | |-- ip +| | | | `-- state +| | | |-- DataDigest +| | | |-- FirstBurstLength +| | | |-- HeaderDigest +| | | |-- ImmediateData +| | | |-- InitialR2T +| | | |-- MaxBurstLength +| | | |-- MaxOutstandingR2T +| | | |-- MaxRecvDataSegmentLength +| | | |-- MaxXmitDataSegmentLength +| | | |-- active_commands +| | | |-- commands +| | | |-- force_close +| | | |-- initiator_name +| | | |-- luns -> ../../luns +| | | |-- reinstating +| | | `-- sid +| | `-- tid +| |-- iqn.2006-10.net.vlnb:tgt1 +| | |-- DataDigest +| | |-- FirstBurstLength +| | |-- HeaderDigest +| | |-- ImmediateData +| | |-- IncomingUser +| | |-- IncomingUser1 +| | |-- InitialR2T +| | |-- MaxBurstLength +| | |-- MaxOutstandingR2T +| | |-- MaxRecvDataSegmentLength +| | |-- MaxXmitDataSegmentLength +| | |-- OutgoingUser +| | |-- NopInInterval +| | |-- QueuedCommands +| | |-- RspTimeout +| | |-- enabled +| | |-- ini_groups +| | | |-- mgmt +| | | `-- special_ini +| | | |-- initiators +| | | | |-- iqn.2005-03.org.open-iscsi:cacdcd2520 +| | | | `-- mgmt +| | | `-- luns +| | | |-- 0 +| | | | |-- device -> ../../../../../../../devices/blockio +| | | | `-- read_only +| | | `-- mgmt +| | |-- luns +| | | |-- 0 +| | | | |-- device -> ../../../../../devices/disk2 +| | | | `-- read_only +| | | |-- 26 +| | | | |-- device -> ../../../../../devices/nullio +| | | | `-- read_only +| | | `-- mgmt +| | |-- rel_tgt_id +| | |-- sessions +| | | `-- iqn.2005-03.org.open-iscsi:cacdcd2520 +| | | |-- 10.170.75.2 +| | | | |-- cid +| | | | |-- ip +| | | | `-- state +| | | |-- DataDigest +| | | |-- FirstBurstLength +| | | |-- HeaderDigest +| | | |-- ImmediateData +| | | |-- InitialR2T +| | | |-- MaxBurstLength +| | | |-- MaxOutstandingR2T +| | | |-- MaxRecvDataSegmentLength +| | | |-- MaxXmitDataSegmentLength +| | | |-- active_commands +| | | |-- commands +| | | |-- force_close +| | | |-- initiator_name +| | | |-- luns -> ../../ini_groups/special_ini/luns +| | | |-- reinstating +| | | `-- sid +| | `-- tid +| |-- mgmt +| |-- open_state +| |-- trace_level +| `-- version +|-- threads +|-- trace_level +`-- version + +Troubleshooting +--------------- + +If you have any problems, start troubleshooting from looking at the +kernel and system logs. In the kernel log iSCSI-SCST and SCST core send +their messages, in the system log iscsi-scstd sends its messages. In +most Linux distributions both those logs are put to /var/log/messages +file. + +Then, it might be helpful to increase level of logging. For kernel +modules you should make the debug build by enabling CONFIG_SCST_DEBUG. + +If after looking on the logs the reason of your problem is still unclear +for you, report to SCST mailing list scst-devel@xxxxxxxxxxxxxxxxxxxxxx + +Work if target's backstorage or link is too slow +------------------------------------------------ + +In some cases you can experience I/O stalls or see in the kernel log +abort or reset messages. It can happen under high I/O load, when your +target's backstorage gets overloaded, or working over a slow link, when +the link can't serve all the queued commands on time, + +To workaround it you can reduce QueuedCommands parameter for the +corresponding target to some lower value, like 8 (default is 32). + +Also see SCST README file for more details about that issue and ways to +prevent it. + +Performance advices +------------------- + +1. If you use Windows XP or Windows 2003+ as initiators, you should +consider to decrease TcpAckFrequency parameter to 1. See +http://support.microsoft.com/kb/328890/ or google for "TcpAckFrequency" +for more details. + +2. See how to get the maximum throughput from iSCSI, for instance, at +http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html. +It's about VMware, but its recommendations apply to other environments +as well. + +3. ISCSI initiators built in pre-CentOS/RHEL 5 reported to have some +performance problems. If you use it, it is strongly advised to upgrade. + +4. If you are going to use your target in an VM environment, for +instance as a shared storage with VMware, make sure all your VMs +connected to the target via *separate* sessions, i.e. each VM has own +connection to the target, not all VMs connected using a single +connection. You can check it using SCST sysfs interface. If you +miss it, you can greatly loose performance of parallel access to your +target from different VMs. This isn't related to the case if your VMs +are using the same shared storage, like with VMFS, for instance. In this +case all your VM hosts will be connected to the target via separate +sessions, which is enough. + +5. Many dual port network adapters are not able to transfer data +simultaneously on both ports, i.e. they transfer data via both ports on +the same speed as via any single port. Thus, using such adapters in MPIO +configuration can't improve performance. To allow MPIO to have double +performance you should either use separate network adapters, or find a +dual-port adapter capable to to transfer data simultaneously on both +ports. You can check it by running 2 iperf's through both ports in +parallel. + +6. Since network offload works much better in the write direction, than +for reading (simplifying, in the read direction often there's additional +data copy) in many cases with 10GbE in a single initiator-target pair +the initiator's CPU is a bottleneck, so you can see the initiator can +read data on much slower rate, than write. You can check it by watching +*each particular* CPU load to find out if any of them is close to 100% +load, including IRQ processing load. Note, many tools like vmstat give +aggregate load on all CPUs, so with 4 cores 25% corresponds to 100% load +of any single CPU. + +7. See SCST core's README for more advices. Especially pay attention to +have io_grouping_type option set correctly. + +Compilation options +------------------- + +There are the following compilation options, that could be commented +in/out in the kernel's module Makefile: + + - CONFIG_SCST_DEBUG - turns on some debugging code, including some logging. + Makes the driver considerably bigger and slower, producing large amount of + log data. + + - CONFIG_SCST_TRACING - turns on ability to log events. Makes the driver + considerably bigger and leads to some performance loss. + + - CONFIG_SCST_EXTRACHECKS - adds extra validity checks in the various places. + + - CONFIG_SCST_ISCSI_DEBUG_DIGEST_FAILURES - simulates digest failures in + random places. + +Credits +------- + +Thanks to: + + * Ming Zhang <blackmagic02881@xxxxxxxxx> for fixes + + * Krzysztof Blaszkowski <kb@xxxxxxxxxxxxxxx> for many fixes + + * Alexey Kuznetsov <kuznet@xxxxxxxxxxxxx> for comments and help in + debugging + + * Tomasz Chmielewski <mangoo@xxxxxxxx> for testing and suggestions + + * Bart Van Assche <bart.vanassche@xxxxxxxxx> for a lot of help + +Vladislav Bolkhovitin <vst@xxxxxxxx>, http://scst.sourceforge.net -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
